[PATCH 00/19] qapi: statically type schema.py

[PATCH 00/19] qapi: statically type schema.py

[John Snow]

Hi! This branch has some comical name like python-qapi-cleanup-pt6-v2
but, simply, it finishes what I started and statically types all of the
QAPI generator code.

GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/1074124051

Patch 1 is an actual fix,
Patch 2 is a minor patch for pylint with no runtime effect,
Patches 3-15 fix individual typing issues that have some runtime effect.
Patch 16 adds typing information,
Patch 17 removes the schema.py exemption from the mypy conf,
Patches 18-19 are optional.

Most of the patches (expect 16) are very small. I'm sure we'll need to
rework parts of this, but hey, gotta start somewhere.

--js

John Snow (19):
  qapi/schema: fix QAPISchemaEntity.__repr__()
  qapi/schema: add pylint suppressions
  qapi/schema: name QAPISchemaInclude entities
  qapi/schema: declare type for QAPISchemaObjectTypeMember.type
  qapi/schema: make c_type() and json_type() abstract methods
  qapi/schema: adjust type narrowing for mypy's benefit
  qapi/introspect: assert schema.lookup_type did not fail
  qapi/schema: add static typing and assertions to lookup_type()
  qapi/schema: assert info is present when necessary
  qapi/schema: make QAPISchemaArrayType.element_type non-Optional
  qapi/schema: fix QAPISchemaArrayType.check's call to resolve_type
  qapi/schema: split "checked" field into "checking" and "checked"
  qapi/schema: fix typing for QAPISchemaVariants.tag_member
  qapi/schema: assert QAPISchemaVariants are QAPISchemaObjectType
  qapi/parser: demote QAPIExpression to Dict[str, Any]
  qapi/schema: add type hints
  qapi/schema: turn on mypy strictness
  qapi/schema: remove unnecessary asserts
  qapi/schema: refactor entity lookup helpers

 docs/sphinx/qapidoc.py |   2 +-
 scripts/qapi/mypy.ini  |   5 -
 scripts/qapi/parser.py |   3 +-
 scripts/qapi/pylintrc  |   5 -
 scripts/qapi/schema.py | 723 ++++++++++++++++++++++++++++-------------
 5 files changed, 498 insertions(+), 240 deletions(-)

-- 
2.41.0

[PATCH 01/19] qapi/schema: fix QAPISchemaEntity.__repr__()

[John Snow]

This needs parentheses to work how you want it to:

>>> "%s-%s-%s" % 'a', 'b', 'c'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: not enough arguments for format string

>>> "%s-%s-%s" % ('a', 'b', 'c')
'a-b-c'

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index d739e558e9e..c79747b2a15 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -76,7 +76,7 @@ def __init__(self, name: str, info, doc, ifcond=None, features=None):
     def __repr__(self):
         if self.name is None:
             return "<%s at 0x%x>" % (type(self).__name__, id(self))
-        return "<%s:%s at 0x%x>" % type(self).__name__, self.name, id(self)
+        return "<%s:%s at 0x%x>" % (type(self).__name__, self.name, id(self))
 
     def c_name(self):
         return c_name(self.name)
-- 
2.41.0

Re: [PATCH 01/19] qapi/schema: fix QAPISchemaEntity.__repr__()

[Philippe Mathieu-Daudé]

On 16/11/23 02:43, John Snow wrote:
> This needs parentheses to work how you want it to:
> 
>>>> "%s-%s-%s" % 'a', 'b', 'c'
> Traceback (most recent call last):
>    File "<stdin>", line 1, in <module>
> TypeError: not enough arguments for format string
> 
>>>> "%s-%s-%s" % ('a', 'b', 'c')
> 'a-b-c'
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   scripts/qapi/schema.py | 2 +-
>   1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index d739e558e9e..c79747b2a15 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -76,7 +76,7 @@ def __init__(self, name: str, info, doc, ifcond=None, features=None):
>       def __repr__(self):
>           if self.name is None:
>               return "<%s at 0x%x>" % (type(self).__name__, id(self))
> -        return "<%s:%s at 0x%x>" % type(self).__name__, self.name, id(self)
> +        return "<%s:%s at 0x%x>" % (type(self).__name__, self.name, id(self))

This is commit 6d133eef98 ("qapi: Fix QAPISchemaEntity.__repr__()").

[PATCH 02/19] qapi/schema: add pylint suppressions

[John Snow]

With this, pylint is happy with the file, so enable it in the
configuration.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/pylintrc  | 5 -----
 scripts/qapi/schema.py | 4 ++++
 2 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index 90546df5345..aafddd3d8d0 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -1,10 +1,5 @@
 [MASTER]
 
-# Add files or directories matching the regex patterns to the ignore list.
-# The regex matches against base names, not paths.
-ignore-patterns=schema.py,
-
-
 [MESSAGES CONTROL]
 
 # Disable the message, report, category or checker with the given id(s). You
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index c79747b2a15..153e703e0ef 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -13,6 +13,7 @@
 # See the COPYING file in the top-level directory.
 
 # TODO catching name collisions in generated code would be nice
+# pylint: disable=too-many-lines
 
 from collections import OrderedDict
 import os
@@ -82,6 +83,7 @@ def c_name(self):
         return c_name(self.name)
 
     def check(self, schema):
+        # pylint: disable=unused-argument
         assert not self._checked
         seen = {}
         for f in self.features:
@@ -116,6 +118,7 @@ def is_implicit(self):
         return not self.info
 
     def visit(self, visitor):
+        # pylint: disable=unused-argument
         assert self._checked
 
     def describe(self):
@@ -134,6 +137,7 @@ def visit_module(self, name):
         pass
 
     def visit_needed(self, entity):
+        # pylint: disable=unused-argument
         # Default to visiting everything
         return True
 
-- 
2.41.0

Re: [PATCH 02/19] qapi/schema: add pylint suppressions

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> With this, pylint is happy with the file, so enable it in the
> configuration.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/pylintrc  | 5 -----
>  scripts/qapi/schema.py | 4 ++++
>  2 files changed, 4 insertions(+), 5 deletions(-)
>
> diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
> index 90546df5345..aafddd3d8d0 100644
> --- a/scripts/qapi/pylintrc
> +++ b/scripts/qapi/pylintrc
> @@ -1,10 +1,5 @@
>  [MASTER]
>  
> -# Add files or directories matching the regex patterns to the ignore list.
> -# The regex matches against base names, not paths.
> -ignore-patterns=schema.py,
> -
> -
>  [MESSAGES CONTROL]
>  
>  # Disable the message, report, category or checker with the given id(s). You
   # can either give multiple identifiers separated by comma (,) or put this
   # option multiple times (only on the command line, not in the configuration
   # file where it should appear only once). You can also use "--disable=all" to
   # disable everything first and then reenable specific checks. For example, if
   # you want to run only the similarities checker, you can use "--disable=all
   # --enable=similarities". If you want to run only the classes checker, but have
   # no Warning level messages displayed, use "--disable=all --enable=classes
   # --disable=W".
   disable=fixme,
           missing-docstring,
           too-many-arguments,
           too-many-branches,
           too-many-statements,
           too-many-instance-attributes,
           consider-using-f-string,
           useless-option-value,

Unrelated, but here goes anyway: sorting these would be nice.

> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index c79747b2a15..153e703e0ef 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -13,6 +13,7 @@
>  # See the COPYING file in the top-level directory.
>  
>  # TODO catching name collisions in generated code would be nice

Blank line to separate unrelated comments, please.

> +# pylint: disable=too-many-lines
>  
>  from collections import OrderedDict
>  import os
> @@ -82,6 +83,7 @@ def c_name(self):
>          return c_name(self.name)
>  
>      def check(self, schema):
> +        # pylint: disable=unused-argument
>          assert not self._checked
>          seen = {}
>          for f in self.features:
> @@ -116,6 +118,7 @@ def is_implicit(self):
>          return not self.info
>  
>      def visit(self, visitor):
> +        # pylint: disable=unused-argument
>          assert self._checked
>  
>      def describe(self):
> @@ -134,6 +137,7 @@ def visit_module(self, name):
>          pass
>  
>      def visit_needed(self, entity):
> +        # pylint: disable=unused-argument
>          # Default to visiting everything
>          return True

Reviewed-by: Markus Armbruster <armbru@redhat.com>

[PATCH 03/19] qapi/schema: name QAPISchemaInclude entities

[John Snow]

It simplifies typing to mandate that entities will always have a name;
to achieve this we can occasionally assign an internal name. This
alleviates errors such as:

qapi/schema.py:287: error: Argument 1 to "__init__" of
"QAPISchemaEntity" has incompatible type "None"; expected "str"
[arg-type]

Trying to fix it the other way by allowing entities to only have
optional names opens up a nightmare portal of whackamole to try and
audit that every other pathway doesn't actually pass a None name when we
expect it to; this is the simpler direction of consitifying the typing.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 153e703e0ef..0fb44452dd5 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -220,7 +220,9 @@ def visit(self, visitor):
 
 class QAPISchemaInclude(QAPISchemaEntity):
     def __init__(self, sub_module, info):
-        super().__init__(None, info, None)
+        # Includes are internal entity objects; and may occur multiple times
+        name = f"q_include_{info.fname}:{info.line}"
+        super().__init__(name, info, None)
         self._sub_module = sub_module
 
     def visit(self, visitor):
-- 
2.41.0

Re: [PATCH 03/19] qapi/schema: name QAPISchemaInclude entities

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> It simplifies typing to mandate that entities will always have a name;
> to achieve this we can occasionally assign an internal name. This
> alleviates errors such as:
>
> qapi/schema.py:287: error: Argument 1 to "__init__" of
> "QAPISchemaEntity" has incompatible type "None"; expected "str"
> [arg-type]
>
> Trying to fix it the other way by allowing entities to only have
> optional names opens up a nightmare portal of whackamole to try and
> audit that every other pathway doesn't actually pass a None name when we
> expect it to; this is the simpler direction of consitifying the typing.

Arguably, that nightmare is compile-time proof of "we are not mistaking
QAPISchemaInclude for a named entity".

When I added the include directive, I shoehorned it into the existing
representation of the QAPI schema as "list of QAPISchemaEntity" by
making it a subtype of QAPISchemaEntity.  That was a somewhat lazy hack.

Note that qapi-code-gen.rst distinguishes between definitions and
directives.

The places where mypy gripes that .name isn't 'str' generally want
something with a name (what qapi-code-gen.rst calls a definition).  If
we somehow pass them an include directive, they'll use None for a name,
which is no good.  mypy is pointing out this problem.

What to do about it?

1. Paper it over: give include directives some made-up name (this
patch).  Now the places use the made-up name instead of None, and mypy
can't see the problem anymore.

2. Assert .name is not None until mypy is happy.  I guess that's what
you called opening "a nightmare portal of whackamole".

3. Clean up the typing: have a type for top-level expression (has no
name), and a subtype for definition (has a name).  Rough sketch
appended.  Thoughts?

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/schema.py | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 153e703e0ef..0fb44452dd5 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -220,7 +220,9 @@ def visit(self, visitor):
>  
>  class QAPISchemaInclude(QAPISchemaEntity):
>      def __init__(self, sub_module, info):
> -        super().__init__(None, info, None)
> +        # Includes are internal entity objects; and may occur multiple times
> +        name = f"q_include_{info.fname}:{info.line}"
> +        super().__init__(name, info, None)
>          self._sub_module = sub_module
>  
>      def visit(self, visitor):

There are two instances of .name is None:

    def __repr__(self) -> str:
        if self.name is None:
            return "<%s at 0x%x>" % (type(self).__name__, id(self))
        return "<%s:%s at 0x%x>" % (type(self).__name__, self.name,
                                    id(self))

and

    def _def_entity(self, ent: QAPISchemaEntity) -> None:
        # Only the predefined types are allowed to not have info
        assert ent.info or self._predefining
        self._entity_list.append(ent)
        if ent.name is None:
            return
        [...]

Don't they need to be updated?



diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 24ad166d52..eaff1df534 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -70,6 +70,45 @@ def is_present(self) -> bool:
 
 
 class QAPISchemaEntity:
+    def __init__(self, info: Optional[QAPISourceInfo]):
+        # For explicitly defined entities, info points to the (explicit)
+        # definition.  For builtins (and their arrays), info is None.
+        # For implicitly defined entities, info points to a place that
+        # triggered the implicit definition (there may be more than one
+        # such place).
+        self.info = info
+        self._module: Optional[QAPISchemaModule] = None
+        self._checked = False
+
+    def __repr__(self) -> str:
+        return "<%s at 0x%x>" % (type(self).__name__, id(self))
+
+    def check(self, schema: QAPISchema) -> None:
+        self._checked = True
+
+    def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
+        pass
+
+    def check_doc(self) -> None:
+        pass
+
+    def _set_module(
+        self, schema: QAPISchema, info: Optional[QAPISourceInfo]
+    ) -> None:
+        assert self._checked
+        fname = info.fname if info else QAPISchemaModule.BUILTIN_MODULE_NAME
+        self._module = schema.module_by_fname(fname)
+        self._module.add_entity(self)
+
+    def set_module(self, schema: QAPISchema) -> None:
+        self._set_module(schema, self.info)
+
+    def visit(self, visitor: QAPISchemaVisitor) -> None:
+        # pylint: disable=unused-argument
+        assert self._checked
+
+
+class QAPISchemaDefinition(QAPISchemaEntity):
     meta: str
 
     def __init__(
@@ -80,24 +119,16 @@ def __init__(
         ifcond: Optional[QAPISchemaIfCond] = None,
         features: Optional[List[QAPISchemaFeature]] = None,
     ):
+        super().__init__(info)
         for f in features or []:
             f.set_defined_in(name)
         self.name = name
-        self._module: Optional[QAPISchemaModule] = None
-        # For explicitly defined entities, info points to the (explicit)
-        # definition.  For builtins (and their arrays), info is None.
-        # For implicitly defined entities, info points to a place that
-        # triggered the implicit definition (there may be more than one
-        # such place).
-        self.info = info
         self.doc = doc
         self._ifcond = ifcond or QAPISchemaIfCond()
         self.features = features or []
         self._checked = False
-
+        
     def __repr__(self) -> str:
-        if self.name is None:
-            return "<%s at 0x%x>" % (type(self).__name__, id(self))
         return "<%s:%s at 0x%x>" % (type(self).__name__, self.name,
                                     id(self))
 
@@ -122,17 +153,6 @@ def check_doc(self) -> None:
         if self.doc:
             self.doc.check()
 
-    def _set_module(
-        self, schema: QAPISchema, info: Optional[QAPISourceInfo]
-    ) -> None:
-        assert self._checked
-        fname = info.fname if info else QAPISchemaModule.BUILTIN_MODULE_NAME
-        self._module = schema.module_by_fname(fname)
-        self._module.add_entity(self)
-
-    def set_module(self, schema: QAPISchema) -> None:
-        self._set_module(schema, self.info)
-
     @property
     def ifcond(self) -> QAPISchemaIfCond:
         assert self._checked
@@ -141,10 +161,6 @@ def ifcond(self) -> QAPISchemaIfCond:
     def is_implicit(self) -> bool:
         return not self.info
 
-    def visit(self, visitor: QAPISchemaVisitor) -> None:
-        # pylint: disable=unused-argument
-        assert self._checked
-
     def describe(self) -> str:
         return "%s '%s'" % (self.meta, self.name)
 
@@ -301,9 +317,7 @@ def visit(self, visitor: QAPISchemaVisitor) -> None:
 
 class QAPISchemaInclude(QAPISchemaEntity):
     def __init__(self, sub_module: QAPISchemaModule, info: QAPISourceInfo):
-        # Includes are internal entity objects; and may occur multiple times
-        name = f"q_include_{info.fname}:{info.line}"
-        super().__init__(name, info, None)
+        super().__init__(info)
         self._sub_module = sub_module
 
     def visit(self, visitor: QAPISchemaVisitor) -> None:
@@ -311,7 +325,7 @@ def visit(self, visitor: QAPISchemaVisitor) -> None:
         visitor.visit_include(self._sub_module.name, self.info)
 
 
-class QAPISchemaType(QAPISchemaEntity):
+class QAPISchemaType(QAPISchemaDefinition):
     # Return the C type for common use.
     # For the types we commonly box, this is a pointer type.
     def c_type(self) -> str:
@@ -977,7 +991,7 @@ def __init__(
         super().__init__(name, info, typ, False, ifcond)
 
 
-class QAPISchemaCommand(QAPISchemaEntity):
+class QAPISchemaCommand(QAPISchemaDefinition):
     meta = 'command'
 
     def __init__(
@@ -1059,7 +1073,7 @@ def visit(self, visitor: QAPISchemaVisitor) -> None:
             self.coroutine)
 
 
-class QAPISchemaEvent(QAPISchemaEntity):
+class QAPISchemaEvent(QAPISchemaDefinition):
     meta = 'event'
 
     def __init__(
@@ -1133,7 +1147,7 @@ def __init__(self, fname: str):
         exprs = check_exprs(parser.exprs)
         self.docs = parser.docs
         self._entity_list: List[QAPISchemaEntity] = []
-        self._entity_dict: Dict[str, QAPISchemaEntity] = {}
+        self._entity_dict: Dict[str, QAPISchemaDefinition] = {}
         self._module_dict: Dict[str, QAPISchemaModule] = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
         self._make_module(QAPISchemaModule.BUILTIN_MODULE_NAME)
@@ -1145,9 +1159,12 @@ def __init__(self, fname: str):
         self.check()
 
     def _def_entity(self, ent: QAPISchemaEntity) -> None:
+        self._entity_list.append(ent)
+
+    def _def_definition(self, ent: QAPISchemaDefinition) -> None:
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
-        self._entity_list.append(ent)
+        self._def_entity(ent)
         if ent.name is None:
             return
         # TODO reject names that differ only in '_' vs. '.'  vs. '-',
@@ -1163,7 +1180,7 @@ def _def_entity(self, ent: QAPISchemaEntity) -> None:
                 ent.info, "%s is already defined" % other_ent.describe())
         self._entity_dict[ent.name] = ent
 
-    def get_entity(self, name: str) -> Optional[QAPISchemaEntity]:
+    def get_definition(self, name: str) -> Optional[QAPISchemaDefinition]:
         return self._entity_dict.get(name)
 
     def get_typed_entity(
@@ -1171,7 +1188,7 @@ def get_typed_entity(
         name: str,
         typ: Type[_EntityType]
     ) -> Optional[_EntityType]:
-        ent = self.get_entity(name)
+        ent = self.get_definition(name)
         if ent is not None and not isinstance(ent, typ):
             etype = type(ent).__name__
             ttype = typ.__name__
@@ -1225,7 +1242,7 @@ def _def_include(self, expr: QAPIExpression) -> None:
     def _def_builtin_type(
         self, name: str, json_type: str, c_type: str
     ) -> None:
-        self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type))
+        self._def_definition(QAPISchemaBuiltinType(name, json_type, c_type))
         # Instantiating only the arrays that are actually used would
         # be nice, but we can't as long as their generated code
         # (qapi-builtin-types.[ch]) may be shared by some other
@@ -1251,14 +1268,14 @@ def _def_predefineds(self) -> None:
             self._def_builtin_type(*t)
         self.the_empty_object_type = QAPISchemaObjectType(
             'q_empty', None, None, None, None, None, [], None)
-        self._def_entity(self.the_empty_object_type)
+        self._def_definition(self.the_empty_object_type)
 
         qtypes = ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist',
                   'qbool']
         qtype_values = self._make_enum_members(
             [{'name': n} for n in qtypes], None)
 
-        self._def_entity(QAPISchemaEnumType('QType', None, None, None, None,
+        self._def_definition(QAPISchemaEnumType('QType', None, None, None, None,
                                             qtype_values, 'QTYPE'))
 
     def _make_features(
@@ -1294,8 +1311,8 @@ def _make_array_type(
         self, element_type: str, info: Optional[QAPISourceInfo]
     ) -> str:
         name = element_type + 'List'    # reserved by check_defn_name_str()
-        if not self.get_entity(name):
-            self._def_entity(QAPISchemaArrayType(name, info, element_type))
+        if not self.get_definition(name):
+            self._def_definition(QAPISchemaArrayType(name, info, element_type))
         return name
 
     def _make_implicit_object_type(
@@ -1317,7 +1334,7 @@ def _make_implicit_object_type(
             # later.
             pass
         else:
-            self._def_entity(QAPISchemaObjectType(
+            self._def_definition(QAPISchemaObjectType(
                 name, info, None, ifcond, None, None, members, None))
         return name
 
@@ -1328,7 +1345,7 @@ def _def_enum_type(self, expr: QAPIExpression) -> None:
         ifcond = QAPISchemaIfCond(expr.get('if'))
         info = expr.info
         features = self._make_features(expr.get('features'), info)
-        self._def_entity(QAPISchemaEnumType(
+        self._def_definition(QAPISchemaEnumType(
             name, info, expr.doc, ifcond, features,
             self._make_enum_members(data, info), prefix))
 
@@ -1367,7 +1384,7 @@ def _def_struct_type(self, expr: QAPIExpression) -> None:
         info = expr.info
         ifcond = QAPISchemaIfCond(expr.get('if'))
         features = self._make_features(expr.get('features'), info)
-        self._def_entity(QAPISchemaObjectType(
+        self._def_definition(QAPISchemaObjectType(
             name, info, expr.doc, ifcond, features, base,
             self._make_members(data, info),
             None))
@@ -1403,7 +1420,7 @@ def _def_union_type(self, expr: QAPIExpression) -> None:
                                info)
             for (key, value) in data.items()]
         members: List[QAPISchemaObjectTypeMember] = []
-        self._def_entity(
+        self._def_definition(
             QAPISchemaObjectType(name, info, expr.doc, ifcond, features,
                                  base, members,
                                  QAPISchemaVariants(
@@ -1422,7 +1439,7 @@ def _def_alternate_type(self, expr: QAPIExpression) -> None:
                                info)
             for (key, value) in data.items()]
         tag_member = QAPISchemaObjectTypeMember('type', info, 'QType', False)
-        self._def_entity(
+        self._def_definition(
             QAPISchemaAlternateType(
                 name, info, expr.doc, ifcond, features,
                 QAPISchemaVariants(None, info, tag_member, variants)))
@@ -1447,7 +1464,7 @@ def _def_command(self, expr: QAPIExpression) -> None:
         if isinstance(rets, list):
             assert len(rets) == 1
             rets = self._make_array_type(rets[0], info)
-        self._def_entity(QAPISchemaCommand(name, info, expr.doc, ifcond,
+        self._def_definition(QAPISchemaCommand(name, info, expr.doc, ifcond,
                                            features, data, rets,
                                            gen, success_response,
                                            boxed, allow_oob, allow_preconfig,
@@ -1464,7 +1481,7 @@ def _def_event(self, expr: QAPIExpression) -> None:
             data = self._make_implicit_object_type(
                 name, info, ifcond,
                 'arg', self._make_members(data, info))
-        self._def_entity(QAPISchemaEvent(name, info, expr.doc, ifcond,
+        self._def_definition(QAPISchemaEvent(name, info, expr.doc, ifcond,
                                          features, data, boxed))
 
     def _def_exprs(self, exprs: List[QAPIExpression]) -> None:

Re: [PATCH 03/19] qapi/schema: name QAPISchemaInclude entities

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 16214 bytes --]

On Tue, Nov 21, 2023, 8:33 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > It simplifies typing to mandate that entities will always have a name;
> > to achieve this we can occasionally assign an internal name. This
> > alleviates errors such as:
> >
> > qapi/schema.py:287: error: Argument 1 to "__init__" of
> > "QAPISchemaEntity" has incompatible type "None"; expected "str"
> > [arg-type]
> >
> > Trying to fix it the other way by allowing entities to only have
> > optional names opens up a nightmare portal of whackamole to try and
> > audit that every other pathway doesn't actually pass a None name when we
> > expect it to; this is the simpler direction of consitifying the typing.
>
> Arguably, that nightmare is compile-time proof of "we are not mistaking
> QAPISchemaInclude for a named entity".
>
> When I added the include directive, I shoehorned it into the existing
> representation of the QAPI schema as "list of QAPISchemaEntity" by
> making it a subtype of QAPISchemaEntity.  That was a somewhat lazy hack.
>
> Note that qapi-code-gen.rst distinguishes between definitions and
> directives.
>
> The places where mypy gripes that .name isn't 'str' generally want
> something with a name (what qapi-code-gen.rst calls a definition).  If
> we somehow pass them an include directive, they'll use None for a name,
> which is no good.  mypy is pointing out this problem.
>
> What to do about it?
>
> 1. Paper it over: give include directives some made-up name (this
> patch).  Now the places use the made-up name instead of None, and mypy
> can't see the problem anymore.
>
> 2. Assert .name is not None until mypy is happy.  I guess that's what
> you called opening "a nightmare portal of whackamole".
>

Yep.


> 3. Clean up the typing: have a type for top-level expression (has no
> name), and a subtype for definition (has a name).  Rough sketch
> appended.  Thoughts?
>

Oh, that'll work. I tried to keep to "minimal SLOC" but where you want to
see deeper fixes, I'm happy to deviate. I'll give it a shot.


> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  scripts/qapi/schema.py | 4 +++-
> >  1 file changed, 3 insertions(+), 1 deletion(-)
> >
> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> > index 153e703e0ef..0fb44452dd5 100644
> > --- a/scripts/qapi/schema.py
> > +++ b/scripts/qapi/schema.py
> > @@ -220,7 +220,9 @@ def visit(self, visitor):
> >
> >  class QAPISchemaInclude(QAPISchemaEntity):
> >      def __init__(self, sub_module, info):
> > -        super().__init__(None, info, None)
> > +        # Includes are internal entity objects; and may occur multiple
> times
> > +        name = f"q_include_{info.fname}:{info.line}"
> > +        super().__init__(name, info, None)
> >          self._sub_module = sub_module
> >
> >      def visit(self, visitor):
>
> There are two instances of .name is None:
>
>     def __repr__(self) -> str:
>         if self.name is None:
>             return "<%s at 0x%x>" % (type(self).__name__, id(self))
>         return "<%s:%s at 0x%x>" % (type(self).__name__, self.name,
>                                     id(self))
>
> and
>
>     def _def_entity(self, ent: QAPISchemaEntity) -> None:
>         # Only the predefined types are allowed to not have info
>         assert ent.info or self._predefining
>         self._entity_list.append(ent)
>         if ent.name is None:
>             return
>         [...]
>
> Don't they need to be updated?
>

Oh, yes. I remove some assertions later in the series but I very likely
missed these if cases.


>
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 24ad166d52..eaff1df534 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -70,6 +70,45 @@ def is_present(self) -> bool:
>
>
>  class QAPISchemaEntity:
> +    def __init__(self, info: Optional[QAPISourceInfo]):
> +        # For explicitly defined entities, info points to the (explicit)
> +        # definition.  For builtins (and their arrays), info is None.
> +        # For implicitly defined entities, info points to a place that
> +        # triggered the implicit definition (there may be more than one
> +        # such place).
> +        self.info = info
> +        self._module: Optional[QAPISchemaModule] = None
> +        self._checked = False
> +
> +    def __repr__(self) -> str:
> +        return "<%s at 0x%x>" % (type(self).__name__, id(self))
> +
> +    def check(self, schema: QAPISchema) -> None:
> +        self._checked = True
> +
> +    def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
> +        pass
> +
> +    def check_doc(self) -> None:
> +        pass
> +
> +    def _set_module(
> +        self, schema: QAPISchema, info: Optional[QAPISourceInfo]
> +    ) -> None:
> +        assert self._checked
> +        fname = info.fname if info else
> QAPISchemaModule.BUILTIN_MODULE_NAME
> +        self._module = schema.module_by_fname(fname)
> +        self._module.add_entity(self)
> +
> +    def set_module(self, schema: QAPISchema) -> None:
> +        self._set_module(schema, self.info)
> +
> +    def visit(self, visitor: QAPISchemaVisitor) -> None:
> +        # pylint: disable=unused-argument
> +        assert self._checked
> +
> +
> +class QAPISchemaDefinition(QAPISchemaEntity):
>      meta: str
>
>      def __init__(
> @@ -80,24 +119,16 @@ def __init__(
>          ifcond: Optional[QAPISchemaIfCond] = None,
>          features: Optional[List[QAPISchemaFeature]] = None,
>      ):
> +        super().__init__(info)
>          for f in features or []:
>              f.set_defined_in(name)
>          self.name = name
> -        self._module: Optional[QAPISchemaModule] = None
> -        # For explicitly defined entities, info points to the (explicit)
> -        # definition.  For builtins (and their arrays), info is None.
> -        # For implicitly defined entities, info points to a place that
> -        # triggered the implicit definition (there may be more than one
> -        # such place).
> -        self.info = info
>          self.doc = doc
>          self._ifcond = ifcond or QAPISchemaIfCond()
>          self.features = features or []
>          self._checked = False
> -
> +
>      def __repr__(self) -> str:
> -        if self.name is None:
> -            return "<%s at 0x%x>" % (type(self).__name__, id(self))
>          return "<%s:%s at 0x%x>" % (type(self).__name__, self.name,
>                                      id(self))
>
> @@ -122,17 +153,6 @@ def check_doc(self) -> None:
>          if self.doc:
>              self.doc.check()
>
> -    def _set_module(
> -        self, schema: QAPISchema, info: Optional[QAPISourceInfo]
> -    ) -> None:
> -        assert self._checked
> -        fname = info.fname if info else
> QAPISchemaModule.BUILTIN_MODULE_NAME
> -        self._module = schema.module_by_fname(fname)
> -        self._module.add_entity(self)
> -
> -    def set_module(self, schema: QAPISchema) -> None:
> -        self._set_module(schema, self.info)
> -
>      @property
>      def ifcond(self) -> QAPISchemaIfCond:
>          assert self._checked
> @@ -141,10 +161,6 @@ def ifcond(self) -> QAPISchemaIfCond:
>      def is_implicit(self) -> bool:
>          return not self.info
>
> -    def visit(self, visitor: QAPISchemaVisitor) -> None:
> -        # pylint: disable=unused-argument
> -        assert self._checked
> -
>      def describe(self) -> str:
>          return "%s '%s'" % (self.meta, self.name)
>
> @@ -301,9 +317,7 @@ def visit(self, visitor: QAPISchemaVisitor) -> None:
>
>  class QAPISchemaInclude(QAPISchemaEntity):
>      def __init__(self, sub_module: QAPISchemaModule, info:
> QAPISourceInfo):
> -        # Includes are internal entity objects; and may occur multiple
> times
> -        name = f"q_include_{info.fname}:{info.line}"
> -        super().__init__(name, info, None)
> +        super().__init__(info)
>          self._sub_module = sub_module
>
>      def visit(self, visitor: QAPISchemaVisitor) -> None:
> @@ -311,7 +325,7 @@ def visit(self, visitor: QAPISchemaVisitor) -> None:
>          visitor.visit_include(self._sub_module.name, self.info)
>
>
> -class QAPISchemaType(QAPISchemaEntity):
> +class QAPISchemaType(QAPISchemaDefinition):
>      # Return the C type for common use.
>      # For the types we commonly box, this is a pointer type.
>      def c_type(self) -> str:
> @@ -977,7 +991,7 @@ def __init__(
>          super().__init__(name, info, typ, False, ifcond)
>
>
> -class QAPISchemaCommand(QAPISchemaEntity):
> +class QAPISchemaCommand(QAPISchemaDefinition):
>      meta = 'command'
>
>      def __init__(
> @@ -1059,7 +1073,7 @@ def visit(self, visitor: QAPISchemaVisitor) -> None:
>              self.coroutine)
>
>
> -class QAPISchemaEvent(QAPISchemaEntity):
> +class QAPISchemaEvent(QAPISchemaDefinition):
>      meta = 'event'
>
>      def __init__(
> @@ -1133,7 +1147,7 @@ def __init__(self, fname: str):
>          exprs = check_exprs(parser.exprs)
>          self.docs = parser.docs
>          self._entity_list: List[QAPISchemaEntity] = []
> -        self._entity_dict: Dict[str, QAPISchemaEntity] = {}
> +        self._entity_dict: Dict[str, QAPISchemaDefinition] = {}
>          self._module_dict: Dict[str, QAPISchemaModule] = OrderedDict()
>          self._schema_dir = os.path.dirname(fname)
>          self._make_module(QAPISchemaModule.BUILTIN_MODULE_NAME)
> @@ -1145,9 +1159,12 @@ def __init__(self, fname: str):
>          self.check()
>
>      def _def_entity(self, ent: QAPISchemaEntity) -> None:
> +        self._entity_list.append(ent)
> +
> +    def _def_definition(self, ent: QAPISchemaDefinition) -> None:
>          # Only the predefined types are allowed to not have info
>          assert ent.info or self._predefining
> -        self._entity_list.append(ent)
> +        self._def_entity(ent)
>          if ent.name is None:
>              return
>          # TODO reject names that differ only in '_' vs. '.'  vs. '-',
> @@ -1163,7 +1180,7 @@ def _def_entity(self, ent: QAPISchemaEntity) -> None:
>                  ent.info, "%s is already defined" % other_ent.describe())
>          self._entity_dict[ent.name] = ent
>
> -    def get_entity(self, name: str) -> Optional[QAPISchemaEntity]:
> +    def get_definition(self, name: str) -> Optional[QAPISchemaDefinition]:
>          return self._entity_dict.get(name)
>
>      def get_typed_entity(
> @@ -1171,7 +1188,7 @@ def get_typed_entity(
>          name: str,
>          typ: Type[_EntityType]
>      ) -> Optional[_EntityType]:
> -        ent = self.get_entity(name)
> +        ent = self.get_definition(name)
>          if ent is not None and not isinstance(ent, typ):
>              etype = type(ent).__name__
>              ttype = typ.__name__
> @@ -1225,7 +1242,7 @@ def _def_include(self, expr: QAPIExpression) -> None:
>      def _def_builtin_type(
>          self, name: str, json_type: str, c_type: str
>      ) -> None:
> -        self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type))
> +        self._def_definition(QAPISchemaBuiltinType(name, json_type,
> c_type))
>          # Instantiating only the arrays that are actually used would
>          # be nice, but we can't as long as their generated code
>          # (qapi-builtin-types.[ch]) may be shared by some other
> @@ -1251,14 +1268,14 @@ def _def_predefineds(self) -> None:
>              self._def_builtin_type(*t)
>          self.the_empty_object_type = QAPISchemaObjectType(
>              'q_empty', None, None, None, None, None, [], None)
> -        self._def_entity(self.the_empty_object_type)
> +        self._def_definition(self.the_empty_object_type)
>
>          qtypes = ['none', 'qnull', 'qnum', 'qstring', 'qdict', 'qlist',
>                    'qbool']
>          qtype_values = self._make_enum_members(
>              [{'name': n} for n in qtypes], None)
>
> -        self._def_entity(QAPISchemaEnumType('QType', None, None, None,
> None,
> +        self._def_definition(QAPISchemaEnumType('QType', None, None,
> None, None,
>                                              qtype_values, 'QTYPE'))
>
>      def _make_features(
> @@ -1294,8 +1311,8 @@ def _make_array_type(
>          self, element_type: str, info: Optional[QAPISourceInfo]
>      ) -> str:
>          name = element_type + 'List'    # reserved by
> check_defn_name_str()
> -        if not self.get_entity(name):
> -            self._def_entity(QAPISchemaArrayType(name, info,
> element_type))
> +        if not self.get_definition(name):
> +            self._def_definition(QAPISchemaArrayType(name, info,
> element_type))
>          return name
>
>      def _make_implicit_object_type(
> @@ -1317,7 +1334,7 @@ def _make_implicit_object_type(
>              # later.
>              pass
>          else:
> -            self._def_entity(QAPISchemaObjectType(
> +            self._def_definition(QAPISchemaObjectType(
>                  name, info, None, ifcond, None, None, members, None))
>          return name
>
> @@ -1328,7 +1345,7 @@ def _def_enum_type(self, expr: QAPIExpression) ->
> None:
>          ifcond = QAPISchemaIfCond(expr.get('if'))
>          info = expr.info
>          features = self._make_features(expr.get('features'), info)
> -        self._def_entity(QAPISchemaEnumType(
> +        self._def_definition(QAPISchemaEnumType(
>              name, info, expr.doc, ifcond, features,
>              self._make_enum_members(data, info), prefix))
>
> @@ -1367,7 +1384,7 @@ def _def_struct_type(self, expr: QAPIExpression) ->
> None:
>          info = expr.info
>          ifcond = QAPISchemaIfCond(expr.get('if'))
>          features = self._make_features(expr.get('features'), info)
> -        self._def_entity(QAPISchemaObjectType(
> +        self._def_definition(QAPISchemaObjectType(
>              name, info, expr.doc, ifcond, features, base,
>              self._make_members(data, info),
>              None))
> @@ -1403,7 +1420,7 @@ def _def_union_type(self, expr: QAPIExpression) ->
> None:
>                                 info)
>              for (key, value) in data.items()]
>          members: List[QAPISchemaObjectTypeMember] = []
> -        self._def_entity(
> +        self._def_definition(
>              QAPISchemaObjectType(name, info, expr.doc, ifcond, features,
>                                   base, members,
>                                   QAPISchemaVariants(
> @@ -1422,7 +1439,7 @@ def _def_alternate_type(self, expr: QAPIExpression)
> -> None:
>                                 info)
>              for (key, value) in data.items()]
>          tag_member = QAPISchemaObjectTypeMember('type', info, 'QType',
> False)
> -        self._def_entity(
> +        self._def_definition(
>              QAPISchemaAlternateType(
>                  name, info, expr.doc, ifcond, features,
>                  QAPISchemaVariants(None, info, tag_member, variants)))
> @@ -1447,7 +1464,7 @@ def _def_command(self, expr: QAPIExpression) -> None:
>          if isinstance(rets, list):
>              assert len(rets) == 1
>              rets = self._make_array_type(rets[0], info)
> -        self._def_entity(QAPISchemaCommand(name, info, expr.doc, ifcond,
> +        self._def_definition(QAPISchemaCommand(name, info, expr.doc,
> ifcond,
>                                             features, data, rets,
>                                             gen, success_response,
>                                             boxed, allow_oob,
> allow_preconfig,
> @@ -1464,7 +1481,7 @@ def _def_event(self, expr: QAPIExpression) -> None:
>              data = self._make_implicit_object_type(
>                  name, info, ifcond,
>                  'arg', self._make_members(data, info))
> -        self._def_entity(QAPISchemaEvent(name, info, expr.doc, ifcond,
> +        self._def_definition(QAPISchemaEvent(name, info, expr.doc, ifcond,
>                                           features, data, boxed))
>
>      def _def_exprs(self, exprs: List[QAPIExpression]) -> None:
>

I'll try the refactor out in a patch at the end of my series and see how
feasible it is.

(I haven't reviewed it deeply yet, so if there's an obvious problem I'll
find it when I go to implement this. conceptually it seems fine.)

>

[-- Attachment #2: Type: text/html, Size: 22220 bytes --]

Re: [PATCH 03/19] qapi/schema: name QAPISchemaInclude entities

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Tue, Nov 21, 2023, 8:33 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > It simplifies typing to mandate that entities will always have a name;
>> > to achieve this we can occasionally assign an internal name. This
>> > alleviates errors such as:
>> >
>> > qapi/schema.py:287: error: Argument 1 to "__init__" of
>> > "QAPISchemaEntity" has incompatible type "None"; expected "str"
>> > [arg-type]
>> >
>> > Trying to fix it the other way by allowing entities to only have
>> > optional names opens up a nightmare portal of whackamole to try and
>> > audit that every other pathway doesn't actually pass a None name when we
>> > expect it to; this is the simpler direction of consitifying the typing.
>>
>> Arguably, that nightmare is compile-time proof of "we are not mistaking
>> QAPISchemaInclude for a named entity".
>>
>> When I added the include directive, I shoehorned it into the existing
>> representation of the QAPI schema as "list of QAPISchemaEntity" by
>> making it a subtype of QAPISchemaEntity.  That was a somewhat lazy hack.
>>
>> Note that qapi-code-gen.rst distinguishes between definitions and
>> directives.
>>
>> The places where mypy gripes that .name isn't 'str' generally want
>> something with a name (what qapi-code-gen.rst calls a definition).  If
>> we somehow pass them an include directive, they'll use None for a name,
>> which is no good.  mypy is pointing out this problem.
>>
>> What to do about it?
>>
>> 1. Paper it over: give include directives some made-up name (this
>> patch).  Now the places use the made-up name instead of None, and mypy
>> can't see the problem anymore.
>>
>> 2. Assert .name is not None until mypy is happy.  I guess that's what
>> you called opening "a nightmare portal of whackamole".
>>
>
> Yep.
>
>
>> 3. Clean up the typing: have a type for top-level expression (has no
>> name), and a subtype for definition (has a name).  Rough sketch
>> appended.  Thoughts?
>>
>
> Oh, that'll work. I tried to keep to "minimal SLOC" but where you want to
> see deeper fixes, I'm happy to deviate. I'll give it a shot.

I do appreciate the minimal fix!  I *like* exploring "minimal" first.
In this case, the exploration led me to not like my lazy hack anymore :)

[...]

> I'll try the refactor out in a patch at the end of my series and see how
> feasible it is.
>
> (I haven't reviewed it deeply yet, so if there's an obvious problem I'll
> find it when I go to implement this. conceptually it seems fine.)

Thanks!

Re: [PATCH 03/19] qapi/schema: name QAPISchemaInclude entities

[John Snow]

On Wed, Nov 22, 2023 at 4:38 AM Markus Armbruster <armbru@redhat.com> wrote:
>
> John Snow <jsnow@redhat.com> writes:
>
> > On Tue, Nov 21, 2023, 8:33 AM Markus Armbruster <armbru@redhat.com> wrote:
> >
> >> John Snow <jsnow@redhat.com> writes:
> >>
> >> > It simplifies typing to mandate that entities will always have a name;
> >> > to achieve this we can occasionally assign an internal name. This
> >> > alleviates errors such as:
> >> >
> >> > qapi/schema.py:287: error: Argument 1 to "__init__" of
> >> > "QAPISchemaEntity" has incompatible type "None"; expected "str"
> >> > [arg-type]
> >> >
> >> > Trying to fix it the other way by allowing entities to only have
> >> > optional names opens up a nightmare portal of whackamole to try and
> >> > audit that every other pathway doesn't actually pass a None name when we
> >> > expect it to; this is the simpler direction of consitifying the typing.
> >>
> >> Arguably, that nightmare is compile-time proof of "we are not mistaking
> >> QAPISchemaInclude for a named entity".
> >>
> >> When I added the include directive, I shoehorned it into the existing
> >> representation of the QAPI schema as "list of QAPISchemaEntity" by
> >> making it a subtype of QAPISchemaEntity.  That was a somewhat lazy hack.
> >>
> >> Note that qapi-code-gen.rst distinguishes between definitions and
> >> directives.
> >>
> >> The places where mypy gripes that .name isn't 'str' generally want
> >> something with a name (what qapi-code-gen.rst calls a definition).  If
> >> we somehow pass them an include directive, they'll use None for a name,
> >> which is no good.  mypy is pointing out this problem.
> >>
> >> What to do about it?
> >>
> >> 1. Paper it over: give include directives some made-up name (this
> >> patch).  Now the places use the made-up name instead of None, and mypy
> >> can't see the problem anymore.
> >>
> >> 2. Assert .name is not None until mypy is happy.  I guess that's what
> >> you called opening "a nightmare portal of whackamole".
> >>
> >
> > Yep.
> >
> >
> >> 3. Clean up the typing: have a type for top-level expression (has no
> >> name), and a subtype for definition (has a name).  Rough sketch
> >> appended.  Thoughts?
> >>
> >
> > Oh, that'll work. I tried to keep to "minimal SLOC" but where you want to
> > see deeper fixes, I'm happy to deviate. I'll give it a shot.
>
> I do appreciate the minimal fix!  I *like* exploring "minimal" first.
> In this case, the exploration led me to not like my lazy hack anymore :)
>
> [...]
>
> > I'll try the refactor out in a patch at the end of my series and see how
> > feasible it is.
> >
> > (I haven't reviewed it deeply yet, so if there's an obvious problem I'll
> > find it when I go to implement this. conceptually it seems fine.)
>
> Thanks!
>

Got this working with minor changes to sphinx's qapidox extension,
which isn't typechecked so it escaped notice. I'm going to continue
working through your feedback as-is, then when I get to the end of the
series, I'll attempt to float this patch forward to replace this patch
under discussion.

Thanks!

--js

[PATCH 04/19] qapi/schema: declare type for QAPISchemaObjectTypeMember.type

[John Snow]

declare, but don't initialize the type of "type" to be QAPISchemaType -
and allow the value to be initialized during check().

This avoids the need for several "assert type is not None" statements
littered throughout the code by asserting it "will always be set."

It's a little hokey, but it works -- at the expense of slightly
incorrect type information before check() is called, anyway. If this
field is accessed before it is initialized, you'll be treated to an
AttributeError exception.

Fixes stuff like this:

qapi/schema.py:657: error: "None" has no attribute "alternate_qtype"  [attr-defined]
qapi/schema.py:662: error: "None" has no attribute "describe"  [attr-defined]

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 0fb44452dd5..c5fdd625452 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -771,7 +771,7 @@ def __init__(self, name, info, typ, optional, ifcond=None, features=None):
             assert isinstance(f, QAPISchemaFeature)
             f.set_defined_in(name)
         self._type_name = typ
-        self.type = None
+        self.type: QAPISchemaType  # set during check(). Kind of hokey.
         self.optional = optional
         self.features = features or []
 
-- 
2.41.0

[PATCH 05/19] qapi/schema: make c_type() and json_type() abstract methods

[John Snow]

These methods should always return a str, it's only the default abstract
implementation that doesn't. They can be marked "abstract" by raising
NotImplementedError(), which requires subclasses to override the method
with the proper return type.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index c5fdd625452..4600a566005 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -233,8 +233,8 @@ def visit(self, visitor):
 class QAPISchemaType(QAPISchemaEntity):
     # Return the C type for common use.
     # For the types we commonly box, this is a pointer type.
-    def c_type(self):
-        pass
+    def c_type(self) -> str:
+        raise NotImplementedError()
 
     # Return the C type to be used in a parameter list.
     def c_param_type(self):
@@ -244,8 +244,8 @@ def c_param_type(self):
     def c_unboxed_type(self):
         return self.c_type()
 
-    def json_type(self):
-        pass
+    def json_type(self) -> str:
+        raise NotImplementedError()
 
     def alternate_qtype(self):
         json2qtype = {
-- 
2.41.0

Re: [PATCH 05/19] qapi/schema: make c_type() and json_type() abstract methods

[Philippe Mathieu-Daudé]

On 16/11/23 02:43, John Snow wrote:
> These methods should always return a str, it's only the default abstract
> implementation that doesn't. They can be marked "abstract" by raising
> NotImplementedError(), which requires subclasses to override the method
> with the proper return type.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   scripts/qapi/schema.py | 8 ++++----
>   1 file changed, 4 insertions(+), 4 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH 05/19] qapi/schema: make c_type() and json_type() abstract methods

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> These methods should always return a str, it's only the default abstract
> implementation that doesn't. They can be marked "abstract" by raising
> NotImplementedError(), which requires subclasses to override the method
> with the proper return type.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/schema.py | 8 ++++----
>  1 file changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index c5fdd625452..4600a566005 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -233,8 +233,8 @@ def visit(self, visitor):
>  class QAPISchemaType(QAPISchemaEntity):
>      # Return the C type for common use.
>      # For the types we commonly box, this is a pointer type.
> -    def c_type(self):
> -        pass
> +    def c_type(self) -> str:
> +        raise NotImplementedError()
>  
>      # Return the C type to be used in a parameter list.
>      def c_param_type(self):
> @@ -244,8 +244,8 @@ def c_param_type(self):
>      def c_unboxed_type(self):
>          return self.c_type()
>  
> -    def json_type(self):
> -        pass
> +    def json_type(self) -> str:
> +        raise NotImplementedError()
>  
>      def alternate_qtype(self):
>          json2qtype = {

I wish abstract methods could be done in a more concise way.

Re: [PATCH 05/19] qapi/schema: make c_type() and json_type() abstract methods

[Daniel P. Berrangé]

On Tue, Nov 21, 2023 at 02:36:54PM +0100, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
> > These methods should always return a str, it's only the default abstract
> > implementation that doesn't. They can be marked "abstract" by raising
> > NotImplementedError(), which requires subclasses to override the method
> > with the proper return type.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  scripts/qapi/schema.py | 8 ++++----
> >  1 file changed, 4 insertions(+), 4 deletions(-)
> >
> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> > index c5fdd625452..4600a566005 100644
> > --- a/scripts/qapi/schema.py
> > +++ b/scripts/qapi/schema.py
> > @@ -233,8 +233,8 @@ def visit(self, visitor):
> >  class QAPISchemaType(QAPISchemaEntity):
> >      # Return the C type for common use.
> >      # For the types we commonly box, this is a pointer type.
> > -    def c_type(self):
> > -        pass
> > +    def c_type(self) -> str:
> > +        raise NotImplementedError()
> >  
> >      # Return the C type to be used in a parameter list.
> >      def c_param_type(self):
> > @@ -244,8 +244,8 @@ def c_param_type(self):
> >      def c_unboxed_type(self):
> >          return self.c_type()
> >  
> > -    def json_type(self):
> > -        pass
> > +    def json_type(self) -> str:
> > +        raise NotImplementedError()
> >  
> >      def alternate_qtype(self):
> >          json2qtype = {
> 
> I wish abstract methods could be done in a more concise way.

The canonical way would be to use the 'abc' module:

  from abc import ABCMeta, abstractmethod

  class A(metaclass=ABCMeta):
      @abstractmethod
      def foo(self): pass

Not sure if the @abstractmethod decorator is enough to keep the static
typing checker happy about a 'str' return type, when there is no body
impl

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 05/19] qapi/schema: make c_type() and json_type() abstract methods

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2843 bytes --]

On Tue, Nov 21, 2023, 8:43 AM Daniel P. Berrangé <berrange@redhat.com>
wrote:

> On Tue, Nov 21, 2023 at 02:36:54PM +0100, Markus Armbruster wrote:
> > John Snow <jsnow@redhat.com> writes:
> >
> > > These methods should always return a str, it's only the default
> abstract
> > > implementation that doesn't. They can be marked "abstract" by raising
> > > NotImplementedError(), which requires subclasses to override the method
> > > with the proper return type.
> > >
> > > Signed-off-by: John Snow <jsnow@redhat.com>
> > > ---
> > >  scripts/qapi/schema.py | 8 ++++----
> > >  1 file changed, 4 insertions(+), 4 deletions(-)
> > >
> > > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> > > index c5fdd625452..4600a566005 100644
> > > --- a/scripts/qapi/schema.py
> > > +++ b/scripts/qapi/schema.py
> > > @@ -233,8 +233,8 @@ def visit(self, visitor):
> > >  class QAPISchemaType(QAPISchemaEntity):
> > >      # Return the C type for common use.
> > >      # For the types we commonly box, this is a pointer type.
> > > -    def c_type(self):
> > > -        pass
> > > +    def c_type(self) -> str:
> > > +        raise NotImplementedError()
> > >
> > >      # Return the C type to be used in a parameter list.
> > >      def c_param_type(self):
> > > @@ -244,8 +244,8 @@ def c_param_type(self):
> > >      def c_unboxed_type(self):
> > >          return self.c_type()
> > >
> > > -    def json_type(self):
> > > -        pass
> > > +    def json_type(self) -> str:
> > > +        raise NotImplementedError()
> > >
> > >      def alternate_qtype(self):
> > >          json2qtype = {
> >
> > I wish abstract methods could be done in a more concise way.
>
> The canonical way would be to use the 'abc' module:
>
>   from abc import ABCMeta, abstractmethod
>
>   class A(metaclass=ABCMeta):
>       @abstractmethod
>       def foo(self): pass
>
> Not sure if the @abstractmethod decorator is enough to keep the static
> typing checker happy about a 'str' return type, when there is no body
> impl
>
> With regards,
> Daniel
> --
> |: https://berrange.com      -o-
> https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-
> https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-
> https://www.instagram.com/dberrange :|
>

In practice, I'm under the belief that mypy and pylint both recognize a
method raising NotImplementedError as marking an abstract method without
needing to explicitly inherit from the ABC.

I suppose there's also
https://docs.python.org/3/library/abc.html#abc.abstractmethod which I am
guessing just does this same thing. I'll see what makes mypy happy. I'm
assuming Markus would like to see something like this decorator to make it
more obvious that it's an abstract method.

--js

>

[-- Attachment #2: Type: text/html, Size: 4668 bytes --]

Re: [PATCH 05/19] qapi/schema: make c_type() and json_type() abstract methods

[Daniel P. Berrangé]

On Tue, Nov 21, 2023 at 11:28:17AM -0500, John Snow wrote:
> On Tue, Nov 21, 2023, 8:43 AM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> 
> > On Tue, Nov 21, 2023 at 02:36:54PM +0100, Markus Armbruster wrote:
> > > John Snow <jsnow@redhat.com> writes:
> > >
> > > > These methods should always return a str, it's only the default
> > abstract
> > > > implementation that doesn't. They can be marked "abstract" by raising
> > > > NotImplementedError(), which requires subclasses to override the method
> > > > with the proper return type.
> > > >
> > > > Signed-off-by: John Snow <jsnow@redhat.com>
> > > > ---
> > > >  scripts/qapi/schema.py | 8 ++++----
> > > >  1 file changed, 4 insertions(+), 4 deletions(-)
> > > >
> > > > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> > > > index c5fdd625452..4600a566005 100644
> > > > --- a/scripts/qapi/schema.py
> > > > +++ b/scripts/qapi/schema.py
> > > > @@ -233,8 +233,8 @@ def visit(self, visitor):
> > > >  class QAPISchemaType(QAPISchemaEntity):
> > > >      # Return the C type for common use.
> > > >      # For the types we commonly box, this is a pointer type.
> > > > -    def c_type(self):
> > > > -        pass
> > > > +    def c_type(self) -> str:
> > > > +        raise NotImplementedError()
> > > >
> > > >      # Return the C type to be used in a parameter list.
> > > >      def c_param_type(self):
> > > > @@ -244,8 +244,8 @@ def c_param_type(self):
> > > >      def c_unboxed_type(self):
> > > >          return self.c_type()
> > > >
> > > > -    def json_type(self):
> > > > -        pass
> > > > +    def json_type(self) -> str:
> > > > +        raise NotImplementedError()
> > > >
> > > >      def alternate_qtype(self):
> > > >          json2qtype = {
> > >
> > > I wish abstract methods could be done in a more concise way.
> >
> > The canonical way would be to use the 'abc' module:
> >
> >   from abc import ABCMeta, abstractmethod
> >
> >   class A(metaclass=ABCMeta):
> >       @abstractmethod
> >       def foo(self): pass
> >
> > Not sure if the @abstractmethod decorator is enough to keep the static
> > typing checker happy about a 'str' return type, when there is no body
> > impl
> 
> In practice, I'm under the belief that mypy and pylint both recognize a
> method raising NotImplementedError as marking an abstract method without
> needing to explicitly inherit from the ABC.
> 
> I suppose there's also
> https://docs.python.org/3/library/abc.html#abc.abstractmethod which I am
> guessing just does this same thing. I'll see what makes mypy happy. I'm
> assuming Markus would like to see something like this decorator to make it
> more obvious that it's an abstract method.

The 'abc' module described is an official PEP standard

  https://peps.python.org/pep-3119/

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH 05/19] qapi/schema: make c_type() and json_type() abstract methods

[Markus Armbruster]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Tue, Nov 21, 2023 at 11:28:17AM -0500, John Snow wrote:
>> On Tue, Nov 21, 2023, 8:43 AM Daniel P. Berrangé <berrange@redhat.com>
>> wrote:
>> 
>> > On Tue, Nov 21, 2023 at 02:36:54PM +0100, Markus Armbruster wrote:
>> > > John Snow <jsnow@redhat.com> writes:
>> > >
>> > > > These methods should always return a str, it's only the default
>> > abstract
>> > > > implementation that doesn't. They can be marked "abstract" by raising
>> > > > NotImplementedError(), which requires subclasses to override the method
>> > > > with the proper return type.
>> > > >
>> > > > Signed-off-by: John Snow <jsnow@redhat.com>
>> > > > ---
>> > > >  scripts/qapi/schema.py | 8 ++++----
>> > > >  1 file changed, 4 insertions(+), 4 deletions(-)
>> > > >
>> > > > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
>> > > > index c5fdd625452..4600a566005 100644
>> > > > --- a/scripts/qapi/schema.py
>> > > > +++ b/scripts/qapi/schema.py
>> > > > @@ -233,8 +233,8 @@ def visit(self, visitor):
>> > > >  class QAPISchemaType(QAPISchemaEntity):
>> > > >      # Return the C type for common use.
>> > > >      # For the types we commonly box, this is a pointer type.
>> > > > -    def c_type(self):
>> > > > -        pass
>> > > > +    def c_type(self) -> str:
>> > > > +        raise NotImplementedError()
>> > > >
>> > > >      # Return the C type to be used in a parameter list.
>> > > >      def c_param_type(self):
>> > > > @@ -244,8 +244,8 @@ def c_param_type(self):
>> > > >      def c_unboxed_type(self):
>> > > >          return self.c_type()
>> > > >
>> > > > -    def json_type(self):
>> > > > -        pass
>> > > > +    def json_type(self) -> str:
>> > > > +        raise NotImplementedError()
>> > > >
>> > > >      def alternate_qtype(self):
>> > > >          json2qtype = {
>> > >
>> > > I wish abstract methods could be done in a more concise way.
>> >
>> > The canonical way would be to use the 'abc' module:
>> >
>> >   from abc import ABCMeta, abstractmethod
>> >
>> >   class A(metaclass=ABCMeta):
>> >       @abstractmethod
>> >       def foo(self): pass
>> >
>> > Not sure if the @abstractmethod decorator is enough to keep the static
>> > typing checker happy about a 'str' return type, when there is no body
>> > impl
>> 
>> In practice, I'm under the belief that mypy and pylint both recognize a
>> method raising NotImplementedError as marking an abstract method without
>> needing to explicitly inherit from the ABC.
>> 
>> I suppose there's also
>> https://docs.python.org/3/library/abc.html#abc.abstractmethod which I am
>> guessing just does this same thing. I'll see what makes mypy happy. I'm
>> assuming Markus would like to see something like this decorator to make it
>> more obvious that it's an abstract method.
>
> The 'abc' module described is an official PEP standard
>
>   https://peps.python.org/pep-3119/

Compare:

    @abstractmethod
    def c_type(self) -> str:
        pass

    def c_type(self) -> str:
        raise NotImplementedError()

I prefer the former, because it's more explicit.

Bonus: prevents accidental instantiation, and sub-classes don't need to
know what's abstract in the super-class, they can blindly use super()
calls.  docs.python.org:

    Using this decorator requires that the class’s metaclass is ABCMeta
    or is derived from it.  A class that has a metaclass derived from
    ABCMeta cannot be instantiated unless all of its abstract methods
    and properties are overridden.  The abstract methods can be called
    using any of the normal ‘super’ call mechanisms.  abstractmethod()
    may be used to declare abstract methods for properties and
    descriptors.

Hardly matters here, but since it's free...

Re: [PATCH 05/19] qapi/schema: make c_type() and json_type() abstract methods

[Daniel P. Berrangé]

On Wed, Nov 22, 2023 at 10:50:47AM +0100, Markus Armbruster wrote:
> Daniel P. Berrangé <berrange@redhat.com> writes:
> 
> > On Tue, Nov 21, 2023 at 11:28:17AM -0500, John Snow wrote:
> >> On Tue, Nov 21, 2023, 8:43 AM Daniel P. Berrangé <berrange@redhat.com>
> >> wrote:
> >> 
> >> > On Tue, Nov 21, 2023 at 02:36:54PM +0100, Markus Armbruster wrote:
> >> > > John Snow <jsnow@redhat.com> writes:
> >> > >
> >> > > > These methods should always return a str, it's only the default
> >> > abstract
> >> > > > implementation that doesn't. They can be marked "abstract" by raising
> >> > > > NotImplementedError(), which requires subclasses to override the method
> >> > > > with the proper return type.
> >> > > >
> >> > > > Signed-off-by: John Snow <jsnow@redhat.com>
> >> > > > ---
> >> > > >  scripts/qapi/schema.py | 8 ++++----
> >> > > >  1 file changed, 4 insertions(+), 4 deletions(-)
> >> > > >
> >> > > > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> >> > > > index c5fdd625452..4600a566005 100644
> >> > > > --- a/scripts/qapi/schema.py
> >> > > > +++ b/scripts/qapi/schema.py
> >> > > > @@ -233,8 +233,8 @@ def visit(self, visitor):
> >> > > >  class QAPISchemaType(QAPISchemaEntity):
> >> > > >      # Return the C type for common use.
> >> > > >      # For the types we commonly box, this is a pointer type.
> >> > > > -    def c_type(self):
> >> > > > -        pass
> >> > > > +    def c_type(self) -> str:
> >> > > > +        raise NotImplementedError()
> >> > > >
> >> > > >      # Return the C type to be used in a parameter list.
> >> > > >      def c_param_type(self):
> >> > > > @@ -244,8 +244,8 @@ def c_param_type(self):
> >> > > >      def c_unboxed_type(self):
> >> > > >          return self.c_type()
> >> > > >
> >> > > > -    def json_type(self):
> >> > > > -        pass
> >> > > > +    def json_type(self) -> str:
> >> > > > +        raise NotImplementedError()
> >> > > >
> >> > > >      def alternate_qtype(self):
> >> > > >          json2qtype = {
> >> > >
> >> > > I wish abstract methods could be done in a more concise way.
> >> >
> >> > The canonical way would be to use the 'abc' module:
> >> >
> >> >   from abc import ABCMeta, abstractmethod
> >> >
> >> >   class A(metaclass=ABCMeta):
> >> >       @abstractmethod
> >> >       def foo(self): pass
> >> >
> >> > Not sure if the @abstractmethod decorator is enough to keep the static
> >> > typing checker happy about a 'str' return type, when there is no body
> >> > impl
> >> 
> >> In practice, I'm under the belief that mypy and pylint both recognize a
> >> method raising NotImplementedError as marking an abstract method without
> >> needing to explicitly inherit from the ABC.
> >> 
> >> I suppose there's also
> >> https://docs.python.org/3/library/abc.html#abc.abstractmethod which I am
> >> guessing just does this same thing. I'll see what makes mypy happy. I'm
> >> assuming Markus would like to see something like this decorator to make it
> >> more obvious that it's an abstract method.
> >
> > The 'abc' module described is an official PEP standard
> >
> >   https://peps.python.org/pep-3119/
> 
> Compare:
> 
>     @abstractmethod
>     def c_type(self) -> str:
>         pass
> 
>     def c_type(self) -> str:
>         raise NotImplementedError()
> 
> I prefer the former, because it's more explicit.
> 
> Bonus: prevents accidental instantiation, and sub-classes don't need to
> know what's abstract in the super-class, they can blindly use super()
> calls. 

Being able to blindly call the parent impl via super() is more than
just a bonus, it is a super compelling reason to use this. It protects
your code against bugs from future re-factoring of the class hierarchy
whether adding or removing parents. Even if we don't expect to need it
for this particular class, I think this justifies having a policy of
using 'abc' everywhere we need abstract methods.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH 06/19] qapi/schema: adjust type narrowing for mypy's benefit

[John Snow]

We already take care to perform some type narrowing for arg_type and
ret_type, but not in a way where mypy can utilize the result. A simple
change to use a temporary variable helps the medicine go down.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 4600a566005..a1094283828 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -825,13 +825,14 @@ def __init__(self, name, info, doc, ifcond, features,
     def check(self, schema):
         super().check(schema)
         if self._arg_type_name:
-            self.arg_type = schema.resolve_type(
+            arg_type = schema.resolve_type(
                 self._arg_type_name, self.info, "command's 'data'")
-            if not isinstance(self.arg_type, QAPISchemaObjectType):
+            if not isinstance(arg_type, QAPISchemaObjectType):
                 raise QAPISemError(
                     self.info,
                     "command's 'data' cannot take %s"
-                    % self.arg_type.describe())
+                    % arg_type.describe())
+            self.arg_type = arg_type
             if self.arg_type.variants and not self.boxed:
                 raise QAPISemError(
                     self.info,
@@ -848,8 +849,7 @@ def check(self, schema):
             if self.name not in self.info.pragma.command_returns_exceptions:
                 typ = self.ret_type
                 if isinstance(typ, QAPISchemaArrayType):
-                    typ = self.ret_type.element_type
-                    assert typ
+                    typ = typ.element_type
                 if not isinstance(typ, QAPISchemaObjectType):
                     raise QAPISemError(
                         self.info,
@@ -885,13 +885,14 @@ def __init__(self, name, info, doc, ifcond, features, arg_type, boxed):
     def check(self, schema):
         super().check(schema)
         if self._arg_type_name:
-            self.arg_type = schema.resolve_type(
+            typ = schema.resolve_type(
                 self._arg_type_name, self.info, "event's 'data'")
-            if not isinstance(self.arg_type, QAPISchemaObjectType):
+            if not isinstance(typ, QAPISchemaObjectType):
                 raise QAPISemError(
                     self.info,
                     "event's 'data' cannot take %s"
-                    % self.arg_type.describe())
+                    % typ.describe())
+            self.arg_type = typ
             if self.arg_type.variants and not self.boxed:
                 raise QAPISemError(
                     self.info,
-- 
2.41.0

Re: [PATCH 06/19] qapi/schema: adjust type narrowing for mypy's benefit

[Philippe Mathieu-Daudé]

On 16/11/23 02:43, John Snow wrote:
> We already take care to perform some type narrowing for arg_type and
> ret_type, but not in a way where mypy can utilize the result. A simple
> change to use a temporary variable helps the medicine go down.
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   scripts/qapi/schema.py | 17 +++++++++--------
>   1 file changed, 9 insertions(+), 8 deletions(-)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

Re: [PATCH 06/19] qapi/schema: adjust type narrowing for mypy's benefit

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> We already take care to perform some type narrowing for arg_type and
> ret_type, but not in a way where mypy can utilize the result. A simple
> change to use a temporary variable helps the medicine go down.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/schema.py | 17 +++++++++--------
>  1 file changed, 9 insertions(+), 8 deletions(-)
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 4600a566005..a1094283828 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -825,13 +825,14 @@ def __init__(self, name, info, doc, ifcond, features,
>      def check(self, schema):
>          super().check(schema)
>          if self._arg_type_name:
> -            self.arg_type = schema.resolve_type(
> +            arg_type = schema.resolve_type(
>                  self._arg_type_name, self.info, "command's 'data'")
> -            if not isinstance(self.arg_type, QAPISchemaObjectType):
> +            if not isinstance(arg_type, QAPISchemaObjectType):
>                  raise QAPISemError(
>                      self.info,
>                      "command's 'data' cannot take %s"
> -                    % self.arg_type.describe())
> +                    % arg_type.describe())
> +            self.arg_type = arg_type
>              if self.arg_type.variants and not self.boxed:
>                  raise QAPISemError(
>                      self.info,
> @@ -848,8 +849,7 @@ def check(self, schema):
>              if self.name not in self.info.pragma.command_returns_exceptions:
>                  typ = self.ret_type
>                  if isinstance(typ, QAPISchemaArrayType):
> -                    typ = self.ret_type.element_type
> -                    assert typ
> +                    typ = typ.element_type
>                  if not isinstance(typ, QAPISchemaObjectType):
>                      raise QAPISemError(
>                          self.info,
> @@ -885,13 +885,14 @@ def __init__(self, name, info, doc, ifcond, features, arg_type, boxed):
>      def check(self, schema):
>          super().check(schema)
>          if self._arg_type_name:
> -            self.arg_type = schema.resolve_type(
> +            typ = schema.resolve_type(
>                  self._arg_type_name, self.info, "event's 'data'")
> -            if not isinstance(self.arg_type, QAPISchemaObjectType):
> +            if not isinstance(typ, QAPISchemaObjectType):
>                  raise QAPISemError(
>                      self.info,
>                      "event's 'data' cannot take %s"
> -                    % self.arg_type.describe())
> +                    % typ.describe())
> +            self.arg_type = typ
>              if self.arg_type.variants and not self.boxed:
>                  raise QAPISemError(
>                      self.info,

Harmless enough.  I can't quite see the mypy problem, though.  Care to
elaborate a bit?

Re: [PATCH 06/19] qapi/schema: adjust type narrowing for mypy's benefit

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 3752 bytes --]

On Tue, Nov 21, 2023, 9:09 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > We already take care to perform some type narrowing for arg_type and
> > ret_type, but not in a way where mypy can utilize the result. A simple
> > change to use a temporary variable helps the medicine go down.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  scripts/qapi/schema.py | 17 +++++++++--------
> >  1 file changed, 9 insertions(+), 8 deletions(-)
> >
> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> > index 4600a566005..a1094283828 100644
> > --- a/scripts/qapi/schema.py
> > +++ b/scripts/qapi/schema.py
> > @@ -825,13 +825,14 @@ def __init__(self, name, info, doc, ifcond,
> features,
> >      def check(self, schema):
> >          super().check(schema)
> >          if self._arg_type_name:
> > -            self.arg_type = schema.resolve_type(
> > +            arg_type = schema.resolve_type(
> >                  self._arg_type_name, self.info, "command's 'data'")
> > -            if not isinstance(self.arg_type, QAPISchemaObjectType):
> > +            if not isinstance(arg_type, QAPISchemaObjectType):
> >                  raise QAPISemError(
> >                      self.info,
> >                      "command's 'data' cannot take %s"
> > -                    % self.arg_type.describe())
> > +                    % arg_type.describe())
> > +            self.arg_type = arg_type
> >              if self.arg_type.variants and not self.boxed:
> >                  raise QAPISemError(
> >                      self.info,
> > @@ -848,8 +849,7 @@ def check(self, schema):
> >              if self.name not in
> self.info.pragma.command_returns_exceptions:
> >                  typ = self.ret_type
> >                  if isinstance(typ, QAPISchemaArrayType):
> > -                    typ = self.ret_type.element_type
> > -                    assert typ
> > +                    typ = typ.element_type
>

In this case, we've narrowed typ but not self.ret_type and mypy is not sure
they're synonymous here (lack of power in mypy's model, maybe?). Work in
terms of the temporary type we've already narrowed so mypy knows we have an
element_type field.

>                  if not isinstance(typ, QAPISchemaObjectType):
> >                      raise QAPISemError(
> >                          self.info,
> > @@ -885,13 +885,14 @@ def __init__(self, name, info, doc, ifcond,
> features, arg_type, boxed):
> >      def check(self, schema):
> >          super().check(schema)
> >          if self._arg_type_name:
> > -            self.arg_type = schema.resolve_type(
> > +            typ = schema.resolve_type(
> >                  self._arg_type_name, self.info, "event's 'data'")
> > -            if not isinstance(self.arg_type, QAPISchemaObjectType):
> > +            if not isinstance(typ, QAPISchemaObjectType):
> >                  raise QAPISemError(
> >                      self.info,
> >                      "event's 'data' cannot take %s"
> > -                    % self.arg_type.describe())
> > +                    % typ.describe())
> > +            self.arg_type = typ
> >              if self.arg_type.variants and not self.boxed:
> >                  raise QAPISemError(
> >                      self.info,
>
> Harmless enough.  I can't quite see the mypy problem, though.  Care to
> elaborate a bit?
>

self.arg_type has a narrower type- or, it WILL at the end of this series -
so we need to narrow a temporary variable first before assigning it to the
object state.

We already perform the necessary check/narrowing, so this is really just
pointing out that it's a bad idea to assign the state before the type
check. Now we type check before assigning state.

--js

[-- Attachment #2: Type: text/html, Size: 6206 bytes --]

Re: [PATCH 06/19] qapi/schema: adjust type narrowing for mypy's benefit

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Tue, Nov 21, 2023, 9:09 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > We already take care to perform some type narrowing for arg_type and
>> > ret_type, but not in a way where mypy can utilize the result. A simple
>> > change to use a temporary variable helps the medicine go down.
>> >
>> > Signed-off-by: John Snow <jsnow@redhat.com>
>> > ---
>> >  scripts/qapi/schema.py | 17 +++++++++--------
>> >  1 file changed, 9 insertions(+), 8 deletions(-)
>> >
>> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
>> > index 4600a566005..a1094283828 100644
>> > --- a/scripts/qapi/schema.py
>> > +++ b/scripts/qapi/schema.py
>> > @@ -825,13 +825,14 @@ def __init__(self, name, info, doc, ifcond, features,
>> >      def check(self, schema):
>> >          super().check(schema)
>> >          if self._arg_type_name:
>> > -            self.arg_type = schema.resolve_type(
>> > +            arg_type = schema.resolve_type(
>> >                  self._arg_type_name, self.info, "command's 'data'")
>> > -            if not isinstance(self.arg_type, QAPISchemaObjectType):
>> > +            if not isinstance(arg_type, QAPISchemaObjectType):
>> >                  raise QAPISemError(
>> >                      self.info,
>> >                      "command's 'data' cannot take %s"
>> > -                    % self.arg_type.describe())
>> > +                    % arg_type.describe())
>> > +            self.arg_type = arg_type
>> >              if self.arg_type.variants and not self.boxed:
>> >                  raise QAPISemError(
>> >                      self.info,

Same story as for QAPISchemaEvent.check() below.  Correct?

>> > @@ -848,8 +849,7 @@ def check(self, schema):
>> >              if self.name not in self.info.pragma.command_returns_exceptions:
>> >                  typ = self.ret_type
>> >                  if isinstance(typ, QAPISchemaArrayType):
>> > -                    typ = self.ret_type.element_type
>> > -                    assert typ
>> > +                    typ = typ.element_type
>>
>
> In this case, we've narrowed typ but not self.ret_type and mypy is not sure
> they're synonymous here (lack of power in mypy's model, maybe?). Work in
> terms of the temporary type we've already narrowed so mypy knows we have an
> element_type field.

The conditional ensures @typ is QAPISchemaArrayType.

In mypy's view, @typ is QAPISchemaArrayType, but self.ret_type is only
Optional[QAPISchemaType].

Therefore, it chokes on self.ret_type.element_type, but is happy with
typ.element_type.

Correct?

Why delete the assertion?  Oh!  Hmm, should the deletion go into PATCH
10?

>>                  if not isinstance(typ, QAPISchemaObjectType):
>> >                      raise QAPISemError(
>> >                          self.info,
>> > @@ -885,13 +885,14 @@ def __init__(self, name, info, doc, ifcond, features, arg_type, boxed):
>> >      def check(self, schema):
>> >          super().check(schema)
>> >          if self._arg_type_name:
>> > -            self.arg_type = schema.resolve_type(
>> > +            typ = schema.resolve_type(
>> >                  self._arg_type_name, self.info, "event's 'data'")
>> > -            if not isinstance(self.arg_type, QAPISchemaObjectType):
>> > +            if not isinstance(typ, QAPISchemaObjectType):
>> >                  raise QAPISemError(
>> >                      self.info,
>> >                      "event's 'data' cannot take %s"
>> > -                    % self.arg_type.describe())
>> > +                    % typ.describe())
>> > +            self.arg_type = typ
>> >              if self.arg_type.variants and not self.boxed:
>> >                  raise QAPISemError(
>> >                      self.info,
>>
>> Harmless enough.  I can't quite see the mypy problem, though.  Care to
>> elaborate a bit?
>>
>
> self.arg_type has a narrower type- or, it WILL at the end of this series -
> so we need to narrow a temporary variable first before assigning it to the
> object state.
>
> We already perform the necessary check/narrowing, so this is really just
> pointing out that it's a bad idea to assign the state before the type
> check. Now we type check before assigning state.

After PATCH 16, .resolve_type() will return QAPISchemaType, and
self.arg_type will be Optional[QAPISchemaObjectType].  Correct?

Re: [PATCH 06/19] qapi/schema: adjust type narrowing for mypy's benefit

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 5110 bytes --]

On Wed, Nov 22, 2023, 7:00 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > On Tue, Nov 21, 2023, 9:09 AM Markus Armbruster <armbru@redhat.com>
> wrote:
> >
> >> John Snow <jsnow@redhat.com> writes:
> >>
> >> > We already take care to perform some type narrowing for arg_type and
> >> > ret_type, but not in a way where mypy can utilize the result. A simple
> >> > change to use a temporary variable helps the medicine go down.
> >> >
> >> > Signed-off-by: John Snow <jsnow@redhat.com>
> >> > ---
> >> >  scripts/qapi/schema.py | 17 +++++++++--------
> >> >  1 file changed, 9 insertions(+), 8 deletions(-)
> >> >
> >> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> >> > index 4600a566005..a1094283828 100644
> >> > --- a/scripts/qapi/schema.py
> >> > +++ b/scripts/qapi/schema.py
> >> > @@ -825,13 +825,14 @@ def __init__(self, name, info, doc, ifcond,
> features,
> >> >      def check(self, schema):
> >> >          super().check(schema)
> >> >          if self._arg_type_name:
> >> > -            self.arg_type = schema.resolve_type(
> >> > +            arg_type = schema.resolve_type(
> >> >                  self._arg_type_name, self.info, "command's 'data'")
> >> > -            if not isinstance(self.arg_type, QAPISchemaObjectType):
> >> > +            if not isinstance(arg_type, QAPISchemaObjectType):
> >> >                  raise QAPISemError(
> >> >                      self.info,
> >> >                      "command's 'data' cannot take %s"
> >> > -                    % self.arg_type.describe())
> >> > +                    % arg_type.describe())
> >> > +            self.arg_type = arg_type
> >> >              if self.arg_type.variants and not self.boxed:
> >> >                  raise QAPISemError(
> >> >                      self.info,
>
> Same story as for QAPISchemaEvent.check() below.  Correct?
>

Yep.


> >> > @@ -848,8 +849,7 @@ def check(self, schema):
> >> >              if self.name not in
> self.info.pragma.command_returns_exceptions:
> >> >                  typ = self.ret_type
> >> >                  if isinstance(typ, QAPISchemaArrayType):
> >> > -                    typ = self.ret_type.element_type
> >> > -                    assert typ
> >> > +                    typ = typ.element_type
> >>
> >
> > In this case, we've narrowed typ but not self.ret_type and mypy is not
> sure
> > they're synonymous here (lack of power in mypy's model, maybe?). Work in
> > terms of the temporary type we've already narrowed so mypy knows we have
> an
> > element_type field.
>
> The conditional ensures @typ is QAPISchemaArrayType.
>
> In mypy's view, @typ is QAPISchemaArrayType, but self.ret_type is only
> Optional[QAPISchemaType].
>
> Therefore, it chokes on self.ret_type.element_type, but is happy with
> typ.element_type.
>
> Correct?
>

I think so, yes. In this conditional block, we need to work in terms of
typ, which has been narrowed. The broader type doesn't have .element_type.


> Why delete the assertion?  Oh!  Hmm, should the deletion go into PATCH
> 10?
>

Yeah, just a patch-splitting goof. I'll repair that.


> >>                  if not isinstance(typ, QAPISchemaObjectType):
> >> >                      raise QAPISemError(
> >> >                          self.info,
> >> > @@ -885,13 +885,14 @@ def __init__(self, name, info, doc, ifcond,
> features, arg_type, boxed):
> >> >      def check(self, schema):
> >> >          super().check(schema)
> >> >          if self._arg_type_name:
> >> > -            self.arg_type = schema.resolve_type(
> >> > +            typ = schema.resolve_type(
> >> >                  self._arg_type_name, self.info, "event's 'data'")
> >> > -            if not isinstance(self.arg_type, QAPISchemaObjectType):
> >> > +            if not isinstance(typ, QAPISchemaObjectType):
> >> >                  raise QAPISemError(
> >> >                      self.info,
> >> >                      "event's 'data' cannot take %s"
> >> > -                    % self.arg_type.describe())
> >> > +                    % typ.describe())
> >> > +            self.arg_type = typ
> >> >              if self.arg_type.variants and not self.boxed:
> >> >                  raise QAPISemError(
> >> >                      self.info,
> >>
> >> Harmless enough.  I can't quite see the mypy problem, though.  Care to
> >> elaborate a bit?
> >>
> >
> > self.arg_type has a narrower type- or, it WILL at the end of this series
> -
> > so we need to narrow a temporary variable first before assigning it to
> the
> > object state.
> >
> > We already perform the necessary check/narrowing, so this is really just
> > pointing out that it's a bad idea to assign the state before the type
> > check. Now we type check before assigning state.
>
> After PATCH 16, .resolve_type() will return QAPISchemaType, and
> self.arg_type will be Optional[QAPISchemaObjectType].  Correct?
>

Sounds right. Sometimes it's a little hard to see what the error is before
the rest of the types go in, a hazard of needing all patches to bisect
without regression.

Do you want a more elaborate commit message?

--js

[-- Attachment #2: Type: text/html, Size: 8545 bytes --]

Re: [PATCH 06/19] qapi/schema: adjust type narrowing for mypy's benefit

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Wed, Nov 22, 2023, 7:00 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > On Tue, Nov 21, 2023, 9:09 AM Markus Armbruster <armbru@redhat.com> wrote:

[...]

>> >> Harmless enough.  I can't quite see the mypy problem, though.  Care to
>> >> elaborate a bit?
>> >>
>> >
>> > self.arg_type has a narrower type- or, it WILL at the end of this series -
>> > so we need to narrow a temporary variable first before assigning it to the
>> > object state.
>> >
>> > We already perform the necessary check/narrowing, so this is really just
>> > pointing out that it's a bad idea to assign the state before the type
>> > check. Now we type check before assigning state.
>>
>> After PATCH 16, .resolve_type() will return QAPISchemaType, and
>> self.arg_type will be Optional[QAPISchemaObjectType].  Correct?
>>
>
> Sounds right. Sometimes it's a little hard to see what the error is before
> the rest of the types go in, a hazard of needing all patches to bisect
> without regression.
>
> Do you want a more elaborate commit message?

Your commit messages of PATCH 3+4 show the error.  Helps.

Maybe

    qapi/schema: Adjust type narrowing for mypy's benefit

    We already take care to perform some type narrowing for arg_type and
    ret_type, but not in a way where mypy can utilize the result once we
    add type hints:

        error message goes here

    A simple change to use a temporary variable helps the medicine go
    down.

[PATCH 07/19] qapi/introspect: assert schema.lookup_type did not fail

[John Snow]

lookup_type() is capable of returning None, but introspect.py isn't
prepared for that. (And rightly so, if these built-in types are absent,
something has gone hugely wrong.)

RFC: This is slightly cumbersome as-is, but a patch at the end of this series
tries to address it with some slightly slicker lookup functions that
don't need as much hand-holding.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/introspect.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 67c7d89aae0..42981bce163 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -227,10 +227,14 @@ def _use_type(self, typ: QAPISchemaType) -> str:
 
         # Map the various integer types to plain int
         if typ.json_type() == 'int':
-            typ = self._schema.lookup_type('int')
+            tmp = self._schema.lookup_type('int')
+            assert tmp is not None
+            typ = tmp
         elif (isinstance(typ, QAPISchemaArrayType) and
               typ.element_type.json_type() == 'int'):
-            typ = self._schema.lookup_type('intList')
+            tmp = self._schema.lookup_type('intList')
+            assert tmp is not None
+            typ = tmp
         # Add type to work queue if new
         if typ not in self._used_types:
             self._used_types.append(typ)
-- 
2.41.0

Re: [PATCH 07/19] qapi/introspect: assert schema.lookup_type did not fail

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> lookup_type() is capable of returning None, but introspect.py isn't
> prepared for that. (And rightly so, if these built-in types are absent,
> something has gone hugely wrong.)
>
> RFC: This is slightly cumbersome as-is, but a patch at the end of this series
> tries to address it with some slightly slicker lookup functions that
> don't need as much hand-holding.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/introspect.py | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
> index 67c7d89aae0..42981bce163 100644
> --- a/scripts/qapi/introspect.py
> +++ b/scripts/qapi/introspect.py
> @@ -227,10 +227,14 @@ def _use_type(self, typ: QAPISchemaType) -> str:
>  
>          # Map the various integer types to plain int
>          if typ.json_type() == 'int':
> -            typ = self._schema.lookup_type('int')
> +            tmp = self._schema.lookup_type('int')
> +            assert tmp is not None

More laconic: assert tmp

> +            typ = tmp
>          elif (isinstance(typ, QAPISchemaArrayType) and
>                typ.element_type.json_type() == 'int'):
> -            typ = self._schema.lookup_type('intList')
> +            tmp = self._schema.lookup_type('intList')
> +            assert tmp is not None
> +            typ = tmp
>          # Add type to work queue if new
>          if typ not in self._used_types:
>              self._used_types.append(typ)

Not fond of naming things @tmp, but I don't have a better name to offer.

We could avoid the lookup by having _def_predefineds() set suitable
attributes, like it serts .the_empty_object_type.  Matter of taste.  Not
now unless you want to.

Re: [PATCH 07/19] qapi/introspect: assert schema.lookup_type did not fail

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2398 bytes --]

On Tue, Nov 21, 2023, 9:17 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > lookup_type() is capable of returning None, but introspect.py isn't
> > prepared for that. (And rightly so, if these built-in types are absent,
> > something has gone hugely wrong.)
> >
> > RFC: This is slightly cumbersome as-is, but a patch at the end of this
> series
> > tries to address it with some slightly slicker lookup functions that
> > don't need as much hand-holding.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  scripts/qapi/introspect.py | 8 ++++++--
> >  1 file changed, 6 insertions(+), 2 deletions(-)
> >
> > diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
> > index 67c7d89aae0..42981bce163 100644
> > --- a/scripts/qapi/introspect.py
> > +++ b/scripts/qapi/introspect.py
> > @@ -227,10 +227,14 @@ def _use_type(self, typ: QAPISchemaType) -> str:
> >
> >          # Map the various integer types to plain int
> >          if typ.json_type() == 'int':
> > -            typ = self._schema.lookup_type('int')
> > +            tmp = self._schema.lookup_type('int')
> > +            assert tmp is not None
>
> More laconic: assert tmp
>

*looks up "laconic"*

hey, "terse" is even fewer letters!

(but, you're right. I think I adopted the "is not none" out of a habit for
distinguishing false-y values from the None value, but in this case we
really wouldn't want to have either, so the shorter form is fine, though
for mypy's sake we only care about guarding against None here.)


> > +            typ = tmp
> >          elif (isinstance(typ, QAPISchemaArrayType) and
> >                typ.element_type.json_type() == 'int'):
> > -            typ = self._schema.lookup_type('intList')
> > +            tmp = self._schema.lookup_type('intList')
> > +            assert tmp is not None
> > +            typ = tmp
> >          # Add type to work queue if new
> >          if typ not in self._used_types:
> >              self._used_types.append(typ)
>
> Not fond of naming things @tmp, but I don't have a better name to offer.
>
> We could avoid the lookup by having _def_predefineds() set suitable
> attributes, like it serts .the_empty_object_type.  Matter of taste.  Not
> now unless you want to.
>

 Check the end of the series for different lookup methods, too. We can
discuss your preferred solution then, perhaps?

>

[-- Attachment #2: Type: text/html, Size: 3808 bytes --]

Re: [PATCH 07/19] qapi/introspect: assert schema.lookup_type did not fail

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Tue, Nov 21, 2023, 9:17 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > lookup_type() is capable of returning None, but introspect.py isn't
>> > prepared for that. (And rightly so, if these built-in types are absent,
>> > something has gone hugely wrong.)
>> >
>> > RFC: This is slightly cumbersome as-is, but a patch at the end of this
>> series
>> > tries to address it with some slightly slicker lookup functions that
>> > don't need as much hand-holding.
>> >
>> > Signed-off-by: John Snow <jsnow@redhat.com>
>> > ---
>> >  scripts/qapi/introspect.py | 8 ++++++--
>> >  1 file changed, 6 insertions(+), 2 deletions(-)
>> >
>> > diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
>> > index 67c7d89aae0..42981bce163 100644
>> > --- a/scripts/qapi/introspect.py
>> > +++ b/scripts/qapi/introspect.py
>> > @@ -227,10 +227,14 @@ def _use_type(self, typ: QAPISchemaType) -> str:
>> >
>> >          # Map the various integer types to plain int
>> >          if typ.json_type() == 'int':
>> > -            typ = self._schema.lookup_type('int')
>> > +            tmp = self._schema.lookup_type('int')
>> > +            assert tmp is not None
>>
>> More laconic: assert tmp
>>
>
> *looks up "laconic"*
>
> hey, "terse" is even fewer letters!

Touché!

> (but, you're right. I think I adopted the "is not none" out of a habit for
> distinguishing false-y values from the None value, but in this case we

It's a good habit.

> really wouldn't want to have either, so the shorter form is fine, though
> for mypy's sake we only care about guarding against None here.)

Right.

>> > +            typ = tmp
>> >          elif (isinstance(typ, QAPISchemaArrayType) and
>> >                typ.element_type.json_type() == 'int'):
>> > -            typ = self._schema.lookup_type('intList')
>> > +            tmp = self._schema.lookup_type('intList')
>> > +            assert tmp is not None
>> > +            typ = tmp
>> >          # Add type to work queue if new
>> >          if typ not in self._used_types:
>> >              self._used_types.append(typ)
>>
>> Not fond of naming things @tmp, but I don't have a better name to offer.
>>
>> We could avoid the lookup by having _def_predefineds() set suitable
>> attributes, like it serts .the_empty_object_type.  Matter of taste.  Not
>> now unless you want to.
>>
>
>  Check the end of the series for different lookup methods, too. We can
> discuss your preferred solution then, perhaps?

Works for me.

[PATCH 08/19] qapi/schema: add static typing and assertions to lookup_type()

[John Snow]

This function is a bit hard to type as-is; mypy needs some assertions to
assist with the type narrowing.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index a1094283828..3308f334872 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -968,8 +968,12 @@ def lookup_entity(self, name, typ=None):
             return None
         return ent
 
-    def lookup_type(self, name):
-        return self.lookup_entity(name, QAPISchemaType)
+    def lookup_type(self, name: str) -> Optional[QAPISchemaType]:
+        typ = self.lookup_entity(name, QAPISchemaType)
+        if typ is None:
+            return None
+        assert isinstance(typ, QAPISchemaType)
+        return typ
 
     def resolve_type(self, name, info, what):
         typ = self.lookup_type(name)
-- 
2.41.0

Re: [PATCH 08/19] qapi/schema: add static typing and assertions to lookup_type()

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> This function is a bit hard to type as-is; mypy needs some assertions to
> assist with the type narrowing.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/schema.py | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index a1094283828..3308f334872 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -968,8 +968,12 @@ def lookup_entity(self, name, typ=None):
>              return None
>          return ent
>  
> -    def lookup_type(self, name):
> -        return self.lookup_entity(name, QAPISchemaType)
> +    def lookup_type(self, name: str) -> Optional[QAPISchemaType]:

Any particular reason not to delay the type hints until PATCH 16?

> +        typ = self.lookup_entity(name, QAPISchemaType)
> +        if typ is None:
> +            return None
> +        assert isinstance(typ, QAPISchemaType)
> +        return typ

Would

           typ = self.lookup_entity(name, QAPISchemaType)
           assert isinstance(typ, Optional[QAPISchemaType])
           return typ

work?

>  
>      def resolve_type(self, name, info, what):
>          typ = self.lookup_type(name)

Re: [PATCH 08/19] qapi/schema: add static typing and assertions to lookup_type()

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2029 bytes --]

On Tue, Nov 21, 2023, 9:21 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > This function is a bit hard to type as-is; mypy needs some assertions to
> > assist with the type narrowing.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  scripts/qapi/schema.py | 8 ++++++--
> >  1 file changed, 6 insertions(+), 2 deletions(-)
> >
> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> > index a1094283828..3308f334872 100644
> > --- a/scripts/qapi/schema.py
> > +++ b/scripts/qapi/schema.py
> > @@ -968,8 +968,12 @@ def lookup_entity(self, name, typ=None):
> >              return None
> >          return ent
> >
> > -    def lookup_type(self, name):
> > -        return self.lookup_entity(name, QAPISchemaType)
> > +    def lookup_type(self, name: str) -> Optional[QAPISchemaType]:
>
> Any particular reason not to delay the type hints until PATCH 16?
>

I forget. In some cases I did things a little differently so that the type
checking would pass for each patch in the series, which sometimes required
some concessions.

Is this one of those cases? Uh, I forget.

If it isn't, its almost certainly the case that I just figured I'd type
this one function in one place instead of splitting it apart into two
patches.

I can try to shift the typing later and see what happens if you prefer it
that way.


> > +        typ = self.lookup_entity(name, QAPISchemaType)
> > +        if typ is None:
> > +            return None
> > +        assert isinstance(typ, QAPISchemaType)
> > +        return typ
>
> Would
>
>            typ = self.lookup_entity(name, QAPISchemaType)
>            assert isinstance(typ, Optional[QAPISchemaType])
>            return typ
>
> work?
>

I don't *think* so, Optional isn't a runtime construct. We can combine it
into "assert x is None or isinstance(x, foo)" though - I believe that's
used elsewhere in the qapi generator.


> >
> >      def resolve_type(self, name, info, what):
> >          typ = self.lookup_type(name)
>
>

[-- Attachment #2: Type: text/html, Size: 3395 bytes --]

Re: [PATCH 08/19] qapi/schema: add static typing and assertions to lookup_type()

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Tue, Nov 21, 2023, 9:21 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > This function is a bit hard to type as-is; mypy needs some assertions to
>> > assist with the type narrowing.
>> >
>> > Signed-off-by: John Snow <jsnow@redhat.com>
>> > ---
>> >  scripts/qapi/schema.py | 8 ++++++--
>> >  1 file changed, 6 insertions(+), 2 deletions(-)
>> >
>> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
>> > index a1094283828..3308f334872 100644
>> > --- a/scripts/qapi/schema.py
>> > +++ b/scripts/qapi/schema.py
>> > @@ -968,8 +968,12 @@ def lookup_entity(self, name, typ=None):
>> >              return None
>> >          return ent
>> >
>> > -    def lookup_type(self, name):
>> > -        return self.lookup_entity(name, QAPISchemaType)
>> > +    def lookup_type(self, name: str) -> Optional[QAPISchemaType]:
>>
>> Any particular reason not to delay the type hints until PATCH 16?
>>
>
> I forget. In some cases I did things a little differently so that the type
> checking would pass for each patch in the series, which sometimes required
> some concessions.
>
> Is this one of those cases? Uh, I forget.
>
> If it isn't, its almost certainly the case that I just figured I'd type
> this one function in one place instead of splitting it apart into two
> patches.
>
> I can try to shift the typing later and see what happens if you prefer it
> that way.

Well, you structured the series as "minor code changes to prepare for
type hints", followed by "add type hints".  So that's what I expect.
When patches deviate from what I expect, I go "am I missing something?"

Adding type hints along the way could work, too.  But let's try to stick
to the plan, and add them all in PATCH 16.

>> > +        typ = self.lookup_entity(name, QAPISchemaType)
>> > +        if typ is None:
>> > +            return None
>> > +        assert isinstance(typ, QAPISchemaType)
>> > +        return typ
>>
>> Would
>>
>>            typ = self.lookup_entity(name, QAPISchemaType)
>>            assert isinstance(typ, Optional[QAPISchemaType])
>>            return typ
>>
>> work?
>>
>
> I don't *think* so, Optional isn't a runtime construct.

Let me try...

    $ python
    Python 3.11.5 (main, Aug 28 2023, 00:00:00) [GCC 12.3.1 20230508 (Red Hat 12.3.1-1)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from typing import Optional
    >>> x=None
    >>> isinstance(x, Optional[str])
    True
    >>> 

>                                                         We can combine it
> into "assert x is None or isinstance(x, foo)" though - I believe that's
> used elsewhere in the qapi generator.

>> >
>> >      def resolve_type(self, name, info, what):
>> >          typ = self.lookup_type(name)
>>
>>

Re: [PATCH 08/19] qapi/schema: add static typing and assertions to lookup_type()

[John Snow]

On Wed, Nov 22, 2023 at 7:09 AM Markus Armbruster <armbru@redhat.com> wrote:
>
> John Snow <jsnow@redhat.com> writes:
>
> > On Tue, Nov 21, 2023, 9:21 AM Markus Armbruster <armbru@redhat.com> wrote:
> >
> >> John Snow <jsnow@redhat.com> writes:
> >>
> >> > This function is a bit hard to type as-is; mypy needs some assertions to
> >> > assist with the type narrowing.
> >> >
> >> > Signed-off-by: John Snow <jsnow@redhat.com>
> >> > ---
> >> >  scripts/qapi/schema.py | 8 ++++++--
> >> >  1 file changed, 6 insertions(+), 2 deletions(-)
> >> >
> >> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> >> > index a1094283828..3308f334872 100644
> >> > --- a/scripts/qapi/schema.py
> >> > +++ b/scripts/qapi/schema.py
> >> > @@ -968,8 +968,12 @@ def lookup_entity(self, name, typ=None):
> >> >              return None
> >> >          return ent
> >> >
> >> > -    def lookup_type(self, name):
> >> > -        return self.lookup_entity(name, QAPISchemaType)
> >> > +    def lookup_type(self, name: str) -> Optional[QAPISchemaType]:
> >>
> >> Any particular reason not to delay the type hints until PATCH 16?
> >>
> >
> > I forget. In some cases I did things a little differently so that the type
> > checking would pass for each patch in the series, which sometimes required
> > some concessions.
> >
> > Is this one of those cases? Uh, I forget.
> >
> > If it isn't, its almost certainly the case that I just figured I'd type
> > this one function in one place instead of splitting it apart into two
> > patches.
> >
> > I can try to shift the typing later and see what happens if you prefer it
> > that way.
>
> Well, you structured the series as "minor code changes to prepare for
> type hints", followed by "add type hints".  So that's what I expect.
> When patches deviate from what I expect, I go "am I missing something?"
>
> Adding type hints along the way could work, too.  But let's try to stick
> to the plan, and add them all in PATCH 16.
>
> >> > +        typ = self.lookup_entity(name, QAPISchemaType)
> >> > +        if typ is None:
> >> > +            return None
> >> > +        assert isinstance(typ, QAPISchemaType)
> >> > +        return typ
> >>
> >> Would
> >>
> >>            typ = self.lookup_entity(name, QAPISchemaType)
> >>            assert isinstance(typ, Optional[QAPISchemaType])
> >>            return typ
> >>
> >> work?
> >>
> >
> > I don't *think* so, Optional isn't a runtime construct.
>
> Let me try...
>
>     $ python
>     Python 3.11.5 (main, Aug 28 2023, 00:00:00) [GCC 12.3.1 20230508 (Red Hat 12.3.1-1)] on linux
>     Type "help", "copyright", "credits" or "license" for more information.
>     >>> from typing import Optional
>     >>> x=None
>     >>> isinstance(x, Optional[str])
>     True
>     >>>

Huh. I ... huh!

Well, this apparently only works in Python 3.10!+

TypeError: Subscripted generics cannot be used with class and instance checks

>
> >                                                         We can combine it
> > into "assert x is None or isinstance(x, foo)" though - I believe that's
> > used elsewhere in the qapi generator.
>
> >> >
> >> >      def resolve_type(self, name, info, what):
> >> >          typ = self.lookup_type(name)
> >>
> >>
>

Re: [PATCH 08/19] qapi/schema: add static typing and assertions to lookup_type()

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Wed, Nov 22, 2023 at 7:09 AM Markus Armbruster <armbru@redhat.com> wrote:
>>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > On Tue, Nov 21, 2023, 9:21 AM Markus Armbruster <armbru@redhat.com> wrote:
>> >
>> >> John Snow <jsnow@redhat.com> writes:
>> >>
>> >> > This function is a bit hard to type as-is; mypy needs some assertions to
>> >> > assist with the type narrowing.
>> >> >
>> >> > Signed-off-by: John Snow <jsnow@redhat.com>
>> >> > ---
>> >> >  scripts/qapi/schema.py | 8 ++++++--
>> >> >  1 file changed, 6 insertions(+), 2 deletions(-)
>> >> >
>> >> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
>> >> > index a1094283828..3308f334872 100644
>> >> > --- a/scripts/qapi/schema.py
>> >> > +++ b/scripts/qapi/schema.py
>> >> > @@ -968,8 +968,12 @@ def lookup_entity(self, name, typ=None):
>> >> >              return None
>> >> >          return ent
>> >> >
>> >> > -    def lookup_type(self, name):
>> >> > -        return self.lookup_entity(name, QAPISchemaType)
>> >> > +    def lookup_type(self, name: str) -> Optional[QAPISchemaType]:

[...]

>> >> > +        typ = self.lookup_entity(name, QAPISchemaType)
>> >> > +        if typ is None:
>> >> > +            return None
>> >> > +        assert isinstance(typ, QAPISchemaType)
>> >> > +        return typ
>> >>
>> >> Would
>> >>
>> >>            typ = self.lookup_entity(name, QAPISchemaType)
>> >>            assert isinstance(typ, Optional[QAPISchemaType])
>> >>            return typ
>> >>
>> >> work?
>> >>
>> >
>> > I don't *think* so, Optional isn't a runtime construct.
>>
>> Let me try...
>>
>>     $ python
>>     Python 3.11.5 (main, Aug 28 2023, 00:00:00) [GCC 12.3.1 20230508 (Red Hat 12.3.1-1)] on linux
>>     Type "help", "copyright", "credits" or "license" for more information.
>>     >>> from typing import Optional
>>     >>> x=None
>>     >>> isinstance(x, Optional[str])
>>     True
>>     >>>
>
> Huh. I ... huh!
>
> Well, this apparently only works in Python 3.10!+
>
> TypeError: Subscripted generics cannot be used with class and instance checks

We should be able to use it "soon" after 3.9 reaches EOL, approximately
October 2025.

*Sigh*

>>
>> >                                                         We can combine it
>> > into "assert x is None or isinstance(x, foo)" though - I believe that's
>> > used elsewhere in the qapi generator.
>>
>> >> >
>> >> >      def resolve_type(self, name, info, what):
>> >> >          typ = self.lookup_type(name)
>> >>
>> >>
>>

[PATCH 09/19] qapi/schema: assert info is present when necessary

[John Snow]

QAPISchemaInfo is sometimes defined as an Optional field because
built-in definitions don't *have* a source definition. As a consequence,
there are a few places where we need to assert that it's present because
the root entity definition only suggests it's "Optional".

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 3308f334872..c9a194103e1 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -733,6 +733,7 @@ def describe(self, info):
             else:
                 assert False
 
+        assert info is not None
         if defined_in != info.defn_name:
             return "%s '%s' of %s '%s'" % (role, self.name, meta, defined_in)
         return "%s '%s'" % (role, self.name)
@@ -823,6 +824,7 @@ def __init__(self, name, info, doc, ifcond, features,
         self.coroutine = coroutine
 
     def check(self, schema):
+        assert self.info is not None
         super().check(schema)
         if self._arg_type_name:
             arg_type = schema.resolve_type(
-- 
2.41.0

Re: [PATCH 09/19] qapi/schema: assert info is present when necessary

[Philippe Mathieu-Daudé]

On 16/11/23 02:43, John Snow wrote:
> QAPISchemaInfo is sometimes defined as an Optional field because
> built-in definitions don't *have* a source definition. As a consequence,
> there are a few places where we need to assert that it's present because
> the root entity definition only suggests it's "Optional".
> 
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>   scripts/qapi/schema.py | 2 ++
>   1 file changed, 2 insertions(+)

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>

[PATCH 10/19] qapi/schema: make QAPISchemaArrayType.element_type non-Optional

[John Snow]

This field should always be present and defined. Change this to be a
runtime @property that can emit an error if it's called prior to
check().

This helps simplify typing by avoiding the need to interrogate the value
for None at multiple callsites.

RFC: Yes, this is a slightly different technique than the one I used for
QAPISchemaObjectTypeMember.type; I think I prefer this one as being a
little less hokey, but it is more SLOC. Dealer's choice for which style
wins out -- now you have an example of both.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index c9a194103e1..462acb2bb61 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -366,7 +366,16 @@ def __init__(self, name, info, element_type):
         super().__init__(name, info, None)
         assert isinstance(element_type, str)
         self._element_type_name = element_type
-        self.element_type = None
+        self._element_type: Optional[QAPISchemaType] = None
+
+    @property
+    def element_type(self) -> QAPISchemaType:
+        if self._element_type is None:
+            raise RuntimeError(
+                "QAPISchemaArray has no element_type until "
+                "after check() has been run."
+            )
+        return self._element_type
 
     def need_has_if_optional(self):
         # When FOO is an array, we still need has_FOO to distinguish
@@ -375,7 +384,7 @@ def need_has_if_optional(self):
 
     def check(self, schema):
         super().check(schema)
-        self.element_type = schema.resolve_type(
+        self._element_type = schema.resolve_type(
             self._element_type_name, self.info,
             self.info and self.info.defn_meta)
         assert not isinstance(self.element_type, QAPISchemaArrayType)
-- 
2.41.0

Re: [PATCH 10/19] qapi/schema: make QAPISchemaArrayType.element_type non-Optional

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> This field should always be present and defined. Change this to be a
> runtime @property that can emit an error if it's called prior to
> check().
>
> This helps simplify typing by avoiding the need to interrogate the value
> for None at multiple callsites.
>
> RFC: Yes, this is a slightly different technique than the one I used for
> QAPISchemaObjectTypeMember.type;

In PATCH 04.

>                                  I think I prefer this one as being a
> little less hokey, but it is more SLOC. Dealer's choice for which style
> wins out -- now you have an example of both.

Thanks for letting us see both.

I believe all the extra lines accomplish is a different exception
RuntimeError with a custom message vs. plain AttributeError.

I don't think the more elaborate exception is worth the extra code.

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/schema.py | 13 +++++++++++--
>  1 file changed, 11 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index c9a194103e1..462acb2bb61 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -366,7 +366,16 @@ def __init__(self, name, info, element_type):
>          super().__init__(name, info, None)
>          assert isinstance(element_type, str)
>          self._element_type_name = element_type
> -        self.element_type = None
> +        self._element_type: Optional[QAPISchemaType] = None
> +
> +    @property
> +    def element_type(self) -> QAPISchemaType:
> +        if self._element_type is None:
> +            raise RuntimeError(
> +                "QAPISchemaArray has no element_type until "
> +                "after check() has been run."
> +            )
> +        return self._element_type
>  
>      def need_has_if_optional(self):
>          # When FOO is an array, we still need has_FOO to distinguish
> @@ -375,7 +384,7 @@ def need_has_if_optional(self):
>  
>      def check(self, schema):
>          super().check(schema)
> -        self.element_type = schema.resolve_type(
> +        self._element_type = schema.resolve_type(
>              self._element_type_name, self.info,
>              self.info and self.info.defn_meta)
>          assert not isinstance(self.element_type, QAPISchemaArrayType)

Re: [PATCH 10/19] qapi/schema: make QAPISchemaArrayType.element_type non-Optional

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 2551 bytes --]

On Tue, Nov 21, 2023, 9:28 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > This field should always be present and defined. Change this to be a
> > runtime @property that can emit an error if it's called prior to
> > check().
> >
> > This helps simplify typing by avoiding the need to interrogate the value
> > for None at multiple callsites.
> >
> > RFC: Yes, this is a slightly different technique than the one I used for
> > QAPISchemaObjectTypeMember.type;
>
> In PATCH 04.
>
> >                                  I think I prefer this one as being a
> > little less hokey, but it is more SLOC. Dealer's choice for which style
> > wins out -- now you have an example of both.
>
> Thanks for letting us see both.
>

My pleasure ;)


> I believe all the extra lines accomplish is a different exception
> RuntimeError with a custom message vs. plain AttributeError.
>
> I don't think the more elaborate exception is worth the extra code.
>

Hmm, shame. You're the boss :)


> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  scripts/qapi/schema.py | 13 +++++++++++--
> >  1 file changed, 11 insertions(+), 2 deletions(-)
> >
> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> > index c9a194103e1..462acb2bb61 100644
> > --- a/scripts/qapi/schema.py
> > +++ b/scripts/qapi/schema.py
> > @@ -366,7 +366,16 @@ def __init__(self, name, info, element_type):
> >          super().__init__(name, info, None)
> >          assert isinstance(element_type, str)
> >          self._element_type_name = element_type
> > -        self.element_type = None
> > +        self._element_type: Optional[QAPISchemaType] = None
> > +
> > +    @property
> > +    def element_type(self) -> QAPISchemaType:
> > +        if self._element_type is None:
> > +            raise RuntimeError(
> > +                "QAPISchemaArray has no element_type until "
> > +                "after check() has been run."
> > +            )
> > +        return self._element_type
> >
> >      def need_has_if_optional(self):
> >          # When FOO is an array, we still need has_FOO to distinguish
> > @@ -375,7 +384,7 @@ def need_has_if_optional(self):
> >
> >      def check(self, schema):
> >          super().check(schema)
> > -        self.element_type = schema.resolve_type(
> > +        self._element_type = schema.resolve_type(
> >              self._element_type_name, self.info,
> >              self.info and self.info.defn_meta)
> >          assert not isinstance(self.element_type, QAPISchemaArrayType)
>
>

[-- Attachment #2: Type: text/html, Size: 4113 bytes --]

[PATCH 11/19] qapi/schema: fix QAPISchemaArrayType.check's call to resolve_type

[John Snow]

There's more conditionals in here than we can reasonably pack into a
terse little statement, so break it apart into something more explicit.

(When would a built-in array ever cause a QAPISemError? I don't know,
maybe never - but the type system wasn't happy all the same.)

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 462acb2bb61..164d86c4064 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -384,9 +384,16 @@ def need_has_if_optional(self):
 
     def check(self, schema):
         super().check(schema)
+
+        if self.info:
+            assert self.info.defn_meta  # guaranteed to be set by expr.py
+            what = self.info.defn_meta
+        else:
+            what = 'built-in array'
+
         self._element_type = schema.resolve_type(
-            self._element_type_name, self.info,
-            self.info and self.info.defn_meta)
+            self._element_type_name, self.info, what
+        )
         assert not isinstance(self.element_type, QAPISchemaArrayType)
 
     def set_module(self, schema):
-- 
2.41.0

Re: [PATCH 11/19] qapi/schema: fix QAPISchemaArrayType.check's call to resolve_type

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> There's more conditionals in here than we can reasonably pack into a
> terse little statement, so break it apart into something more explicit.
>
> (When would a built-in array ever cause a QAPISemError? I don't know,
> maybe never - but the type system wasn't happy all the same.)
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/schema.py | 11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 462acb2bb61..164d86c4064 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -384,9 +384,16 @@ def need_has_if_optional(self):
>  
>      def check(self, schema):
>          super().check(schema)
> +
> +        if self.info:
> +            assert self.info.defn_meta  # guaranteed to be set by expr.py
> +            what = self.info.defn_meta
> +        else:
> +            what = 'built-in array'
> +
>          self._element_type = schema.resolve_type(
> -            self._element_type_name, self.info,
> -            self.info and self.info.defn_meta)
> +            self._element_type_name, self.info, what
> +        )
>          assert not isinstance(self.element_type, QAPISchemaArrayType)
>  
>      def set_module(self, schema):

What problem are you solving here?

Re: [PATCH 11/19] qapi/schema: fix QAPISchemaArrayType.check's call to resolve_type

[John Snow]

On Wed, Nov 22, 2023 at 7:59 AM Markus Armbruster <armbru@redhat.com> wrote:
>
> John Snow <jsnow@redhat.com> writes:
>
> > There's more conditionals in here than we can reasonably pack into a
> > terse little statement, so break it apart into something more explicit.
> >
> > (When would a built-in array ever cause a QAPISemError? I don't know,
> > maybe never - but the type system wasn't happy all the same.)
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  scripts/qapi/schema.py | 11 +++++++++--
> >  1 file changed, 9 insertions(+), 2 deletions(-)
> >
> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> > index 462acb2bb61..164d86c4064 100644
> > --- a/scripts/qapi/schema.py
> > +++ b/scripts/qapi/schema.py
> > @@ -384,9 +384,16 @@ def need_has_if_optional(self):
> >
> >      def check(self, schema):
> >          super().check(schema)
> > +
> > +        if self.info:
> > +            assert self.info.defn_meta  # guaranteed to be set by expr.py
> > +            what = self.info.defn_meta
> > +        else:
> > +            what = 'built-in array'
> > +
> >          self._element_type = schema.resolve_type(
> > -            self._element_type_name, self.info,
> > -            self.info and self.info.defn_meta)
> > +            self._element_type_name, self.info, what
> > +        )
> >          assert not isinstance(self.element_type, QAPISchemaArrayType)
> >
> >      def set_module(self, schema):
>
> What problem are you solving here?
>

1. "self.info and self.info.defn_meta" is the wrong type ifn't self.info
2. self.info.defn_meta is *also* not guaranteed by static types

ultimately: we need to assert self.info and self.info.defn_meta both;
but it's possible (?) that we don't have self.info in the case that
we're a built-in array, so I handle that.

Re: [PATCH 11/19] qapi/schema: fix QAPISchemaArrayType.check's call to resolve_type

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Wed, Nov 22, 2023 at 7:59 AM Markus Armbruster <armbru@redhat.com> wrote:
>>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > There's more conditionals in here than we can reasonably pack into a
>> > terse little statement, so break it apart into something more explicit.
>> >
>> > (When would a built-in array ever cause a QAPISemError? I don't know,
>> > maybe never - but the type system wasn't happy all the same.)
>> >
>> > Signed-off-by: John Snow <jsnow@redhat.com>
>> > ---
>> >  scripts/qapi/schema.py | 11 +++++++++--
>> >  1 file changed, 9 insertions(+), 2 deletions(-)
>> >
>> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
>> > index 462acb2bb61..164d86c4064 100644
>> > --- a/scripts/qapi/schema.py
>> > +++ b/scripts/qapi/schema.py
>> > @@ -384,9 +384,16 @@ def need_has_if_optional(self):
>> >
>> >      def check(self, schema):
>> >          super().check(schema)
>> > +
>> > +        if self.info:
>> > +            assert self.info.defn_meta  # guaranteed to be set by expr.py
>> > +            what = self.info.defn_meta
>> > +        else:
>> > +            what = 'built-in array'
>> > +
>> >          self._element_type = schema.resolve_type(
>> > -            self._element_type_name, self.info,
>> > -            self.info and self.info.defn_meta)
>> > +            self._element_type_name, self.info, what
>> > +        )
0>> >          assert not isinstance(self.element_type, QAPISchemaArrayType)
>> >
>> >      def set_module(self, schema):
>>
>> What problem are you solving here?
>>
>
> 1. "self.info and self.info.defn_meta" is the wrong type ifn't self.info

self.info is Optional[QAPISourceInfo].

When self.info, then self.info.defn_meta is is Optional[str].

Naive me expects self.info and self.info.defn_meta to be Optional[str].
Playing with mypy...  it seems to be Union[QAPISourceInfo, None, str].
Type inference too weak.

> 2. self.info.defn_meta is *also* not guaranteed by static types

Yes.  We know it's not None ("guaranteed to be set by expr.py"), but the
type system doesn't.

> ultimately: we need to assert self.info and self.info.defn_meta both;
> but it's possible (?) that we don't have self.info in the case that
> we're a built-in array, so I handle that.

This bring us back to the question in your commit message: "When would a
built-in array ever cause a QAPISemError?"  Short answer: never.

Long answer.  We're dealing with a *specific* QAPISemError here, namely
.resolve_type()'s "uses unknown type".  If this happens for a built-in
array, it's a programming error.

Let's commit such an error to see what happens: stick

        self._make_array_type('xxx', None)

Dies like this:

    Traceback (most recent call last):
      File "/work/armbru/qemu/scripts/qapi/main.py", line 94, in main
        generate(args.schema,
      File "/work/armbru/qemu/scripts/qapi/main.py", line 50, in generate
        schema = QAPISchema(schema_file)
                 ^^^^^^^^^^^^^^^^^^^^^^^
      File "/work/armbru/qemu/scripts/qapi/schema.py", line 938, in __init__
        self.check()
      File "/work/armbru/qemu/scripts/qapi/schema.py", line 1225, in check
        ent.check(self)
      File "/work/armbru/qemu/scripts/qapi/schema.py", line 373, in check
        self.element_type = schema.resolve_type(
                            ^^^^^^^^^^^^^^^^^^^^
      File "/work/armbru/qemu/scripts/qapi/schema.py", line 973, in resolve_type
        raise QAPISemError(
    qapi.error.QAPISemError: <exception str() failed>

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/work/armbru/qemu/scripts/qapi-gen.py", line 19, in <module>
        sys.exit(main.main())
                 ^^^^^^^^^^^
      File "/work/armbru/qemu/scripts/qapi/main.py", line 101, in main
        print(err, file=sys.stderr)
      File "/work/armbru/qemu/scripts/qapi/error.py", line 41, in __str__
        assert self.info is not None
               ^^^^^^^^^^^^^^^^^^^^^
    AssertionError

Same before and after your patch.  The patch's change of what=None to
what='built-in array' has no effect.

Here's a slightly simpler patch:

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 46004689f0..feb0023d25 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -479,7 +479,7 @@ def check(self, schema: QAPISchema) -> None:
         super().check(schema)
         self._element_type = schema.resolve_type(
             self._element_type_name, self.info,
-            self.info and self.info.defn_meta)
+            self.info.defn_meta if self.info else None)
         assert not isinstance(self.element_type, QAPISchemaArrayType)
 
     def set_module(self, schema: QAPISchema) -> None:
@@ -1193,7 +1193,7 @@ def resolve_type(
         self,
         name: str,
         info: Optional[QAPISourceInfo],
-        what: Union[str, Callable[[Optional[QAPISourceInfo]], str]],
+        what: Union[None, str, Callable[[Optional[QAPISourceInfo]], str]],
     ) -> QAPISchemaType:
         typ = self.lookup_type(name)
         if not typ:

The first hunk works around mypy's type inference weakness.  It rewrites

    A and B

as

    B if A else A

and then partially evaluates to

    B if A else None

exploiting the fact that falsy A can only be None.  It replaces this
patch.

The second hunk corrects .resolve_type()'s typing to accept what=None.
It's meant to be squashed into PATCH 16.

What do you think?

Re: [PATCH 11/19] qapi/schema: fix QAPISchemaArrayType.check's call to resolve_type

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 7596 bytes --]

On Thu, Nov 23, 2023, 8:03 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > On Wed, Nov 22, 2023 at 7:59 AM Markus Armbruster <armbru@redhat.com>
> wrote:
> >>
> >> John Snow <jsnow@redhat.com> writes:
> >>
> >> > There's more conditionals in here than we can reasonably pack into a
> >> > terse little statement, so break it apart into something more
> explicit.
> >> >
> >> > (When would a built-in array ever cause a QAPISemError? I don't know,
> >> > maybe never - but the type system wasn't happy all the same.)
> >> >
> >> > Signed-off-by: John Snow <jsnow@redhat.com>
> >> > ---
> >> >  scripts/qapi/schema.py | 11 +++++++++--
> >> >  1 file changed, 9 insertions(+), 2 deletions(-)
> >> >
> >> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> >> > index 462acb2bb61..164d86c4064 100644
> >> > --- a/scripts/qapi/schema.py
> >> > +++ b/scripts/qapi/schema.py
> >> > @@ -384,9 +384,16 @@ def need_has_if_optional(self):
> >> >
> >> >      def check(self, schema):
> >> >          super().check(schema)
> >> > +
> >> > +        if self.info:
> >> > +            assert self.info.defn_meta  # guaranteed to be set by
> expr.py
> >> > +            what = self.info.defn_meta
> >> > +        else:
> >> > +            what = 'built-in array'
> >> > +
> >> >          self._element_type = schema.resolve_type(
> >> > -            self._element_type_name, self.info,
> >> > -            self.info and self.info.defn_meta)
> >> > +            self._element_type_name, self.info, what
> >> > +        )
> 0>> >          assert not isinstance(self.element_type,
> QAPISchemaArrayType)
> >> >
> >> >      def set_module(self, schema):
> >>
> >> What problem are you solving here?
> >>
> >
> > 1. "self.info and self.info.defn_meta" is the wrong type ifn't self.info
>
> self.info is Optional[QAPISourceInfo].
>
> When self.info, then self.info.defn_meta is is Optional[str].
>
> Naive me expects self.info and self.info.defn_meta to be Optional[str].
> Playing with mypy...  it seems to be Union[QAPISourceInfo, None, str].
> Type inference too weak.
>

I think my expectations match yours: "x and y" should return either x or y,
so the resulting type would naively be Union[X | Y], which would indeed be
Union[QAPISourceInfo | None | str], but:

If QAPISourceInfo is *false-y*, but not None, it'd be possible for the
expression to yield a QAPISourceInfo. mypy does not understand that
QAPISourceInfo can never be false-y.

(That I know of. Maybe there's a trick to annotate it. I like your solution
below better anyway, just curious about the exact nature of this
limitation.)


> > 2. self.info.defn_meta is *also* not guaranteed by static types
>
> Yes.  We know it's not None ("guaranteed to be set by expr.py"), but the
> type system doesn't.
>

Mmhmm.


> > ultimately: we need to assert self.info and self.info.defn_meta both;
> > but it's possible (?) that we don't have self.info in the case that
> > we're a built-in array, so I handle that.
>
> This bring us back to the question in your commit message: "When would a
> built-in array ever cause a QAPISemError?"  Short answer: never.
>

Right, okay. I just couldn't guarantee it statically. I knew this patch was
a little bananas, sorry for tossing you the stinkbomb.


> Long answer.  We're dealing with a *specific* QAPISemError here, namely
> .resolve_type()'s "uses unknown type".  If this happens for a built-in
> array, it's a programming error.
>
> Let's commit such an error to see what happens: stick
>
>         self._make_array_type('xxx', None)
>
> Dies like this:
>
>     Traceback (most recent call last):
>       File "/work/armbru/qemu/scripts/qapi/main.py", line 94, in main
>         generate(args.schema,
>       File "/work/armbru/qemu/scripts/qapi/main.py", line 50, in generate
>         schema = QAPISchema(schema_file)
>                  ^^^^^^^^^^^^^^^^^^^^^^^
>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 938, in
> __init__
>         self.check()
>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 1225, in check
>         ent.check(self)
>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 373, in check
>         self.element_type = schema.resolve_type(
>                             ^^^^^^^^^^^^^^^^^^^^
>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 973, in
> resolve_type
>         raise QAPISemError(
>     qapi.error.QAPISemError: <exception str() failed>
>
>     During handling of the above exception, another exception occurred:
>
>     Traceback (most recent call last):
>       File "/work/armbru/qemu/scripts/qapi-gen.py", line 19, in <module>
>         sys.exit(main.main())
>                  ^^^^^^^^^^^
>       File "/work/armbru/qemu/scripts/qapi/main.py", line 101, in main
>         print(err, file=sys.stderr)
>       File "/work/armbru/qemu/scripts/qapi/error.py", line 41, in __str__
>         assert self.info is not None
>                ^^^^^^^^^^^^^^^^^^^^^
>     AssertionError
>
> Same before and after your patch.  The patch's change of what=None to
> what='built-in array' has no effect.
>
> Here's a slightly simpler patch:
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 46004689f0..feb0023d25 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -479,7 +479,7 @@ def check(self, schema: QAPISchema) -> None:
>          super().check(schema)
>          self._element_type = schema.resolve_type(
>              self._element_type_name, self.info,
> -            self.info and self.info.defn_meta)
> +            self.info.defn_meta if self.info else None)
>

Yep.

         assert not isinstance(self.element_type, QAPISchemaArrayType)
>
>      def set_module(self, schema: QAPISchema) -> None:
> @@ -1193,7 +1193,7 @@ def resolve_type(
>          self,
>          name: str,
>          info: Optional[QAPISourceInfo],
> -        what: Union[str, Callable[[Optional[QAPISourceInfo]], str]],
> +        what: Union[None, str, Callable[[Optional[QAPISourceInfo]], str]],
>      ) -> QAPISchemaType:
>          typ = self.lookup_type(name)
>          if not typ:
>
> The first hunk works around mypy's type inference weakness.  It rewrites
>
>     A and B
>
> as
>
>     B if A else A
>
> and then partially evaluates to
>
>     B if A else None
>
> exploiting the fact that falsy A can only be None.  It replaces this
> patch.
>

Sounds good to me!


> The second hunk corrects .resolve_type()'s typing to accept what=None.
> It's meant to be squashed into PATCH 16.
>
> What do you think?
>

I'm on my mobile again, but at a glance I like it. Except that I'm a little
reluctant to allow what to be None if this is the *only* caller known to
possibly do it, and only in a circumstance that we assert elsewhere that it
should never happen.

Can we do:

what = self.info.defn_meta if self.info else None
assert what [is not None]  # Depending on taste

instead?

No sem error, no new unit test needed, assertion provides the correct frame
of mind (programmer error), stronger typing on resolve_type.

(I really love eliminating None when I can as a rule because I like how
much more it tells you about the nature of all callers, it's a much
stronger decree. Worth pursuing where you can, IMO, but I'm not gonna die
on the hill for a patch like this - just sharing my tendencies for
discussion.)

--js

>

[-- Attachment #2: Type: text/html, Size: 12313 bytes --]

Re: [PATCH 11/19] qapi/schema: fix QAPISchemaArrayType.check's call to resolve_type

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Thu, Nov 23, 2023, 8:03 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > On Wed, Nov 22, 2023 at 7:59 AM Markus Armbruster <armbru@redhat.com> wrote:
>> >>
>> >> John Snow <jsnow@redhat.com> writes:
>> >>
>> >> > There's more conditionals in here than we can reasonably pack into a
>> >> > terse little statement, so break it apart into something more> explicit.
>> >> >
>> >> > (When would a built-in array ever cause a QAPISemError? I don't know,
>> >> > maybe never - but the type system wasn't happy all the same.)
>> >> >
>> >> > Signed-off-by: John Snow <jsnow@redhat.com>
>> >> > ---
>> >> >  scripts/qapi/schema.py | 11 +++++++++--
>> >> >  1 file changed, 9 insertions(+), 2 deletions(-)
>> >> >
>> >> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
>> >> > index 462acb2bb61..164d86c4064 100644
>> >> > --- a/scripts/qapi/schema.py
>> >> > +++ b/scripts/qapi/schema.py
>> >> > @@ -384,9 +384,16 @@ def need_has_if_optional(self):
>> >> >
>> >> >      def check(self, schema):
>> >> >          super().check(schema)
>> >> > +
>> >> > +        if self.info:
>> >> > +            assert self.info.defn_meta  # guaranteed to be set by> expr.py
>> >> > +            what = self.info.defn_meta
>> >> > +        else:
>> >> > +            what = 'built-in array'
>> >> > +
>> >> >          self._element_type = schema.resolve_type(
>> >> > -            self._element_type_name, self.info,
>> >> > -            self.info and self.info.defn_meta)
>> >> > +            self._element_type_name, self.info, what
>> >> > +        )
>> 0>> >          assert not isinstance(self.element_type, QAPISchemaArrayType)
>> >> >
>> >> >      def set_module(self, schema):
>> >>
>> >> What problem are you solving here?
>> >>
>> >
>> > 1. "self.info and self.info.defn_meta" is the wrong type ifn't self.info
>>
>> self.info is Optional[QAPISourceInfo].
>>
>> When self.info, then self.info.defn_meta is is Optional[str].
>>
>> Naive me expects self.info and self.info.defn_meta to be Optional[str].
>> Playing with mypy...  it seems to be Union[QAPISourceInfo, None, str].
>> Type inference too weak.
>>
>
> I think my expectations match yours: "x and y" should return either x or y,
> so the resulting type would naively be Union[X | Y], which would indeed be
> Union[QAPISourceInfo | None | str], but:
>
> If QAPISourceInfo is *false-y*, but not None, it'd be possible for the
> expression to yield a QAPISourceInfo. mypy does not understand that
> QAPISourceInfo can never be false-y.
>
> (That I know of. Maybe there's a trick to annotate it. I like your solution
> below better anyway, just curious about the exact nature of this
> limitation.)
>
>
>> > 2. self.info.defn_meta is *also* not guaranteed by static types
>>
>> Yes.  We know it's not None ("guaranteed to be set by expr.py"), but the
>> type system doesn't.
>>
>
> Mmhmm.
>
>
>> > ultimately: we need to assert self.info and self.info.defn_meta both;
>> > but it's possible (?) that we don't have self.info in the case that
>> > we're a built-in array, so I handle that.
>>
>> This bring us back to the question in your commit message: "When would a
>> built-in array ever cause a QAPISemError?"  Short answer: never.
>
> Right, okay. I just couldn't guarantee it statically. I knew this patch was
> a little bananas, sorry for tossing you the stinkbomb.

No need to be sorry!  Feels like an efficient way to collaborate with
me.

>> Long answer.  We're dealing with a *specific* QAPISemError here, namely
>> .resolve_type()'s "uses unknown type".  If this happens for a built-in
>> array, it's a programming error.
>>
>> Let's commit such an error to see what happens: stick
>>
>>         self._make_array_type('xxx', None)
>>
>> Dies like this:
>>
>>     Traceback (most recent call last):
>>       File "/work/armbru/qemu/scripts/qapi/main.py", line 94, in main
>>         generate(args.schema,
>>       File "/work/armbru/qemu/scripts/qapi/main.py", line 50, in generate
>>         schema = QAPISchema(schema_file)
>>                  ^^^^^^^^^^^^^^^^^^^^^^^
>>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 938, in
>> __init__
>>         self.check()
>>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 1225, in check
>>         ent.check(self)
>>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 373, in check
>>         self.element_type = schema.resolve_type(
>>                             ^^^^^^^^^^^^^^^^^^^^
>>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 973, in
>> resolve_type
>>         raise QAPISemError(
>>     qapi.error.QAPISemError: <exception str() failed>
>>
>>     During handling of the above exception, another exception occurred:
>>
>>     Traceback (most recent call last):
>>       File "/work/armbru/qemu/scripts/qapi-gen.py", line 19, in <module>
>>         sys.exit(main.main())
>>                  ^^^^^^^^^^^
>>       File "/work/armbru/qemu/scripts/qapi/main.py", line 101, in main
>>         print(err, file=sys.stderr)
>>       File "/work/armbru/qemu/scripts/qapi/error.py", line 41, in __str__
>>         assert self.info is not None
>>                ^^^^^^^^^^^^^^^^^^^^^
>>     AssertionError
>>
>> Same before and after your patch.  The patch's change of what=None to
>> what='built-in array' has no effect.
>>
>> Here's a slightly simpler patch:
>>
>> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
>> index 46004689f0..feb0023d25 100644
>> --- a/scripts/qapi/schema.py
>> +++ b/scripts/qapi/schema.py
>> @@ -479,7 +479,7 @@ def check(self, schema: QAPISchema) -> None:
>>          super().check(schema)
>>          self._element_type = schema.resolve_type(
>>              self._element_type_name, self.info,
>> -            self.info and self.info.defn_meta)
>> +            self.info.defn_meta if self.info else None)
>>
>
> Yep.
>
>          assert not isinstance(self.element_type, QAPISchemaArrayType)
>>
>>      def set_module(self, schema: QAPISchema) -> None:
>> @@ -1193,7 +1193,7 @@ def resolve_type(
>>          self,
>>          name: str,
>>          info: Optional[QAPISourceInfo],
>> -        what: Union[str, Callable[[Optional[QAPISourceInfo]], str]],
>> +        what: Union[None, str, Callable[[Optional[QAPISourceInfo]], str]],
>>      ) -> QAPISchemaType:
>>          typ = self.lookup_type(name)
>>          if not typ:
>>
>> The first hunk works around mypy's type inference weakness.  It rewrites
>>
>>     A and B
>>
>> as
>>
>>     B if A else A
>>
>> and then partially evaluates to
>>
>>     B if A else None
>>
>> exploiting the fact that falsy A can only be None.  It replaces this
>> patch.
>
> Sounds good to me!

Does it need a comment explaining the somewhat awkward coding?  Probably
not.

>> The second hunk corrects .resolve_type()'s typing to accept what=None.
>> It's meant to be squashed into PATCH 16.
>>
>> What do you think?
>>
>
> I'm on my mobile again, but at a glance I like it. Except that I'm a little
> reluctant to allow what to be None if this is the *only* caller known to
> possibly do it, and only in a circumstance that we assert elsewhere that it
> should never happen.
>
> Can we do:
>
> what = self.info.defn_meta if self.info else None
> assert what [is not None]  # Depending on taste
>
> instead?
>
> No sem error, no new unit test needed, assertion provides the correct frame
> of mind (programmer error), stronger typing on resolve_type.
>
> (I really love eliminating None when I can as a rule because I like how
> much more it tells you about the nature of all callers, it's a much
> stronger decree. Worth pursuing where you can, IMO, but I'm not gonna die
> on the hill for a patch like this - just sharing my tendencies for
> discussion.)

Suggest you post the patch, so I can see it more easily in context.

Re: [PATCH 11/19] qapi/schema: fix QAPISchemaArrayType.check's call to resolve_type

[John Snow]

On Thu, Jan 11, 2024 at 4:33 AM Markus Armbruster <armbru@redhat.com> wrote:
>
> John Snow <jsnow@redhat.com> writes:
>
> > On Thu, Nov 23, 2023, 8:03 AM Markus Armbruster <armbru@redhat.com> wrote:
> >
> >> John Snow <jsnow@redhat.com> writes:
> >>
> >> > On Wed, Nov 22, 2023 at 7:59 AM Markus Armbruster <armbru@redhat.com> wrote:
> >> >>
> >> >> John Snow <jsnow@redhat.com> writes:
> >> >>
> >> >> > There's more conditionals in here than we can reasonably pack into a
> >> >> > terse little statement, so break it apart into something more> explicit.
> >> >> >
> >> >> > (When would a built-in array ever cause a QAPISemError? I don't know,
> >> >> > maybe never - but the type system wasn't happy all the same.)
> >> >> >
> >> >> > Signed-off-by: John Snow <jsnow@redhat.com>
> >> >> > ---
> >> >> >  scripts/qapi/schema.py | 11 +++++++++--
> >> >> >  1 file changed, 9 insertions(+), 2 deletions(-)
> >> >> >
> >> >> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> >> >> > index 462acb2bb61..164d86c4064 100644
> >> >> > --- a/scripts/qapi/schema.py
> >> >> > +++ b/scripts/qapi/schema.py
> >> >> > @@ -384,9 +384,16 @@ def need_has_if_optional(self):
> >> >> >
> >> >> >      def check(self, schema):
> >> >> >          super().check(schema)
> >> >> > +
> >> >> > +        if self.info:
> >> >> > +            assert self.info.defn_meta  # guaranteed to be set by> expr.py
> >> >> > +            what = self.info.defn_meta
> >> >> > +        else:
> >> >> > +            what = 'built-in array'
> >> >> > +
> >> >> >          self._element_type = schema.resolve_type(
> >> >> > -            self._element_type_name, self.info,
> >> >> > -            self.info and self.info.defn_meta)
> >> >> > +            self._element_type_name, self.info, what
> >> >> > +        )
> >> 0>> >          assert not isinstance(self.element_type, QAPISchemaArrayType)
> >> >> >
> >> >> >      def set_module(self, schema):
> >> >>
> >> >> What problem are you solving here?
> >> >>
> >> >
> >> > 1. "self.info and self.info.defn_meta" is the wrong type ifn't self.info
> >>
> >> self.info is Optional[QAPISourceInfo].
> >>
> >> When self.info, then self.info.defn_meta is is Optional[str].
> >>
> >> Naive me expects self.info and self.info.defn_meta to be Optional[str].
> >> Playing with mypy...  it seems to be Union[QAPISourceInfo, None, str].
> >> Type inference too weak.
> >>
> >
> > I think my expectations match yours: "x and y" should return either x or y,
> > so the resulting type would naively be Union[X | Y], which would indeed be
> > Union[QAPISourceInfo | None | str], but:
> >
> > If QAPISourceInfo is *false-y*, but not None, it'd be possible for the
> > expression to yield a QAPISourceInfo. mypy does not understand that
> > QAPISourceInfo can never be false-y.
> >
> > (That I know of. Maybe there's a trick to annotate it. I like your solution
> > below better anyway, just curious about the exact nature of this
> > limitation.)
> >
> >
> >> > 2. self.info.defn_meta is *also* not guaranteed by static types
> >>
> >> Yes.  We know it's not None ("guaranteed to be set by expr.py"), but the
> >> type system doesn't.
> >>
> >
> > Mmhmm.
> >
> >
> >> > ultimately: we need to assert self.info and self.info.defn_meta both;
> >> > but it's possible (?) that we don't have self.info in the case that
> >> > we're a built-in array, so I handle that.
> >>
> >> This bring us back to the question in your commit message: "When would a
> >> built-in array ever cause a QAPISemError?"  Short answer: never.
> >
> > Right, okay. I just couldn't guarantee it statically. I knew this patch was
> > a little bananas, sorry for tossing you the stinkbomb.
>
> No need to be sorry!  Feels like an efficient way to collaborate with
> me.
>
> >> Long answer.  We're dealing with a *specific* QAPISemError here, namely
> >> .resolve_type()'s "uses unknown type".  If this happens for a built-in
> >> array, it's a programming error.
> >>
> >> Let's commit such an error to see what happens: stick
> >>
> >>         self._make_array_type('xxx', None)
> >>
> >> Dies like this:
> >>
> >>     Traceback (most recent call last):
> >>       File "/work/armbru/qemu/scripts/qapi/main.py", line 94, in main
> >>         generate(args.schema,
> >>       File "/work/armbru/qemu/scripts/qapi/main.py", line 50, in generate
> >>         schema = QAPISchema(schema_file)
> >>                  ^^^^^^^^^^^^^^^^^^^^^^^
> >>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 938, in
> >> __init__
> >>         self.check()
> >>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 1225, in check
> >>         ent.check(self)
> >>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 373, in check
> >>         self.element_type = schema.resolve_type(
> >>                             ^^^^^^^^^^^^^^^^^^^^
> >>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 973, in
> >> resolve_type
> >>         raise QAPISemError(
> >>     qapi.error.QAPISemError: <exception str() failed>
> >>
> >>     During handling of the above exception, another exception occurred:
> >>
> >>     Traceback (most recent call last):
> >>       File "/work/armbru/qemu/scripts/qapi-gen.py", line 19, in <module>
> >>         sys.exit(main.main())
> >>                  ^^^^^^^^^^^
> >>       File "/work/armbru/qemu/scripts/qapi/main.py", line 101, in main
> >>         print(err, file=sys.stderr)
> >>       File "/work/armbru/qemu/scripts/qapi/error.py", line 41, in __str__
> >>         assert self.info is not None
> >>                ^^^^^^^^^^^^^^^^^^^^^
> >>     AssertionError
> >>
> >> Same before and after your patch.  The patch's change of what=None to
> >> what='built-in array' has no effect.
> >>
> >> Here's a slightly simpler patch:
> >>
> >> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> >> index 46004689f0..feb0023d25 100644
> >> --- a/scripts/qapi/schema.py
> >> +++ b/scripts/qapi/schema.py
> >> @@ -479,7 +479,7 @@ def check(self, schema: QAPISchema) -> None:
> >>          super().check(schema)
> >>          self._element_type = schema.resolve_type(
> >>              self._element_type_name, self.info,
> >> -            self.info and self.info.defn_meta)
> >> +            self.info.defn_meta if self.info else None)
> >>
> >
> > Yep.
> >
> >          assert not isinstance(self.element_type, QAPISchemaArrayType)
> >>
> >>      def set_module(self, schema: QAPISchema) -> None:
> >> @@ -1193,7 +1193,7 @@ def resolve_type(
> >>          self,
> >>          name: str,
> >>          info: Optional[QAPISourceInfo],
> >> -        what: Union[str, Callable[[Optional[QAPISourceInfo]], str]],
> >> +        what: Union[None, str, Callable[[Optional[QAPISourceInfo]], str]],
> >>      ) -> QAPISchemaType:
> >>          typ = self.lookup_type(name)
> >>          if not typ:
> >>
> >> The first hunk works around mypy's type inference weakness.  It rewrites
> >>
> >>     A and B
> >>
> >> as
> >>
> >>     B if A else A
> >>
> >> and then partially evaluates to
> >>
> >>     B if A else None
> >>
> >> exploiting the fact that falsy A can only be None.  It replaces this
> >> patch.
> >
> > Sounds good to me!
>
> Does it need a comment explaining the somewhat awkward coding?  Probably
> not.

git blame should cover it for the curious; otherwise if someone
decides to simplify it they'll find out quickly enough when the test
chirps. (Oh, assuming I actually get this into a test suite soon...)

--js

>
> >> The second hunk corrects .resolve_type()'s typing to accept what=None.
> >> It's meant to be squashed into PATCH 16.
> >>
> >> What do you think?
> >>
> >
> > I'm on my mobile again, but at a glance I like it. Except that I'm a little
> > reluctant to allow what to be None if this is the *only* caller known to
> > possibly do it, and only in a circumstance that we assert elsewhere that it
> > should never happen.
> >
> > Can we do:
> >
> > what = self.info.defn_meta if self.info else None
> > assert what [is not None]  # Depending on taste
> >
> > instead?
> >
> > No sem error, no new unit test needed, assertion provides the correct frame
> > of mind (programmer error), stronger typing on resolve_type.
> >
> > (I really love eliminating None when I can as a rule because I like how
> > much more it tells you about the nature of all callers, it's a much
> > stronger decree. Worth pursuing where you can, IMO, but I'm not gonna die
> > on the hill for a patch like this - just sharing my tendencies for
> > discussion.)
>
> Suggest you post the patch, so I can see it more easily in context.

Kay, coming right up.

--js

[PATCH 12/19] qapi/schema: split "checked" field into "checking" and "checked"

[John Snow]

Differentiate between "actively in the process of checking" and
"checking has completed". This allows us to clean up the types of some
internal fields such as QAPISchemaObjectType's members field which
currently uses "None" as a canary for determining if check has
completed.

This simplifies the typing from a cumbersome Optional[List[T]] to merely
a List[T], which is more pythonic: it is safe to iterate over an empty
list with "for x in []" whereas with an Optional[List[T]] you have to
rely on the more cumbersome "if L: for x in L: ..."

RFC: are we guaranteed to have members here? can we just use "if
members" without adding the new field?

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 24 +++++++++++++++---------
 1 file changed, 15 insertions(+), 9 deletions(-)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 164d86c4064..200bc0730d6 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -18,7 +18,7 @@
 from collections import OrderedDict
 import os
 import re
-from typing import List, Optional
+from typing import List, Optional, cast
 
 from .common import (
     POINTER_SUFFIX,
@@ -447,22 +447,24 @@ def __init__(self, name, info, doc, ifcond, features,
         self.base = None
         self.local_members = local_members
         self.variants = variants
-        self.members = None
+        self.members: List[QAPISchemaObjectTypeMember] = []
+        self._checking = False
 
     def check(self, schema):
         # This calls another type T's .check() exactly when the C
         # struct emitted by gen_object() contains that T's C struct
         # (pointers don't count).
-        if self.members is not None:
-            # A previous .check() completed: nothing to do
-            return
-        if self._checked:
+        if self._checking:
             # Recursed: C struct contains itself
             raise QAPISemError(self.info,
                                "object %s contains itself" % self.name)
+        if self._checked:
+            # A previous .check() completed: nothing to do
+            return
 
+        self._checking = True
         super().check(schema)
-        assert self._checked and self.members is None
+        assert self._checked and not self.members
 
         seen = OrderedDict()
         if self._base_name:
@@ -479,13 +481,17 @@ def check(self, schema):
         for m in self.local_members:
             m.check(schema)
             m.check_clash(self.info, seen)
-        members = seen.values()
+
+        # check_clash is abstract, but local_members is asserted to be
+        # List[QAPISchemaObjectTypeMember]. Cast to the narrower type.
+        members = cast(List[QAPISchemaObjectTypeMember], list(seen.values()))
 
         if self.variants:
             self.variants.check(schema, seen)
             self.variants.check_clash(self.info, seen)
 
-        self.members = members  # mark completed
+        self.members = members
+        self._checking = False  # mark completed
 
     # Check that the members of this type do not cause duplicate JSON members,
     # and update seen to track the members seen so far. Report any errors
-- 
2.41.0

Re: [PATCH 12/19] qapi/schema: split "checked" field into "checking" and "checked"

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Differentiate between "actively in the process of checking" and
> "checking has completed". This allows us to clean up the types of some
> internal fields such as QAPISchemaObjectType's members field which
> currently uses "None" as a canary for determining if check has
> completed.

Certain members become valid only after .check().  Two ways to code
that:

1. Assign to such members only in .check().  If you try to use them
before .check(), AttributeError.  Nice.  Drawback: pylint is unhappy,
W0201 attribute-defined-outside-init.

2. Assign None in .__init__(), and the real value in .check().  If you
try to use them before .check(), you get None, which hopefully leads to
an error.  Meh, but pylint is happy.

I picked 2. because pylint's warning made me go "when in Rome..."

With type hints, we can declare in .__init__(), and assign in .check().
Gives me the AttributeError I like, and pylint remains happy.  What do
you think?

Splitting .checked feels like a separate change, though.  I can't quite
see why we need it.  Help me out: what problem does it solve?

> This simplifies the typing from a cumbersome Optional[List[T]] to merely
> a List[T], which is more pythonic: it is safe to iterate over an empty
> list with "for x in []" whereas with an Optional[List[T]] you have to
> rely on the more cumbersome "if L: for x in L: ..."

Yes, but when L is None, it's *invalid*, and for i in L *should* fail
when L is invalid.

I think the actual problem is something else.  By adding the type hint
Optional[List[T]], the valid uses of L become type errors.  We really
want L to be a List[T].  Doesn't mean we have to (or want to) make uses
of invalid L "work".

> RFC: are we guaranteed to have members here? can we just use "if
> members" without adding the new field?

I'm afraid I don't understand these questions.

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/schema.py | 24 +++++++++++++++---------
>  1 file changed, 15 insertions(+), 9 deletions(-)
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 164d86c4064..200bc0730d6 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -18,7 +18,7 @@
>  from collections import OrderedDict
>  import os
>  import re
> -from typing import List, Optional
> +from typing import List, Optional, cast
>  
>  from .common import (
>      POINTER_SUFFIX,
> @@ -447,22 +447,24 @@ def __init__(self, name, info, doc, ifcond, features,
>          self.base = None
>          self.local_members = local_members
>          self.variants = variants
> -        self.members = None
> +        self.members: List[QAPISchemaObjectTypeMember] = []

Can we do

           self.members: List[QAPISchemaObjectTypeMember]

?

> +        self._checking = False
>  
>      def check(self, schema):
>          # This calls another type T's .check() exactly when the C
>          # struct emitted by gen_object() contains that T's C struct
>          # (pointers don't count).
> -        if self.members is not None:
> -            # A previous .check() completed: nothing to do
> -            return
> -        if self._checked:
> +        if self._checking:
>              # Recursed: C struct contains itself
>              raise QAPISemError(self.info,
>                                 "object %s contains itself" % self.name)
> +        if self._checked:
> +            # A previous .check() completed: nothing to do
> +            return
>  
> +        self._checking = True
>          super().check(schema)
> -        assert self._checked and self.members is None
> +        assert self._checked and not self.members

If we assign only in .check(), we can't read self.members to find out
whether it's valid.  We could perhaps mess with .__dict__() instead.
Not sure it's worthwhile.  Dumb down the assertion?

>  
>          seen = OrderedDict()
>          if self._base_name:
> @@ -479,13 +481,17 @@ def check(self, schema):
>          for m in self.local_members:
>              m.check(schema)
>              m.check_clash(self.info, seen)
> -        members = seen.values()
> +
> +        # check_clash is abstract, but local_members is asserted to be
> +        # List[QAPISchemaObjectTypeMember]. Cast to the narrower type.
> +        members = cast(List[QAPISchemaObjectTypeMember], list(seen.values()))
>  
>          if self.variants:
>              self.variants.check(schema, seen)
>              self.variants.check_clash(self.info, seen)
>  
> -        self.members = members  # mark completed
> +        self.members = members
> +        self._checking = False  # mark completed
>  
>      # Check that the members of this type do not cause duplicate JSON members,
>      # and update seen to track the members seen so far. Report any errors

Re: [PATCH 12/19] qapi/schema: split "checked" field into "checking" and "checked"

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 7768 bytes --]

On Wed, Nov 22, 2023, 9:02 AM Markus Armbruster <armbru@redhat.com> wrote:

> John Snow <jsnow@redhat.com> writes:
>
> > Differentiate between "actively in the process of checking" and
> > "checking has completed". This allows us to clean up the types of some
> > internal fields such as QAPISchemaObjectType's members field which
> > currently uses "None" as a canary for determining if check has
> > completed.
>
> Certain members become valid only after .check().  Two ways to code
> that:
>
> 1. Assign to such members only in .check().  If you try to use them
> before .check(), AttributeError.  Nice.  Drawback: pylint is unhappy,
> W0201 attribute-defined-outside-init.
>

Can be overcome by declaring the field in __init__, which satisfies both
the linter and my developer usability sense (Classes should be easy to have
their properties enumerated by looking in one well known place.)


> 2. Assign None in .__init__(), and the real value in .check().  If you
> try to use them before .check(), you get None, which hopefully leads to
> an error.  Meh, but pylint is happy.
>
> I picked 2. because pylint's warning made me go "when in Rome..."
>

Yep, this is perfectly cromulent dynamically typed Python. It's not the
Roman's fault I'm packing us up to go to another empire.


> With type hints, we can declare in .__init__(), and assign in .check().
> Gives me the AttributeError I like, and pylint remains happy.  What do
> you think?
>

Sounds good to me in general, I already changed this for 2/3 of my other
uses of @property.

(I'm only reluctant because I don't really like that it's a "lie", but in
this case, without potentially significant rewrites, it's a reasonable type
band-aid. Once we're type checked, we can refactor more confidently if we
so desire, to make certain patterns less prominent or landmine-y.)


> Splitting .checked feels like a separate change, though.  I can't quite
> see why we need it.  Help me out: what problem does it solve?
>

Mechanically, I wanted to eliminate the Optional type from the members
field, but you have conditionals in the code that use the presence or
absence of this field as a query to determine if we had run check or not
yet.

So I did the stupidest possible thing and added a "checked" variable to
explicitly represent it.


> > This simplifies the typing from a cumbersome Optional[List[T]] to merely
> > a List[T], which is more pythonic: it is safe to iterate over an empty
> > list with "for x in []" whereas with an Optional[List[T]] you have to
> > rely on the more cumbersome "if L: for x in L: ..."
>
> Yes, but when L is None, it's *invalid*, and for i in L *should* fail
> when L is invalid.
>

Sure, but it's so invalid that it causes static typing errors.

You can't write "for x in None" in a way that makes mypy happy, None is not
iterable.

If you want to preserve the behavior of "iterating an empty collection is
an Assertion", you need a custom iterator that throws an exception when the
collection is empty. I can do that, if you'd like, but I think it's
actually fine to just allow the collection to be empty and to just catch
the error in check() with either an assertion (oops, something went wrong
and the list is empty, this should never happen) or a QAPISemError (oops,
you didn't specify any members, which is illegal.)

I'd prefer to catch this in check and just allow the iterator to permit
empty iterators at the callsite, knowing it'll never happen.


> I think the actual problem is something else.  By adding the type hint
> Optional[List[T]], the valid uses of L become type errors.  We really
> want L to be a List[T].  Doesn't mean we have to (or want to) make uses
> of invalid L "work".
>

I didn't think I did allow for invalid uses to work - if the list should
semantically never be empty, I think it's fine to enforce that in schema.py
during construction of the schema object and to assume all uses of "for x
in L: ..." are inherently valid.


> > RFC: are we guaranteed to have members here? can we just use "if
> > members" without adding the new field?
>
> I'm afraid I don't understand these questions.
>

I think you answered this one for me; I was asking if it was ever valid in
any circumstance to have an empty members list, but I think you've laid out
in your response that it isn't.

And I think with that knowledge I can simplify this patch, but don't quite
recall. (On my mobile again, please excuse my apparent laziness.)


> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  scripts/qapi/schema.py | 24 +++++++++++++++---------
> >  1 file changed, 15 insertions(+), 9 deletions(-)
> >
> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> > index 164d86c4064..200bc0730d6 100644
> > --- a/scripts/qapi/schema.py
> > +++ b/scripts/qapi/schema.py
> > @@ -18,7 +18,7 @@
> >  from collections import OrderedDict
> >  import os
> >  import re
> > -from typing import List, Optional
> > +from typing import List, Optional, cast
> >
> >  from .common import (
> >      POINTER_SUFFIX,
> > @@ -447,22 +447,24 @@ def __init__(self, name, info, doc, ifcond,
> features,
> >          self.base = None
> >          self.local_members = local_members
> >          self.variants = variants
> > -        self.members = None
> > +        self.members: List[QAPISchemaObjectTypeMember] = []
>
> Can we do
>
>            self.members: List[QAPISchemaObjectTypeMember]
>
> ?
>

Possibly, but also possibly I can just initialize it to an empty list.


> > +        self._checking = False
> >
> >      def check(self, schema):
> >          # This calls another type T's .check() exactly when the C
> >          # struct emitted by gen_object() contains that T's C struct
> >          # (pointers don't count).
> > -        if self.members is not None:
> > -            # A previous .check() completed: nothing to do
> > -            return
> > -        if self._checked:
> > +        if self._checking:
> >              # Recursed: C struct contains itself
> >              raise QAPISemError(self.info,
> >                                 "object %s contains itself" % self.name)
> > +        if self._checked:
> > +            # A previous .check() completed: nothing to do
> > +            return
> >
> > +        self._checking = True
> >          super().check(schema)
> > -        assert self._checked and self.members is None
> > +        assert self._checked and not self.members
>
> If we assign only in .check(), we can't read self.members to find out
> whether it's valid.  We could perhaps mess with .__dict__() instead.
> Not sure it's worthwhile.  Dumb down the assertion?
>

If I initialize to an empty list (and don't mess with the checked member)
maybe

assert self._checked and not self.members

would be perfectly acceptable.


> >
> >          seen = OrderedDict()
> >          if self._base_name:
> > @@ -479,13 +481,17 @@ def check(self, schema):
> >          for m in self.local_members:
> >              m.check(schema)
> >              m.check_clash(self.info, seen)
> > -        members = seen.values()
> > +
> > +        # check_clash is abstract, but local_members is asserted to be
> > +        # List[QAPISchemaObjectTypeMember]. Cast to the narrower type.
> > +        members = cast(List[QAPISchemaObjectTypeMember],
> list(seen.values()))
> >
> >          if self.variants:
> >              self.variants.check(schema, seen)
> >              self.variants.check_clash(self.info, seen)
> >
> > -        self.members = members  # mark completed
> > +        self.members = members
> > +        self._checking = False  # mark completed
> >
> >      # Check that the members of this type do not cause duplicate JSON
> members,
> >      # and update seen to track the members seen so far. Report any
> errors
>
>

[-- Attachment #2: Type: text/html, Size: 12185 bytes --]

Re: [PATCH 12/19] qapi/schema: split "checked" field into "checking" and "checked"

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Wed, Nov 22, 2023, 9:02 AM Markus Armbruster <armbru@redhat.com> wrote:
>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > Differentiate between "actively in the process of checking" and
>> > "checking has completed". This allows us to clean up the types of some
>> > internal fields such as QAPISchemaObjectType's members field which
>> > currently uses "None" as a canary for determining if check has
>> > completed.
>>
>> Certain members become valid only after .check().  Two ways to code
>> that:
>>
>> 1. Assign to such members only in .check().  If you try to use them
>> before .check(), AttributeError.  Nice.  Drawback: pylint is unhappy,
>> W0201 attribute-defined-outside-init.
>>
>
> Can be overcome by declaring the field in __init__, which satisfies both
> the linter and my developer usability sense (Classes should be easy to have
> their properties enumerated by looking in one well known place.)
>
>
>> 2. Assign None in .__init__(), and the real value in .check().  If you
>> try to use them before .check(), you get None, which hopefully leads to
>> an error.  Meh, but pylint is happy.
>>
>> I picked 2. because pylint's warning made me go "when in Rome..."
>>
>
> Yep, this is perfectly cromulent dynamically typed Python. It's not the
> Roman's fault I'm packing us up to go to another empire.
>
>
>> With type hints, we can declare in .__init__(), and assign in .check().
>> Gives me the AttributeError I like, and pylint remains happy.  What do
>> you think?
>>
>
> Sounds good to me in general, I already changed this for 2/3 of my other
> uses of @property.
>
> (I'm only reluctant because I don't really like that it's a "lie", but in
> this case, without potentially significant rewrites, it's a reasonable type
> band-aid. Once we're type checked, we can refactor more confidently if we
> so desire, to make certain patterns less prominent or landmine-y.)

The general problem is "attribute value is valid only after a state
transition" (here: .member is valid only after .check()).

We want to catch uses of the attribute before it becomes valid.

We want to keep pylint and mypy happy.

Solutions:

1. Initialize in .__init__() to some invalid value.  Set it to the valid
   value in .check().

1.a. Pick the "natural" invalid value: None

   How to catch: assert attribute value is valid (here: .members is not
   None).  Easy to forget.  Better: when the use will safely choke on
   the invalid value (here: elide for uses like for m in .members),
   catch is automatic.

   Pylint: fine.

   Mypy: adding None to the set of values changes the type from T to
   Optional[T].  Because of this, mypy commonly can't prove valid uses
   are valid.  Keeping it happy requires cluttering the code with
   assertions and such.  Meh.

   Note: catching invalid uses is a run time check.  Mypy won't.

1.b. Pick an invalid value of type T (here: [])

   How to catch: same as 1.a., except automatic catch is rare.  Meh.

   Pylint: fine.

   Mypy: fine.

2. Declare in .__init__() without initializing.  Initialize to valid
   value in .check()

   How to catch: always automatic.  Good, me want!

   Pylint: fine.

   Mypy: fine.

   Note: catching invalid uses is a run time check.  Mypy won't.

3. Express the state transition in the type system

   To catch invalid uses statically with mypy, we need to use different
   types before and after the state transition.  Feels possible.  Also
   feels ludicrously overengineered.

May I have 2., please?

>> Splitting .checked feels like a separate change, though.  I can't quite
>> see why we need it.  Help me out: what problem does it solve?
>>
>
> Mechanically, I wanted to eliminate the Optional type from the members
> field, but you have conditionals in the code that use the presence or
> absence of this field as a query to determine if we had run check or not
> yet.
>
> So I did the stupidest possible thing and added a "checked" variable to
> explicitly represent it.

If 2. complicates the existing "have we .check()ed?" code too much, then
adding such a variable may indeed be useful.

>> > This simplifies the typing from a cumbersome Optional[List[T]] to merely
>> > a List[T], which is more pythonic: it is safe to iterate over an empty
>> > list with "for x in []" whereas with an Optional[List[T]] you have to
>> > rely on the more cumbersome "if L: for x in L: ..."
>>
>> Yes, but when L is None, it's *invalid*, and for i in L *should* fail
>> when L is invalid.
>>
>
> Sure, but it's so invalid that it causes static typing errors.
>
> You can't write "for x in None" in a way that makes mypy happy, None is not
> iterable.

A variable that is declared, but not initialized (2. above) also not
iterable, and it does make mypy happy, doesn't it?

> If you want to preserve the behavior of "iterating an empty collection is
> an Assertion", you need a custom iterator that throws an exception when the
> collection is empty. I can do that, if you'd like, but I think it's
> actually fine to just allow the collection to be empty and to just catch
> the error in check() with either an assertion (oops, something went wrong
> and the list is empty, this should never happen) or a QAPISemError (oops,
> you didn't specify any members, which is illegal.)
>
> I'd prefer to catch this in check and just allow the iterator to permit
> empty iterators at the callsite, knowing it'll never happen.
>
>
>> I think the actual problem is something else.  By adding the type hint
>> Optional[List[T]], the valid uses of L become type errors.  We really
>> want L to be a List[T].  Doesn't mean we have to (or want to) make uses
>> of invalid L "work".
>>
>
> I didn't think I did allow for invalid uses to work - if the list should
> semantically never be empty, I think it's fine to enforce that in schema.py
> during construction of the schema object and to assume all uses of "for x
> in L: ..." are inherently valid.
>
>
>> > RFC: are we guaranteed to have members here? can we just use "if
>> > members" without adding the new field?
>>
>> I'm afraid I don't understand these questions.
>>
>
> I think you answered this one for me; I was asking if it was ever valid in
> any circumstance to have an empty members list, but I think you've laid out
> in your response that it isn't.
>
> And I think with that knowledge I can simplify this patch, but don't quite
> recall. (On my mobile again, please excuse my apparent laziness.)

Feel excused!

[...]

[PATCH 13/19] qapi/schema: fix typing for QAPISchemaVariants.tag_member

[John Snow]

There are two related changes here:

(1) We need to perform type narrowing for resolving the type of
    tag_member during check(), and

(2) tag_member is a delayed initialization field, but we can hide it
    behind a property that raises an Exception if it's called too
    early. This simplifies the typing in quite a few places and avoids
    needing to assert that the "tag_member is not None" at a dozen
    callsites, which can be confusing and suggest the wrong thing to a
    drive-by contributor.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 22 +++++++++++++++++++---
 1 file changed, 19 insertions(+), 3 deletions(-)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 200bc0730d6..476b19aed61 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -627,25 +627,39 @@ def __init__(self, tag_name, info, tag_member, variants):
             assert isinstance(v, QAPISchemaVariant)
         self._tag_name = tag_name
         self.info = info
-        self.tag_member = tag_member
+        self._tag_member: Optional[QAPISchemaObjectTypeMember] = tag_member
         self.variants = variants
 
+    @property
+    def tag_member(self) -> 'QAPISchemaObjectTypeMember':
+        if self._tag_member is None:
+            raise RuntimeError(
+                "QAPISchemaVariants has no tag_member property until "
+                "after check() has been run."
+            )
+        return self._tag_member
+
     def set_defined_in(self, name):
         for v in self.variants:
             v.set_defined_in(name)
 
     def check(self, schema, seen):
         if self._tag_name:      # union
-            self.tag_member = seen.get(c_name(self._tag_name))
+            # We need to narrow the member type:
+            tmp = seen.get(c_name(self._tag_name))
+            assert tmp is None or isinstance(tmp, QAPISchemaObjectTypeMember)
+            self._tag_member = tmp
+
             base = "'base'"
             # Pointing to the base type when not implicit would be
             # nice, but we don't know it here
-            if not self.tag_member or self._tag_name != self.tag_member.name:
+            if not self._tag_member or self._tag_name != self._tag_member.name:
                 raise QAPISemError(
                     self.info,
                     "discriminator '%s' is not a member of %s"
                     % (self._tag_name, base))
             # Here we do:
+            assert self.tag_member.defined_in
             base_type = schema.lookup_type(self.tag_member.defined_in)
             assert base_type
             if not base_type.is_implicit():
@@ -666,11 +680,13 @@ def check(self, schema, seen):
                     "discriminator member '%s' of %s must not be conditional"
                     % (self._tag_name, base))
         else:                   # alternate
+            assert self._tag_member
             assert isinstance(self.tag_member.type, QAPISchemaEnumType)
             assert not self.tag_member.optional
             assert not self.tag_member.ifcond.is_present()
         if self._tag_name:      # union
             # branches that are not explicitly covered get an empty type
+            assert self.tag_member.defined_in
             cases = {v.name for v in self.variants}
             for m in self.tag_member.type.members:
                 if m.name not in cases:
-- 
2.41.0

Re: [PATCH 13/19] qapi/schema: fix typing for QAPISchemaVariants.tag_member

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> There are two related changes here:
>
> (1) We need to perform type narrowing for resolving the type of
>     tag_member during check(), and
>
> (2) tag_member is a delayed initialization field, but we can hide it
>     behind a property that raises an Exception if it's called too
>     early. This simplifies the typing in quite a few places and avoids
>     needing to assert that the "tag_member is not None" at a dozen
>     callsites, which can be confusing and suggest the wrong thing to a
>     drive-by contributor.
>
> Signed-off-by: John Snow <jsnow@redhat.com>

Without looking closely: review of PATCH 10 applies, doesn't it?

Re: [PATCH 13/19] qapi/schema: fix typing for QAPISchemaVariants.tag_member

[John Snow]

On Wed, Nov 22, 2023 at 9:05 AM Markus Armbruster <armbru@redhat.com> wrote:
>
> John Snow <jsnow@redhat.com> writes:
>
> > There are two related changes here:
> >
> > (1) We need to perform type narrowing for resolving the type of
> >     tag_member during check(), and
> >
> > (2) tag_member is a delayed initialization field, but we can hide it
> >     behind a property that raises an Exception if it's called too
> >     early. This simplifies the typing in quite a few places and avoids
> >     needing to assert that the "tag_member is not None" at a dozen
> >     callsites, which can be confusing and suggest the wrong thing to a
> >     drive-by contributor.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
>
> Without looking closely: review of PATCH 10 applies, doesn't it?
>

Yep!

Re: [PATCH 13/19] qapi/schema: fix typing for QAPISchemaVariants.tag_member

[John Snow]

On Wed, Nov 22, 2023 at 11:02 AM John Snow <jsnow@redhat.com> wrote:
>
> On Wed, Nov 22, 2023 at 9:05 AM Markus Armbruster <armbru@redhat.com> wrote:
> >
> > John Snow <jsnow@redhat.com> writes:
> >
> > > There are two related changes here:
> > >
> > > (1) We need to perform type narrowing for resolving the type of
> > >     tag_member during check(), and
> > >
> > > (2) tag_member is a delayed initialization field, but we can hide it
> > >     behind a property that raises an Exception if it's called too
> > >     early. This simplifies the typing in quite a few places and avoids
> > >     needing to assert that the "tag_member is not None" at a dozen
> > >     callsites, which can be confusing and suggest the wrong thing to a
> > >     drive-by contributor.
> > >
> > > Signed-off-by: John Snow <jsnow@redhat.com>
> >
> > Without looking closely: review of PATCH 10 applies, doesn't it?
> >
>
> Yep!

Hm, actually, maybe not quite as cleanly.

The problem is we *are* initializing that field immediately with
whatever we were passed in during __init__, which means the field is
indeed Optional. Later, during check(), we happen to eliminate that
usage of None.

To remove the use of the @property trick here, we could:

... declare the field, then only initialize it if we were passed a
non-None value. But then check() would need to rely on something like
hasattr to check if it was set or not, which is maybe an unfortunate
code smell.
So I think you'd still wind up needing a ._tag_member field which is
Optional and always gets set during __init__, then setting a proper
.tag_member field during check().

Or I could just leave this one as-is. Or something else. I think the
dirt has to get swept somewhere, because we don't *always* have enough
information to fully initialize it at __init__ time, it's a
conditional delayed initialization, unlike the others which are
unconditionally delayed.

--js

Re: [PATCH 13/19] qapi/schema: fix typing for QAPISchemaVariants.tag_member

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Wed, Nov 22, 2023 at 11:02 AM John Snow <jsnow@redhat.com> wrote:
>>
>> On Wed, Nov 22, 2023 at 9:05 AM Markus Armbruster <armbru@redhat.com> wrote:
>> >
>> > John Snow <jsnow@redhat.com> writes:
>> >
>> > > There are two related changes here:
>> > >
>> > > (1) We need to perform type narrowing for resolving the type of
>> > >     tag_member during check(), and
>> > >
>> > > (2) tag_member is a delayed initialization field, but we can hide it
>> > >     behind a property that raises an Exception if it's called too
>> > >     early. This simplifies the typing in quite a few places and avoids
>> > >     needing to assert that the "tag_member is not None" at a dozen
>> > >     callsites, which can be confusing and suggest the wrong thing to a
>> > >     drive-by contributor.
>> > >
>> > > Signed-off-by: John Snow <jsnow@redhat.com>
>> >
>> > Without looking closely: review of PATCH 10 applies, doesn't it?
>> >
>>
>> Yep!
>
> Hm, actually, maybe not quite as cleanly.
>
> The problem is we *are* initializing that field immediately with
> whatever we were passed in during __init__, which means the field is
> indeed Optional. Later, during check(), we happen to eliminate that
> usage of None.

You're right.

QAPISchemaVariants.__init__() takes @tag_name and @tag_member.  Exactly
one of them must be None.  When creating a union's QAPISchemaVariants,
it's tag_member, and when creating an alternate's, it's tag_name.

Why?

A union's tag is an ordinary member selected by name via
'discriminator': TAG_NAME.  We can't resolve the name at this time,
because it may be buried arbitrarily deep in the base type chain.

An alternate's tag is an implicitly created "member" of type 'QType'.
"Member" in scare-quotes, because is special: it exists in C, but not on
the wire, and not in introspection.

Historical note: simple unions also had an implictly created tag member,
and its type was the implicit enum type enumerating the branches.

So _def_union_type() passes TAG_NAME to .__init__(), and
_def_alternate_type() creates and passes the implicit tag member.
Hardly elegant, but it works.

> To remove the use of the @property trick here, we could:
>
> ... declare the field, then only initialize it if we were passed a
> non-None value. But then check() would need to rely on something like
> hasattr to check if it was set or not, which is maybe an unfortunate
> code smell.
> So I think you'd still wind up needing a ._tag_member field which is
> Optional and always gets set during __init__, then setting a proper
> .tag_member field during check().
>
> Or I could just leave this one as-is. Or something else. I think the
> dirt has to get swept somewhere, because we don't *always* have enough
> information to fully initialize it at __init__ time, it's a
> conditional delayed initialization, unlike the others which are
> unconditionally delayed.

Yes.

Here's a possible "something else":

1. Drop parameter .__init__() parameter @tag_member, and leave
.tag_member unset there.

2. Set .tag_member in .check(): if .tag_name, look up that member (no
change).  Else, it's an alternate; create the alternate's implicit tag
member.

Drawback: before, we create AST in just one place, namely
QAPISchema._def_exprs().  Now we also create some in .check().

Here's another "something else":

1. Fuse parameters .__init__() @tag_member and @tag_name.  The type
becomes Union.  Store for .check().

2. Set .tag_member in .check(): if we stored a name, look up that
member, else we must have stored an implicit member, so use that.

3. We check "is this a union?" like if self._tag_name.  Needs
adjustment.

Feels a bit awkward to me.

We can also do nothing, as you said.  We don't *have* to express
".check() resolves unresolved tag member" in the type system.  We can
just live with .tag_member remaining Optional.

Differently awkward, I guess.

Thoughts?

Re: [PATCH 13/19] qapi/schema: fix typing for QAPISchemaVariants.tag_member

[John Snow]

On Wed, Jan 10, 2024 at 2:53 AM Markus Armbruster <armbru@redhat.com> wrote:
>
> John Snow <jsnow@redhat.com> writes:
>
> > On Wed, Nov 22, 2023 at 11:02 AM John Snow <jsnow@redhat.com> wrote:
> >>
> >> On Wed, Nov 22, 2023 at 9:05 AM Markus Armbruster <armbru@redhat.com> wrote:
> >> >
> >> > John Snow <jsnow@redhat.com> writes:
> >> >
> >> > > There are two related changes here:
> >> > >
> >> > > (1) We need to perform type narrowing for resolving the type of
> >> > >     tag_member during check(), and
> >> > >
> >> > > (2) tag_member is a delayed initialization field, but we can hide it
> >> > >     behind a property that raises an Exception if it's called too
> >> > >     early. This simplifies the typing in quite a few places and avoids
> >> > >     needing to assert that the "tag_member is not None" at a dozen
> >> > >     callsites, which can be confusing and suggest the wrong thing to a
> >> > >     drive-by contributor.
> >> > >
> >> > > Signed-off-by: John Snow <jsnow@redhat.com>
> >> >
> >> > Without looking closely: review of PATCH 10 applies, doesn't it?
> >> >
> >>
> >> Yep!
> >
> > Hm, actually, maybe not quite as cleanly.
> >
> > The problem is we *are* initializing that field immediately with
> > whatever we were passed in during __init__, which means the field is
> > indeed Optional. Later, during check(), we happen to eliminate that
> > usage of None.
>
> You're right.
>
> QAPISchemaVariants.__init__() takes @tag_name and @tag_member.  Exactly
> one of them must be None.  When creating a union's QAPISchemaVariants,
> it's tag_member, and when creating an alternate's, it's tag_name.
>
> Why?
>
> A union's tag is an ordinary member selected by name via
> 'discriminator': TAG_NAME.  We can't resolve the name at this time,
> because it may be buried arbitrarily deep in the base type chain.
>
> An alternate's tag is an implicitly created "member" of type 'QType'.
> "Member" in scare-quotes, because is special: it exists in C, but not on
> the wire, and not in introspection.
>
> Historical note: simple unions also had an implictly created tag member,
> and its type was the implicit enum type enumerating the branches.
>
> So _def_union_type() passes TAG_NAME to .__init__(), and
> _def_alternate_type() creates and passes the implicit tag member.
> Hardly elegant, but it works.
>
> > To remove the use of the @property trick here, we could:
> >
> > ... declare the field, then only initialize it if we were passed a
> > non-None value. But then check() would need to rely on something like
> > hasattr to check if it was set or not, which is maybe an unfortunate
> > code smell.
> > So I think you'd still wind up needing a ._tag_member field which is
> > Optional and always gets set during __init__, then setting a proper
> > .tag_member field during check().
> >
> > Or I could just leave this one as-is. Or something else. I think the
> > dirt has to get swept somewhere, because we don't *always* have enough
> > information to fully initialize it at __init__ time, it's a
> > conditional delayed initialization, unlike the others which are
> > unconditionally delayed.
>
> Yes.
>
> Here's a possible "something else":
>
> 1. Drop parameter .__init__() parameter @tag_member, and leave
> .tag_member unset there.
>
> 2. Set .tag_member in .check(): if .tag_name, look up that member (no
> change).  Else, it's an alternate; create the alternate's implicit tag
> member.
>
> Drawback: before, we create AST in just one place, namely
> QAPISchema._def_exprs().  Now we also create some in .check().

I suppose I don't have a concrete argument against this beyond "It
doesn't seem prettier than using the @property getter."

>
> Here's another "something else":
>
> 1. Fuse parameters .__init__() @tag_member and @tag_name.  The type
> becomes Union.  Store for .check().
>
> 2. Set .tag_member in .check(): if we stored a name, look up that
> member, else we must have stored an implicit member, so use that.
>
> 3. We check "is this a union?" like if self._tag_name.  Needs
> adjustment.
>
> Feels a bit awkward to me.

Yeah, a little. Mechanically simple, though, I think.

>
> We can also do nothing, as you said.  We don't *have* to express
> ".check() resolves unresolved tag member" in the type system.  We can
> just live with .tag_member remaining Optional.

This is the only option I'm sure I don't want - it's misleading to
users of the API for the purposes of new generators using a fully
realized schema object. I think it's important to remove Optional[]
where possible to avoid the question "When will this be set to None?"
if the answer is just "For your purposes, never."

It's an implementation detail of object initialization leaking out.

(Also, I just counted and leaving the field as Optional adds 22 new
type errors; that's a lot of callsites to bandage with new conditions.
nah.)

The *other* way to not do anything is to just leave the @property
solution in O:-)

>
> Differently awkward, I guess.
>
> Thoughts?

Partial to the getter, unless option 1 or 2 leads to simplification of
the check() code, which I haven't really experimented with. If that's
something you'd really rather avoid, I might ask for you to decide on
your preferred alternative - I don't have strong feelings between 'em.

(Not helpful, oops. Thanks for your feedback and review, though.
You've successfully talked me down to a much smaller series over time
:p)

--js

Re: [PATCH 13/19] qapi/schema: fix typing for QAPISchemaVariants.tag_member

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Wed, Jan 10, 2024 at 2:53 AM Markus Armbruster <armbru@redhat.com> wrote:
>>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > On Wed, Nov 22, 2023 at 11:02 AM John Snow <jsnow@redhat.com> wrote:
>> >>
>> >> On Wed, Nov 22, 2023 at 9:05 AM Markus Armbruster <armbru@redhat.com> wrote:
>> >> >
>> >> > John Snow <jsnow@redhat.com> writes:
>> >> >
>> >> > > There are two related changes here:
>> >> > >
>> >> > > (1) We need to perform type narrowing for resolving the type of
>> >> > >     tag_member during check(), and
>> >> > >
>> >> > > (2) tag_member is a delayed initialization field, but we can hide it
>> >> > >     behind a property that raises an Exception if it's called too
>> >> > >     early. This simplifies the typing in quite a few places and avoids
>> >> > >     needing to assert that the "tag_member is not None" at a dozen
>> >> > >     callsites, which can be confusing and suggest the wrong thing to a
>> >> > >     drive-by contributor.
>> >> > >
>> >> > > Signed-off-by: John Snow <jsnow@redhat.com>
>> >> >
>> >> > Without looking closely: review of PATCH 10 applies, doesn't it?
>> >> >
>> >>
>> >> Yep!
>> >
>> > Hm, actually, maybe not quite as cleanly.
>> >
>> > The problem is we *are* initializing that field immediately with
>> > whatever we were passed in during __init__, which means the field is
>> > indeed Optional. Later, during check(), we happen to eliminate that
>> > usage of None.
>>
>> You're right.
>>
>> QAPISchemaVariants.__init__() takes @tag_name and @tag_member.  Exactly
>> one of them must be None.  When creating a union's QAPISchemaVariants,
>> it's tag_member, and when creating an alternate's, it's tag_name.
>>
>> Why?
>>
>> A union's tag is an ordinary member selected by name via
>> 'discriminator': TAG_NAME.  We can't resolve the name at this time,
>> because it may be buried arbitrarily deep in the base type chain.
>>
>> An alternate's tag is an implicitly created "member" of type 'QType'.
>> "Member" in scare-quotes, because is special: it exists in C, but not on
>> the wire, and not in introspection.
>>
>> Historical note: simple unions also had an implictly created tag member,
>> and its type was the implicit enum type enumerating the branches.
>>
>> So _def_union_type() passes TAG_NAME to .__init__(), and
>> _def_alternate_type() creates and passes the implicit tag member.
>> Hardly elegant, but it works.
>>
>> > To remove the use of the @property trick here, we could:
>> >
>> > ... declare the field, then only initialize it if we were passed a
>> > non-None value. But then check() would need to rely on something like
>> > hasattr to check if it was set or not, which is maybe an unfortunate
>> > code smell.
>> > So I think you'd still wind up needing a ._tag_member field which is
>> > Optional and always gets set during __init__, then setting a proper
>> > .tag_member field during check().
>> >
>> > Or I could just leave this one as-is. Or something else. I think the
>> > dirt has to get swept somewhere, because we don't *always* have enough
>> > information to fully initialize it at __init__ time, it's a
>> > conditional delayed initialization, unlike the others which are
>> > unconditionally delayed.
>>
>> Yes.
>>
>> Here's a possible "something else":
>>
>> 1. Drop parameter .__init__() parameter @tag_member, and leave
>> .tag_member unset there.
>>
>> 2. Set .tag_member in .check(): if .tag_name, look up that member (no
>> change).  Else, it's an alternate; create the alternate's implicit tag
>> member.
>>
>> Drawback: before, we create AST in just one place, namely
>> QAPISchema._def_exprs().  Now we also create some in .check().
>
> I suppose I don't have a concrete argument against this beyond "It
> doesn't seem prettier than using the @property getter."
>
>>
>> Here's another "something else":
>>
>> 1. Fuse parameters .__init__() @tag_member and @tag_name.  The type
>> becomes Union.  Store for .check().
>>
>> 2. Set .tag_member in .check(): if we stored a name, look up that
>> member, else we must have stored an implicit member, so use that.
>>
>> 3. We check "is this a union?" like if self._tag_name.  Needs
>> adjustment.
>>
>> Feels a bit awkward to me.
>
> Yeah, a little. Mechanically simple, though, I think.
>
>> We can also do nothing, as you said.  We don't *have* to express
>> ".check() resolves unresolved tag member" in the type system.  We can
>> just live with .tag_member remaining Optional.
>
> This is the only option I'm sure I don't want - it's misleading to
> users of the API for the purposes of new generators using a fully
> realized schema object. I think it's important to remove Optional[]
> where possible to avoid the question "When will this be set to None?"
> if the answer is just "For your purposes, never."
>
> It's an implementation detail of object initialization leaking out.
>
> (Also, I just counted and leaving the field as Optional adds 22 new
> type errors; that's a lot of callsites to bandage with new conditions.
> nah.)
>
> The *other* way to not do anything is to just leave the @property
> solution in O:-)
>
>>
>> Differently awkward, I guess.
>>
>> Thoughts?
>
> Partial to the getter, unless option 1 or 2 leads to simplification of
> the check() code,

Put a pin into that.

>                   which I haven't really experimented with. If that's
> something you'd really rather avoid, I might ask for you to decide on
> your preferred alternative - I don't have strong feelings between 'em.

All the solutions so far give me slight "there has to be a better way"
vibes.  Staring at QAPISchemaVariants, I realized: more than half of the
actual code are under "if union / else" conditionals.  So I tried the
more object-oriented solution: classes instead of conditionals.  Diff
appended.

Shedding the conditionals does lead to slightly simple .check(), so
maybe you like it.

Once this is done, we can narrow .tag_member's type from
Optional[QAPISchemaObjectTypeMember] to QAPISchemaObjectTypeMember the
exact same way as QAPISchemaObjectTypeMember.type: replace
self.tag_member = None by self.tag_member: QAPISchemaObjectTypeMember.

Admittedly a bit more churn that your solution, but the result has
slightly less code, and feels slitghly cleaner to me.  Thoughts?

> (Not helpful, oops. Thanks for your feedback and review, though.
> You've successfully talked me down to a much smaller series over time
> :p)


diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index c38df61a6d..e5cea8004e 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -26,6 +26,7 @@
 from .gen import QAPISchemaMonolithicCVisitor
 from .schema import (
     QAPISchema,
+    QAPISchemaAlternatives,
     QAPISchemaArrayType,
     QAPISchemaBuiltinType,
     QAPISchemaEntity,
@@ -343,12 +344,12 @@ def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo],
     def visit_alternate_type(self, name: str, info: Optional[QAPISourceInfo],
                              ifcond: QAPISchemaIfCond,
                              features: List[QAPISchemaFeature],
-                             variants: QAPISchemaVariants) -> None:
+                             alternatives: QAPISchemaAlternatives) -> None:
         self._gen_tree(
             name, 'alternate',
             {'members': [Annotated({'type': self._use_type(m.type)},
                                    m.ifcond)
-                         for m in variants.variants]},
+                         for m in alternatives.variants]},
             ifcond, features
         )
 
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 0d9a70ab4c..949ee6bfd4 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -563,8 +563,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
 
     def __init__(self, name, info, doc, ifcond, features, variants):
         super().__init__(name, info, doc, ifcond, features)
-        assert isinstance(variants, QAPISchemaVariants)
-        assert variants.tag_member
+        assert isinstance(variants, QAPISchemaAlternatives)
         variants.set_defined_in(name)
         variants.tag_member.set_defined_in(self.name)
         self.variants = variants
@@ -625,19 +624,12 @@ def visit(self, visitor):
             self.name, self.info, self.ifcond, self.features, self.variants)
 
 
-class QAPISchemaVariants:
-    def __init__(self, tag_name, info, tag_member, variants):
-        # Unions pass tag_name but not tag_member.
-        # Alternates pass tag_member but not tag_name.
-        # After check(), tag_member is always set.
-        assert bool(tag_member) != bool(tag_name)
-        assert (isinstance(tag_name, str) or
-                isinstance(tag_member, QAPISchemaObjectTypeMember))
+class QAPISchemaVariantsBase:
+    def __init__(self, info, variants):
         for v in variants:
             assert isinstance(v, QAPISchemaVariant)
-        self._tag_name = tag_name
         self.info = info
-        self.tag_member = tag_member
+        self.tag_member = None
         self.variants = variants
 
     def set_defined_in(self, name):
@@ -645,48 +637,6 @@ def set_defined_in(self, name):
             v.set_defined_in(name)
 
     def check(self, schema, seen):
-        if self._tag_name:      # union
-            self.tag_member = seen.get(c_name(self._tag_name))
-            base = "'base'"
-            # Pointing to the base type when not implicit would be
-            # nice, but we don't know it here
-            if not self.tag_member or self._tag_name != self.tag_member.name:
-                raise QAPISemError(
-                    self.info,
-                    "discriminator '%s' is not a member of %s"
-                    % (self._tag_name, base))
-            # Here we do:
-            base_type = schema.resolve_type(self.tag_member.defined_in)
-            if not base_type.is_implicit():
-                base = "base type '%s'" % self.tag_member.defined_in
-            if not isinstance(self.tag_member.type, QAPISchemaEnumType):
-                raise QAPISemError(
-                    self.info,
-                    "discriminator member '%s' of %s must be of enum type"
-                    % (self._tag_name, base))
-            if self.tag_member.optional:
-                raise QAPISemError(
-                    self.info,
-                    "discriminator member '%s' of %s must not be optional"
-                    % (self._tag_name, base))
-            if self.tag_member.ifcond.is_present():
-                raise QAPISemError(
-                    self.info,
-                    "discriminator member '%s' of %s must not be conditional"
-                    % (self._tag_name, base))
-        else:                   # alternate
-            assert isinstance(self.tag_member.type, QAPISchemaEnumType)
-            assert not self.tag_member.optional
-            assert not self.tag_member.ifcond.is_present()
-        if self._tag_name:      # union
-            # branches that are not explicitly covered get an empty type
-            cases = {v.name for v in self.variants}
-            for m in self.tag_member.type.members:
-                if m.name not in cases:
-                    v = QAPISchemaVariant(m.name, self.info,
-                                          'q_empty', m.ifcond)
-                    v.set_defined_in(self.tag_member.defined_in)
-                    self.variants.append(v)
         if not self.variants:
             raise QAPISemError(self.info, "union has no branches")
         for v in self.variants:
@@ -713,6 +663,65 @@ def check_clash(self, info, seen):
             v.type.check_clash(info, dict(seen))
 
 
+class QAPISchemaVariants(QAPISchemaVariantsBase):
+    def __init__(self, info, variants, tag_name):
+        assert isinstance(tag_name, str)
+        super().__init__(info, variants)
+        self._tag_name = tag_name
+
+    def check(self, schema, seen):
+        self.tag_member = seen.get(c_name(self._tag_name))
+        base = "'base'"
+        # Pointing to the base type when not implicit would be
+        # nice, but we don't know it here
+        if not self.tag_member or self._tag_name != self.tag_member.name:
+            raise QAPISemError(
+                self.info,
+                "discriminator '%s' is not a member of %s"
+                % (self._tag_name, base))
+        # Here we do:
+        base_type = schema.resolve_type(self.tag_member.defined_in)
+        if not base_type.is_implicit():
+            base = "base type '%s'" % self.tag_member.defined_in
+        if not isinstance(self.tag_member.type, QAPISchemaEnumType):
+            raise QAPISemError(
+                self.info,
+                "discriminator member '%s' of %s must be of enum type"
+                % (self._tag_name, base))
+        if self.tag_member.optional:
+            raise QAPISemError(
+                self.info,
+                "discriminator member '%s' of %s must not be optional"
+                % (self._tag_name, base))
+        if self.tag_member.ifcond.is_present():
+            raise QAPISemError(
+                self.info,
+                "discriminator member '%s' of %s must not be conditional"
+                % (self._tag_name, base))
+        # branches that are not explicitly covered get an empty type
+        cases = {v.name for v in self.variants}
+        for m in self.tag_member.type.members:
+            if m.name not in cases:
+                v = QAPISchemaVariant(m.name, self.info,
+                                      'q_empty', m.ifcond)
+                v.set_defined_in(self.tag_member.defined_in)
+                self.variants.append(v)
+        super().check(schema, seen)
+
+
+class QAPISchemaAlternatives(QAPISchemaVariantsBase):
+    def __init__(self, info, variants, tag_member):
+        assert isinstance(tag_member, QAPISchemaObjectTypeMember)
+        super().__init__(info, variants)
+        self.tag_member = tag_member
+
+    def check(self, schema, seen):
+        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        assert not self.tag_member.optional
+        assert not self.tag_member.ifcond.is_present()
+        super().check(schema, seen)
+
+
 class QAPISchemaMember:
     """ Represents object members, enum members and features """
     role = 'member'
@@ -1184,7 +1193,7 @@ def _def_union_type(self, expr: QAPIExpression):
             QAPISchemaObjectType(name, info, expr.doc, ifcond, features,
                                  base, members,
                                  QAPISchemaVariants(
-                                     tag_name, info, None, variants)))
+                                     info, variants, tag_name)))
 
     def _def_alternate_type(self, expr: QAPIExpression):
         name = expr['alternate']
@@ -1202,7 +1211,7 @@ def _def_alternate_type(self, expr: QAPIExpression):
         self._def_definition(
             QAPISchemaAlternateType(
                 name, info, expr.doc, ifcond, features,
-                QAPISchemaVariants(None, info, tag_member, variants)))
+                QAPISchemaAlternatives(info, variants, tag_member)))
 
     def _def_command(self, expr: QAPIExpression):
         name = expr['command']
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index c39d054d2c..05da30b855 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -23,6 +23,7 @@
 )
 from .schema import (
     QAPISchema,
+    QAPISchemaAlternatives,
     QAPISchemaEnumMember,
     QAPISchemaFeature,
     QAPISchemaIfCond,
@@ -369,11 +370,11 @@ def visit_alternate_type(self,
                              info: Optional[QAPISourceInfo],
                              ifcond: QAPISchemaIfCond,
                              features: List[QAPISchemaFeature],
-                             variants: QAPISchemaVariants) -> None:
+                             alternatives: QAPISchemaAlternatives) -> None:
         with ifcontext(ifcond, self._genh):
             self._genh.preamble_add(gen_fwd_object_or_array(name))
         self._genh.add(gen_object(name, ifcond, None,
-                                  [variants.tag_member], variants))
+                                  [alternatives.tag_member], alternatives))
         with ifcontext(ifcond, self._genh, self._genc):
             self._gen_type_cleanup(name)
 
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index c56ea4d724..725bfcef50 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -28,6 +28,7 @@
 )
 from .schema import (
     QAPISchema,
+    QAPISchemaAlternatives,
     QAPISchemaEnumMember,
     QAPISchemaEnumType,
     QAPISchemaFeature,
@@ -222,7 +223,8 @@ def gen_visit_enum(name: str) -> str:
                  c_name=c_name(name))
 
 
-def gen_visit_alternate(name: str, variants: QAPISchemaVariants) -> str:
+def gen_visit_alternate(name: str,
+                        alternatives: QAPISchemaAlternatives) -> str:
     ret = mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name,
@@ -244,7 +246,7 @@ def gen_visit_alternate(name: str, variants: QAPISchemaVariants) -> str:
 ''',
                 c_name=c_name(name))
 
-    for var in variants.variants:
+    for var in alternatives.variants:
         ret += var.ifcond.gen_if()
         ret += mcgen('''
     case %(case)s:
@@ -414,10 +416,10 @@ def visit_alternate_type(self,
                              info: Optional[QAPISourceInfo],
                              ifcond: QAPISchemaIfCond,
                              features: List[QAPISchemaFeature],
-                             variants: QAPISchemaVariants) -> None:
+                             alternatives: QAPISchemaAlternatives) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.add(gen_visit_decl(name))
-            self._genc.add(gen_visit_alternate(name, variants))
+            self._genc.add(gen_visit_alternate(name, alternatives))
 
 
 def gen_visit(schema: QAPISchema,

Re: [PATCH 13/19] qapi/schema: fix typing for QAPISchemaVariants.tag_member

[Markus Armbruster]

Hmm, there's more union-specific code to move out of the base.  Revised
patch:

diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index 658c288f8f..4a2e62d919 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -328,7 +328,8 @@ def visit_object_type(self, name, info, ifcond, features,
                       + self._nodes_for_sections(doc)
                       + self._nodes_for_if_section(ifcond))
 
-    def visit_alternate_type(self, name, info, ifcond, features, variants):
+    def visit_alternate_type(self, name, info, ifcond, features,
+                             alternatives):
         doc = self._cur_doc
         self._add_doc('Alternate',
                       self._nodes_for_members(doc, 'Members')
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index c38df61a6d..e5cea8004e 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -26,6 +26,7 @@
 from .gen import QAPISchemaMonolithicCVisitor
 from .schema import (
     QAPISchema,
+    QAPISchemaAlternatives,
     QAPISchemaArrayType,
     QAPISchemaBuiltinType,
     QAPISchemaEntity,
@@ -343,12 +344,12 @@ def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo],
     def visit_alternate_type(self, name: str, info: Optional[QAPISourceInfo],
                              ifcond: QAPISchemaIfCond,
                              features: List[QAPISchemaFeature],
-                             variants: QAPISchemaVariants) -> None:
+                             alternatives: QAPISchemaAlternatives) -> None:
         self._gen_tree(
             name, 'alternate',
             {'members': [Annotated({'type': self._use_type(m.type)},
                                    m.ifcond)
-                         for m in variants.variants]},
+                         for m in alternatives.variants]},
             ifcond, features
         )
 
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 0d9a70ab4c..f64e337ba2 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -187,7 +187,8 @@ def visit_object_type_flat(self, name, info, ifcond, features,
                                members, variants):
         pass
 
-    def visit_alternate_type(self, name, info, ifcond, features, variants):
+    def visit_alternate_type(self, name, info, ifcond, features,
+                             alternatives):
         pass
 
     def visit_command(self, name, info, ifcond, features,
@@ -563,8 +564,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
 
     def __init__(self, name, info, doc, ifcond, features, variants):
         super().__init__(name, info, doc, ifcond, features)
-        assert isinstance(variants, QAPISchemaVariants)
-        assert variants.tag_member
+        assert isinstance(variants, QAPISchemaAlternatives)
         variants.set_defined_in(name)
         variants.tag_member.set_defined_in(self.name)
         self.variants = variants
@@ -625,19 +625,12 @@ def visit(self, visitor):
             self.name, self.info, self.ifcond, self.features, self.variants)
 
 
-class QAPISchemaVariants:
-    def __init__(self, tag_name, info, tag_member, variants):
-        # Unions pass tag_name but not tag_member.
-        # Alternates pass tag_member but not tag_name.
-        # After check(), tag_member is always set.
-        assert bool(tag_member) != bool(tag_name)
-        assert (isinstance(tag_name, str) or
-                isinstance(tag_member, QAPISchemaObjectTypeMember))
+class QAPISchemaVariantsBase:
+    def __init__(self, info, variants):
         for v in variants:
             assert isinstance(v, QAPISchemaVariant)
-        self._tag_name = tag_name
         self.info = info
-        self.tag_member = tag_member
+        self.tag_member = None
         self.variants = variants
 
     def set_defined_in(self, name):
@@ -645,66 +638,8 @@ def set_defined_in(self, name):
             v.set_defined_in(name)
 
     def check(self, schema, seen):
-        if self._tag_name:      # union
-            self.tag_member = seen.get(c_name(self._tag_name))
-            base = "'base'"
-            # Pointing to the base type when not implicit would be
-            # nice, but we don't know it here
-            if not self.tag_member or self._tag_name != self.tag_member.name:
-                raise QAPISemError(
-                    self.info,
-                    "discriminator '%s' is not a member of %s"
-                    % (self._tag_name, base))
-            # Here we do:
-            base_type = schema.resolve_type(self.tag_member.defined_in)
-            if not base_type.is_implicit():
-                base = "base type '%s'" % self.tag_member.defined_in
-            if not isinstance(self.tag_member.type, QAPISchemaEnumType):
-                raise QAPISemError(
-                    self.info,
-                    "discriminator member '%s' of %s must be of enum type"
-                    % (self._tag_name, base))
-            if self.tag_member.optional:
-                raise QAPISemError(
-                    self.info,
-                    "discriminator member '%s' of %s must not be optional"
-                    % (self._tag_name, base))
-            if self.tag_member.ifcond.is_present():
-                raise QAPISemError(
-                    self.info,
-                    "discriminator member '%s' of %s must not be conditional"
-                    % (self._tag_name, base))
-        else:                   # alternate
-            assert isinstance(self.tag_member.type, QAPISchemaEnumType)
-            assert not self.tag_member.optional
-            assert not self.tag_member.ifcond.is_present()
-        if self._tag_name:      # union
-            # branches that are not explicitly covered get an empty type
-            cases = {v.name for v in self.variants}
-            for m in self.tag_member.type.members:
-                if m.name not in cases:
-                    v = QAPISchemaVariant(m.name, self.info,
-                                          'q_empty', m.ifcond)
-                    v.set_defined_in(self.tag_member.defined_in)
-                    self.variants.append(v)
-        if not self.variants:
-            raise QAPISemError(self.info, "union has no branches")
         for v in self.variants:
             v.check(schema)
-            # Union names must match enum values; alternate names are
-            # checked separately. Use 'seen' to tell the two apart.
-            if seen:
-                if v.name not in self.tag_member.type.member_names():
-                    raise QAPISemError(
-                        self.info,
-                        "branch '%s' is not a value of %s"
-                        % (v.name, self.tag_member.type.describe()))
-                if not isinstance(v.type, QAPISchemaObjectType):
-                    raise QAPISemError(
-                        self.info,
-                        "%s cannot use %s"
-                        % (v.describe(self.info), v.type.describe()))
-                v.type.check(schema)
 
     def check_clash(self, info, seen):
         for v in self.variants:
@@ -713,6 +648,79 @@ def check_clash(self, info, seen):
             v.type.check_clash(info, dict(seen))
 
 
+class QAPISchemaVariants(QAPISchemaVariantsBase):
+    def __init__(self, info, variants, tag_name):
+        assert isinstance(tag_name, str)
+        super().__init__(info, variants)
+        self._tag_name = tag_name
+
+    def check(self, schema, seen):
+        self.tag_member = seen.get(c_name(self._tag_name))
+        base = "'base'"
+        # Pointing to the base type when not implicit would be
+        # nice, but we don't know it here
+        if not self.tag_member or self._tag_name != self.tag_member.name:
+            raise QAPISemError(
+                self.info,
+                "discriminator '%s' is not a member of %s"
+                % (self._tag_name, base))
+        # Here we do:
+        base_type = schema.resolve_type(self.tag_member.defined_in)
+        if not base_type.is_implicit():
+            base = "base type '%s'" % self.tag_member.defined_in
+        if not isinstance(self.tag_member.type, QAPISchemaEnumType):
+            raise QAPISemError(
+                self.info,
+                "discriminator member '%s' of %s must be of enum type"
+                % (self._tag_name, base))
+        if self.tag_member.optional:
+            raise QAPISemError(
+                self.info,
+                "discriminator member '%s' of %s must not be optional"
+                % (self._tag_name, base))
+        if self.tag_member.ifcond.is_present():
+            raise QAPISemError(
+                self.info,
+                "discriminator member '%s' of %s must not be conditional"
+                % (self._tag_name, base))
+        # branches that are not explicitly covered get an empty type
+        cases = {v.name for v in self.variants}
+        for m in self.tag_member.type.members:
+            if m.name not in cases:
+                v = QAPISchemaVariant(m.name, self.info,
+                                      'q_empty', m.ifcond)
+                v.set_defined_in(self.tag_member.defined_in)
+                self.variants.append(v)
+        if not self.variants:
+            raise QAPISemError(self.info, "union has no branches")
+        super().check(schema, seen)
+        for v in self.variants:
+            if v.name not in self.tag_member.type.member_names():
+                raise QAPISemError(
+                    self.info,
+                    "branch '%s' is not a value of %s"
+                    % (v.name, self.tag_member.type.describe()))
+            if not isinstance(v.type, QAPISchemaObjectType):
+                raise QAPISemError(
+                    self.info,
+                    "%s cannot use %s"
+                    % (v.describe(self.info), v.type.describe()))
+            v.type.check(schema)
+
+
+class QAPISchemaAlternatives(QAPISchemaVariantsBase):
+    def __init__(self, info, variants, tag_member):
+        assert isinstance(tag_member, QAPISchemaObjectTypeMember)
+        super().__init__(info, variants)
+        self.tag_member = tag_member
+
+    def check(self, schema, seen):
+        super().check(schema, seen)
+        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        assert not self.tag_member.optional
+        assert not self.tag_member.ifcond.is_present()
+
+
 class QAPISchemaMember:
     """ Represents object members, enum members and features """
     role = 'member'
@@ -1184,7 +1192,7 @@ def _def_union_type(self, expr: QAPIExpression):
             QAPISchemaObjectType(name, info, expr.doc, ifcond, features,
                                  base, members,
                                  QAPISchemaVariants(
-                                     tag_name, info, None, variants)))
+                                     info, variants, tag_name)))
 
     def _def_alternate_type(self, expr: QAPIExpression):
         name = expr['alternate']
@@ -1202,7 +1210,7 @@ def _def_alternate_type(self, expr: QAPIExpression):
         self._def_definition(
             QAPISchemaAlternateType(
                 name, info, expr.doc, ifcond, features,
-                QAPISchemaVariants(None, info, tag_member, variants)))
+                QAPISchemaAlternatives(info, variants, tag_member)))
 
     def _def_command(self, expr: QAPIExpression):
         name = expr['command']
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index c39d054d2c..05da30b855 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -23,6 +23,7 @@
 )
 from .schema import (
     QAPISchema,
+    QAPISchemaAlternatives,
     QAPISchemaEnumMember,
     QAPISchemaFeature,
     QAPISchemaIfCond,
@@ -369,11 +370,11 @@ def visit_alternate_type(self,
                              info: Optional[QAPISourceInfo],
                              ifcond: QAPISchemaIfCond,
                              features: List[QAPISchemaFeature],
-                             variants: QAPISchemaVariants) -> None:
+                             alternatives: QAPISchemaAlternatives) -> None:
         with ifcontext(ifcond, self._genh):
             self._genh.preamble_add(gen_fwd_object_or_array(name))
         self._genh.add(gen_object(name, ifcond, None,
-                                  [variants.tag_member], variants))
+                                  [alternatives.tag_member], alternatives))
         with ifcontext(ifcond, self._genh, self._genc):
             self._gen_type_cleanup(name)
 
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index c56ea4d724..725bfcef50 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -28,6 +28,7 @@
 )
 from .schema import (
     QAPISchema,
+    QAPISchemaAlternatives,
     QAPISchemaEnumMember,
     QAPISchemaEnumType,
     QAPISchemaFeature,
@@ -222,7 +223,8 @@ def gen_visit_enum(name: str) -> str:
                  c_name=c_name(name))
 
 
-def gen_visit_alternate(name: str, variants: QAPISchemaVariants) -> str:
+def gen_visit_alternate(name: str,
+                        alternatives: QAPISchemaAlternatives) -> str:
     ret = mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name,
@@ -244,7 +246,7 @@ def gen_visit_alternate(name: str, variants: QAPISchemaVariants) -> str:
 ''',
                 c_name=c_name(name))
 
-    for var in variants.variants:
+    for var in alternatives.variants:
         ret += var.ifcond.gen_if()
         ret += mcgen('''
     case %(case)s:
@@ -414,10 +416,10 @@ def visit_alternate_type(self,
                              info: Optional[QAPISourceInfo],
                              ifcond: QAPISchemaIfCond,
                              features: List[QAPISchemaFeature],
-                             variants: QAPISchemaVariants) -> None:
+                             alternatives: QAPISchemaAlternatives) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.add(gen_visit_decl(name))
-            self._genc.add(gen_visit_alternate(name, variants))
+            self._genc.add(gen_visit_alternate(name, alternatives))
 
 
 def gen_visit(schema: QAPISchema,
diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
index 14f7b62a44..b66ceb81b8 100755
--- a/tests/qapi-schema/test-qapi.py
+++ b/tests/qapi-schema/test-qapi.py
@@ -61,9 +61,10 @@ def visit_object_type(self, name, info, ifcond, features,
         self._print_if(ifcond)
         self._print_features(features)
 
-    def visit_alternate_type(self, name, info, ifcond, features, variants):
+    def visit_alternate_type(self, name, info, ifcond, features,
+                             alternatives):
         print('alternate %s' % name)
-        self._print_variants(variants)
+        self._print_variants(alternatives)
         self._print_if(ifcond)
         self._print_features(features)
 

Re: [PATCH 13/19] qapi/schema: fix typing for QAPISchemaVariants.tag_member

[Markus Armbruster]

Still more...

diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index 658c288f8f..4a2e62d919 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -328,7 +328,8 @@ def visit_object_type(self, name, info, ifcond, features,
                       + self._nodes_for_sections(doc)
                       + self._nodes_for_if_section(ifcond))
 
-    def visit_alternate_type(self, name, info, ifcond, features, variants):
+    def visit_alternate_type(self, name, info, ifcond, features,
+                             alternatives):
         doc = self._cur_doc
         self._add_doc('Alternate',
                       self._nodes_for_members(doc, 'Members')
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index c38df61a6d..e5cea8004e 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -26,6 +26,7 @@
 from .gen import QAPISchemaMonolithicCVisitor
 from .schema import (
     QAPISchema,
+    QAPISchemaAlternatives,
     QAPISchemaArrayType,
     QAPISchemaBuiltinType,
     QAPISchemaEntity,
@@ -343,12 +344,12 @@ def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo],
     def visit_alternate_type(self, name: str, info: Optional[QAPISourceInfo],
                              ifcond: QAPISchemaIfCond,
                              features: List[QAPISchemaFeature],
-                             variants: QAPISchemaVariants) -> None:
+                             alternatives: QAPISchemaAlternatives) -> None:
         self._gen_tree(
             name, 'alternate',
             {'members': [Annotated({'type': self._use_type(m.type)},
                                    m.ifcond)
-                         for m in variants.variants]},
+                         for m in alternatives.variants]},
             ifcond, features
         )
 
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 0d9a70ab4c..f18aac7199 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -187,7 +187,8 @@ def visit_object_type_flat(self, name, info, ifcond, features,
                                members, variants):
         pass
 
-    def visit_alternate_type(self, name, info, ifcond, features, variants):
+    def visit_alternate_type(self, name, info, ifcond, features,
+                             alternatives):
         pass
 
     def visit_command(self, name, info, ifcond, features,
@@ -563,8 +564,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
 
     def __init__(self, name, info, doc, ifcond, features, variants):
         super().__init__(name, info, doc, ifcond, features)
-        assert isinstance(variants, QAPISchemaVariants)
-        assert variants.tag_member
+        assert isinstance(variants, QAPISchemaAlternatives)
         variants.set_defined_in(name)
         variants.tag_member.set_defined_in(self.name)
         self.variants = variants
@@ -625,19 +625,12 @@ def visit(self, visitor):
             self.name, self.info, self.ifcond, self.features, self.variants)
 
 
-class QAPISchemaVariants:
-    def __init__(self, tag_name, info, tag_member, variants):
-        # Unions pass tag_name but not tag_member.
-        # Alternates pass tag_member but not tag_name.
-        # After check(), tag_member is always set.
-        assert bool(tag_member) != bool(tag_name)
-        assert (isinstance(tag_name, str) or
-                isinstance(tag_member, QAPISchemaObjectTypeMember))
+class QAPISchemaVariantsBase:
+    def __init__(self, info, variants):
         for v in variants:
             assert isinstance(v, QAPISchemaVariant)
-        self._tag_name = tag_name
         self.info = info
-        self.tag_member = tag_member
+        self.tag_member = None
         self.variants = variants
 
     def set_defined_in(self, name):
@@ -645,66 +638,68 @@ def set_defined_in(self, name):
             v.set_defined_in(name)
 
     def check(self, schema, seen):
-        if self._tag_name:      # union
-            self.tag_member = seen.get(c_name(self._tag_name))
-            base = "'base'"
-            # Pointing to the base type when not implicit would be
-            # nice, but we don't know it here
-            if not self.tag_member or self._tag_name != self.tag_member.name:
-                raise QAPISemError(
-                    self.info,
-                    "discriminator '%s' is not a member of %s"
-                    % (self._tag_name, base))
-            # Here we do:
-            base_type = schema.resolve_type(self.tag_member.defined_in)
-            if not base_type.is_implicit():
-                base = "base type '%s'" % self.tag_member.defined_in
-            if not isinstance(self.tag_member.type, QAPISchemaEnumType):
-                raise QAPISemError(
-                    self.info,
-                    "discriminator member '%s' of %s must be of enum type"
-                    % (self._tag_name, base))
-            if self.tag_member.optional:
-                raise QAPISemError(
-                    self.info,
-                    "discriminator member '%s' of %s must not be optional"
-                    % (self._tag_name, base))
-            if self.tag_member.ifcond.is_present():
-                raise QAPISemError(
-                    self.info,
-                    "discriminator member '%s' of %s must not be conditional"
-                    % (self._tag_name, base))
-        else:                   # alternate
-            assert isinstance(self.tag_member.type, QAPISchemaEnumType)
-            assert not self.tag_member.optional
-            assert not self.tag_member.ifcond.is_present()
-        if self._tag_name:      # union
-            # branches that are not explicitly covered get an empty type
-            cases = {v.name for v in self.variants}
-            for m in self.tag_member.type.members:
-                if m.name not in cases:
-                    v = QAPISchemaVariant(m.name, self.info,
-                                          'q_empty', m.ifcond)
-                    v.set_defined_in(self.tag_member.defined_in)
-                    self.variants.append(v)
-        if not self.variants:
-            raise QAPISemError(self.info, "union has no branches")
         for v in self.variants:
             v.check(schema)
-            # Union names must match enum values; alternate names are
-            # checked separately. Use 'seen' to tell the two apart.
-            if seen:
-                if v.name not in self.tag_member.type.member_names():
-                    raise QAPISemError(
-                        self.info,
-                        "branch '%s' is not a value of %s"
-                        % (v.name, self.tag_member.type.describe()))
-                if not isinstance(v.type, QAPISchemaObjectType):
-                    raise QAPISemError(
-                        self.info,
-                        "%s cannot use %s"
-                        % (v.describe(self.info), v.type.describe()))
-                v.type.check(schema)
+
+
+class QAPISchemaVariants(QAPISchemaVariantsBase):
+    def __init__(self, info, variants, tag_name):
+        assert isinstance(tag_name, str)
+        super().__init__(info, variants)
+        self._tag_name = tag_name
+
+    def check(self, schema, seen):
+        self.tag_member = seen.get(c_name(self._tag_name))
+        base = "'base'"
+        # Pointing to the base type when not implicit would be
+        # nice, but we don't know it here
+        if not self.tag_member or self._tag_name != self.tag_member.name:
+            raise QAPISemError(
+                self.info,
+                "discriminator '%s' is not a member of %s"
+                % (self._tag_name, base))
+        # Here we do:
+        base_type = schema.resolve_type(self.tag_member.defined_in)
+        if not base_type.is_implicit():
+            base = "base type '%s'" % self.tag_member.defined_in
+        if not isinstance(self.tag_member.type, QAPISchemaEnumType):
+            raise QAPISemError(
+                self.info,
+                "discriminator member '%s' of %s must be of enum type"
+                % (self._tag_name, base))
+        if self.tag_member.optional:
+            raise QAPISemError(
+                self.info,
+                "discriminator member '%s' of %s must not be optional"
+                % (self._tag_name, base))
+        if self.tag_member.ifcond.is_present():
+            raise QAPISemError(
+                self.info,
+                "discriminator member '%s' of %s must not be conditional"
+                % (self._tag_name, base))
+        # branches that are not explicitly covered get an empty type
+        cases = {v.name for v in self.variants}
+        for m in self.tag_member.type.members:
+            if m.name not in cases:
+                v = QAPISchemaVariant(m.name, self.info,
+                                      'q_empty', m.ifcond)
+                v.set_defined_in(self.tag_member.defined_in)
+                self.variants.append(v)
+        if not self.variants:
+            raise QAPISemError(self.info, "union has no branches")
+        super().check(schema, seen)
+        for v in self.variants:
+            if v.name not in self.tag_member.type.member_names():
+                raise QAPISemError(
+                    self.info,
+                    "branch '%s' is not a value of %s"
+                    % (v.name, self.tag_member.type.describe()))
+            if not isinstance(v.type, QAPISchemaObjectType):
+                raise QAPISemError(
+                    self.info,
+                    "%s cannot use %s"
+                    % (v.describe(self.info), v.type.describe()))
+            v.type.check(schema)
 
     def check_clash(self, info, seen):
         for v in self.variants:
@@ -713,6 +708,19 @@ def check_clash(self, info, seen):
             v.type.check_clash(info, dict(seen))
 
 
+class QAPISchemaAlternatives(QAPISchemaVariantsBase):
+    def __init__(self, info, variants, tag_member):
+        assert isinstance(tag_member, QAPISchemaObjectTypeMember)
+        super().__init__(info, variants)
+        self.tag_member = tag_member
+
+    def check(self, schema, seen):
+        super().check(schema, seen)
+        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
+        assert not self.tag_member.optional
+        assert not self.tag_member.ifcond.is_present()
+
+
 class QAPISchemaMember:
     """ Represents object members, enum members and features """
     role = 'member'
@@ -1184,7 +1192,7 @@ def _def_union_type(self, expr: QAPIExpression):
             QAPISchemaObjectType(name, info, expr.doc, ifcond, features,
                                  base, members,
                                  QAPISchemaVariants(
-                                     tag_name, info, None, variants)))
+                                     info, variants, tag_name)))
 
     def _def_alternate_type(self, expr: QAPIExpression):
         name = expr['alternate']
@@ -1202,7 +1210,7 @@ def _def_alternate_type(self, expr: QAPIExpression):
         self._def_definition(
             QAPISchemaAlternateType(
                 name, info, expr.doc, ifcond, features,
-                QAPISchemaVariants(None, info, tag_member, variants)))
+                QAPISchemaAlternatives(info, variants, tag_member)))
 
     def _def_command(self, expr: QAPIExpression):
         name = expr['command']
diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
index c39d054d2c..05da30b855 100644
--- a/scripts/qapi/types.py
+++ b/scripts/qapi/types.py
@@ -23,6 +23,7 @@
 )
 from .schema import (
     QAPISchema,
+    QAPISchemaAlternatives,
     QAPISchemaEnumMember,
     QAPISchemaFeature,
     QAPISchemaIfCond,
@@ -369,11 +370,11 @@ def visit_alternate_type(self,
                              info: Optional[QAPISourceInfo],
                              ifcond: QAPISchemaIfCond,
                              features: List[QAPISchemaFeature],
-                             variants: QAPISchemaVariants) -> None:
+                             alternatives: QAPISchemaAlternatives) -> None:
         with ifcontext(ifcond, self._genh):
             self._genh.preamble_add(gen_fwd_object_or_array(name))
         self._genh.add(gen_object(name, ifcond, None,
-                                  [variants.tag_member], variants))
+                                  [alternatives.tag_member], alternatives))
         with ifcontext(ifcond, self._genh, self._genc):
             self._gen_type_cleanup(name)
 
diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
index c56ea4d724..725bfcef50 100644
--- a/scripts/qapi/visit.py
+++ b/scripts/qapi/visit.py
@@ -28,6 +28,7 @@
 )
 from .schema import (
     QAPISchema,
+    QAPISchemaAlternatives,
     QAPISchemaEnumMember,
     QAPISchemaEnumType,
     QAPISchemaFeature,
@@ -222,7 +223,8 @@ def gen_visit_enum(name: str) -> str:
                  c_name=c_name(name))
 
 
-def gen_visit_alternate(name: str, variants: QAPISchemaVariants) -> str:
+def gen_visit_alternate(name: str,
+                        alternatives: QAPISchemaAlternatives) -> str:
     ret = mcgen('''
 
 bool visit_type_%(c_name)s(Visitor *v, const char *name,
@@ -244,7 +246,7 @@ def gen_visit_alternate(name: str, variants: QAPISchemaVariants) -> str:
 ''',
                 c_name=c_name(name))
 
-    for var in variants.variants:
+    for var in alternatives.variants:
         ret += var.ifcond.gen_if()
         ret += mcgen('''
     case %(case)s:
@@ -414,10 +416,10 @@ def visit_alternate_type(self,
                              info: Optional[QAPISourceInfo],
                              ifcond: QAPISchemaIfCond,
                              features: List[QAPISchemaFeature],
-                             variants: QAPISchemaVariants) -> None:
+                             alternatives: QAPISchemaAlternatives) -> None:
         with ifcontext(ifcond, self._genh, self._genc):
             self._genh.add(gen_visit_decl(name))
-            self._genc.add(gen_visit_alternate(name, variants))
+            self._genc.add(gen_visit_alternate(name, alternatives))
 
 
 def gen_visit(schema: QAPISchema,
diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
index 14f7b62a44..b66ceb81b8 100755
--- a/tests/qapi-schema/test-qapi.py
+++ b/tests/qapi-schema/test-qapi.py
@@ -61,9 +61,10 @@ def visit_object_type(self, name, info, ifcond, features,
         self._print_if(ifcond)
         self._print_features(features)
 
-    def visit_alternate_type(self, name, info, ifcond, features, variants):
+    def visit_alternate_type(self, name, info, ifcond, features,
+                             alternatives):
         print('alternate %s' % name)
-        self._print_variants(variants)
+        self._print_variants(alternatives)
         self._print_if(ifcond)
         self._print_features(features)
 

Re: [PATCH 13/19] qapi/schema: fix typing for QAPISchemaVariants.tag_member

[John Snow]

On Wed, Jan 17, 2024 at 5:53 AM Markus Armbruster <armbru@redhat.com> wrote:
>
> Still more...
>

Would you hate me if I suggested that we punt this to after type
checking is applied? i.e. let's do the stupid @property thing for now,
and we'll rebase this proposal and put it in right afterwards.

(Admittedly, it's just easier for me to review the impact on the
typing work if I already have that baseline to work from...)

> diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
> index 658c288f8f..4a2e62d919 100644
> --- a/docs/sphinx/qapidoc.py
> +++ b/docs/sphinx/qapidoc.py
> @@ -328,7 +328,8 @@ def visit_object_type(self, name, info, ifcond, features,
>                        + self._nodes_for_sections(doc)
>                        + self._nodes_for_if_section(ifcond))
>
> -    def visit_alternate_type(self, name, info, ifcond, features, variants):
> +    def visit_alternate_type(self, name, info, ifcond, features,
> +                             alternatives):
>          doc = self._cur_doc
>          self._add_doc('Alternate',
>                        self._nodes_for_members(doc, 'Members')
> diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
> index c38df61a6d..e5cea8004e 100644
> --- a/scripts/qapi/introspect.py
> +++ b/scripts/qapi/introspect.py
> @@ -26,6 +26,7 @@
>  from .gen import QAPISchemaMonolithicCVisitor
>  from .schema import (
>      QAPISchema,
> +    QAPISchemaAlternatives,
>      QAPISchemaArrayType,
>      QAPISchemaBuiltinType,
>      QAPISchemaEntity,
> @@ -343,12 +344,12 @@ def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo],
>      def visit_alternate_type(self, name: str, info: Optional[QAPISourceInfo],
>                               ifcond: QAPISchemaIfCond,
>                               features: List[QAPISchemaFeature],
> -                             variants: QAPISchemaVariants) -> None:
> +                             alternatives: QAPISchemaAlternatives) -> None:
>          self._gen_tree(
>              name, 'alternate',
>              {'members': [Annotated({'type': self._use_type(m.type)},
>                                     m.ifcond)
> -                         for m in variants.variants]},
> +                         for m in alternatives.variants]},
>              ifcond, features
>          )
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 0d9a70ab4c..f18aac7199 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -187,7 +187,8 @@ def visit_object_type_flat(self, name, info, ifcond, features,
>                                 members, variants):
>          pass
>
> -    def visit_alternate_type(self, name, info, ifcond, features, variants):
> +    def visit_alternate_type(self, name, info, ifcond, features,
> +                             alternatives):
>          pass
>
>      def visit_command(self, name, info, ifcond, features,
> @@ -563,8 +564,7 @@ class QAPISchemaAlternateType(QAPISchemaType):
>
>      def __init__(self, name, info, doc, ifcond, features, variants):
>          super().__init__(name, info, doc, ifcond, features)
> -        assert isinstance(variants, QAPISchemaVariants)
> -        assert variants.tag_member
> +        assert isinstance(variants, QAPISchemaAlternatives)
>          variants.set_defined_in(name)
>          variants.tag_member.set_defined_in(self.name)
>          self.variants = variants
> @@ -625,19 +625,12 @@ def visit(self, visitor):
>              self.name, self.info, self.ifcond, self.features, self.variants)
>
>
> -class QAPISchemaVariants:
> -    def __init__(self, tag_name, info, tag_member, variants):
> -        # Unions pass tag_name but not tag_member.
> -        # Alternates pass tag_member but not tag_name.
> -        # After check(), tag_member is always set.
> -        assert bool(tag_member) != bool(tag_name)
> -        assert (isinstance(tag_name, str) or
> -                isinstance(tag_member, QAPISchemaObjectTypeMember))
> +class QAPISchemaVariantsBase:
> +    def __init__(self, info, variants):
>          for v in variants:
>              assert isinstance(v, QAPISchemaVariant)
> -        self._tag_name = tag_name
>          self.info = info
> -        self.tag_member = tag_member
> +        self.tag_member = None
>          self.variants = variants
>
>      def set_defined_in(self, name):
> @@ -645,66 +638,68 @@ def set_defined_in(self, name):
>              v.set_defined_in(name)
>
>      def check(self, schema, seen):
> -        if self._tag_name:      # union
> -            self.tag_member = seen.get(c_name(self._tag_name))
> -            base = "'base'"
> -            # Pointing to the base type when not implicit would be
> -            # nice, but we don't know it here
> -            if not self.tag_member or self._tag_name != self.tag_member.name:
> -                raise QAPISemError(
> -                    self.info,
> -                    "discriminator '%s' is not a member of %s"
> -                    % (self._tag_name, base))
> -            # Here we do:
> -            base_type = schema.resolve_type(self.tag_member.defined_in)
> -            if not base_type.is_implicit():
> -                base = "base type '%s'" % self.tag_member.defined_in
> -            if not isinstance(self.tag_member.type, QAPISchemaEnumType):
> -                raise QAPISemError(
> -                    self.info,
> -                    "discriminator member '%s' of %s must be of enum type"
> -                    % (self._tag_name, base))
> -            if self.tag_member.optional:
> -                raise QAPISemError(
> -                    self.info,
> -                    "discriminator member '%s' of %s must not be optional"
> -                    % (self._tag_name, base))
> -            if self.tag_member.ifcond.is_present():
> -                raise QAPISemError(
> -                    self.info,
> -                    "discriminator member '%s' of %s must not be conditional"
> -                    % (self._tag_name, base))
> -        else:                   # alternate
> -            assert isinstance(self.tag_member.type, QAPISchemaEnumType)
> -            assert not self.tag_member.optional
> -            assert not self.tag_member.ifcond.is_present()
> -        if self._tag_name:      # union
> -            # branches that are not explicitly covered get an empty type
> -            cases = {v.name for v in self.variants}
> -            for m in self.tag_member.type.members:
> -                if m.name not in cases:
> -                    v = QAPISchemaVariant(m.name, self.info,
> -                                          'q_empty', m.ifcond)
> -                    v.set_defined_in(self.tag_member.defined_in)
> -                    self.variants.append(v)
> -        if not self.variants:
> -            raise QAPISemError(self.info, "union has no branches")
>          for v in self.variants:
>              v.check(schema)
> -            # Union names must match enum values; alternate names are
> -            # checked separately. Use 'seen' to tell the two apart.
> -            if seen:
> -                if v.name not in self.tag_member.type.member_names():
> -                    raise QAPISemError(
> -                        self.info,
> -                        "branch '%s' is not a value of %s"
> -                        % (v.name, self.tag_member.type.describe()))
> -                if not isinstance(v.type, QAPISchemaObjectType):
> -                    raise QAPISemError(
> -                        self.info,
> -                        "%s cannot use %s"
> -                        % (v.describe(self.info), v.type.describe()))
> -                v.type.check(schema)
> +
> +
> +class QAPISchemaVariants(QAPISchemaVariantsBase):
> +    def __init__(self, info, variants, tag_name):
> +        assert isinstance(tag_name, str)
> +        super().__init__(info, variants)
> +        self._tag_name = tag_name
> +
> +    def check(self, schema, seen):
> +        self.tag_member = seen.get(c_name(self._tag_name))
> +        base = "'base'"
> +        # Pointing to the base type when not implicit would be
> +        # nice, but we don't know it here
> +        if not self.tag_member or self._tag_name != self.tag_member.name:
> +            raise QAPISemError(
> +                self.info,
> +                "discriminator '%s' is not a member of %s"
> +                % (self._tag_name, base))
> +        # Here we do:
> +        base_type = schema.resolve_type(self.tag_member.defined_in)
> +        if not base_type.is_implicit():
> +            base = "base type '%s'" % self.tag_member.defined_in
> +        if not isinstance(self.tag_member.type, QAPISchemaEnumType):
> +            raise QAPISemError(
> +                self.info,
> +                "discriminator member '%s' of %s must be of enum type"
> +                % (self._tag_name, base))
> +        if self.tag_member.optional:
> +            raise QAPISemError(
> +                self.info,
> +                "discriminator member '%s' of %s must not be optional"
> +                % (self._tag_name, base))
> +        if self.tag_member.ifcond.is_present():
> +            raise QAPISemError(
> +                self.info,
> +                "discriminator member '%s' of %s must not be conditional"
> +                % (self._tag_name, base))
> +        # branches that are not explicitly covered get an empty type
> +        cases = {v.name for v in self.variants}
> +        for m in self.tag_member.type.members:
> +            if m.name not in cases:
> +                v = QAPISchemaVariant(m.name, self.info,
> +                                      'q_empty', m.ifcond)
> +                v.set_defined_in(self.tag_member.defined_in)
> +                self.variants.append(v)
> +        if not self.variants:
> +            raise QAPISemError(self.info, "union has no branches")
> +        super().check(schema, seen)
> +        for v in self.variants:
> +            if v.name not in self.tag_member.type.member_names():
> +                raise QAPISemError(
> +                    self.info,
> +                    "branch '%s' is not a value of %s"
> +                    % (v.name, self.tag_member.type.describe()))
> +            if not isinstance(v.type, QAPISchemaObjectType):
> +                raise QAPISemError(
> +                    self.info,
> +                    "%s cannot use %s"
> +                    % (v.describe(self.info), v.type.describe()))
> +            v.type.check(schema)
>
>      def check_clash(self, info, seen):
>          for v in self.variants:
> @@ -713,6 +708,19 @@ def check_clash(self, info, seen):
>              v.type.check_clash(info, dict(seen))
>
>
> +class QAPISchemaAlternatives(QAPISchemaVariantsBase):
> +    def __init__(self, info, variants, tag_member):
> +        assert isinstance(tag_member, QAPISchemaObjectTypeMember)
> +        super().__init__(info, variants)
> +        self.tag_member = tag_member
> +
> +    def check(self, schema, seen):
> +        super().check(schema, seen)
> +        assert isinstance(self.tag_member.type, QAPISchemaEnumType)
> +        assert not self.tag_member.optional
> +        assert not self.tag_member.ifcond.is_present()
> +
> +
>  class QAPISchemaMember:
>      """ Represents object members, enum members and features """
>      role = 'member'
> @@ -1184,7 +1192,7 @@ def _def_union_type(self, expr: QAPIExpression):
>              QAPISchemaObjectType(name, info, expr.doc, ifcond, features,
>                                   base, members,
>                                   QAPISchemaVariants(
> -                                     tag_name, info, None, variants)))
> +                                     info, variants, tag_name)))
>
>      def _def_alternate_type(self, expr: QAPIExpression):
>          name = expr['alternate']
> @@ -1202,7 +1210,7 @@ def _def_alternate_type(self, expr: QAPIExpression):
>          self._def_definition(
>              QAPISchemaAlternateType(
>                  name, info, expr.doc, ifcond, features,
> -                QAPISchemaVariants(None, info, tag_member, variants)))
> +                QAPISchemaAlternatives(info, variants, tag_member)))
>
>      def _def_command(self, expr: QAPIExpression):
>          name = expr['command']
> diff --git a/scripts/qapi/types.py b/scripts/qapi/types.py
> index c39d054d2c..05da30b855 100644
> --- a/scripts/qapi/types.py
> +++ b/scripts/qapi/types.py
> @@ -23,6 +23,7 @@
>  )
>  from .schema import (
>      QAPISchema,
> +    QAPISchemaAlternatives,
>      QAPISchemaEnumMember,
>      QAPISchemaFeature,
>      QAPISchemaIfCond,
> @@ -369,11 +370,11 @@ def visit_alternate_type(self,
>                               info: Optional[QAPISourceInfo],
>                               ifcond: QAPISchemaIfCond,
>                               features: List[QAPISchemaFeature],
> -                             variants: QAPISchemaVariants) -> None:
> +                             alternatives: QAPISchemaAlternatives) -> None:
>          with ifcontext(ifcond, self._genh):
>              self._genh.preamble_add(gen_fwd_object_or_array(name))
>          self._genh.add(gen_object(name, ifcond, None,
> -                                  [variants.tag_member], variants))
> +                                  [alternatives.tag_member], alternatives))
>          with ifcontext(ifcond, self._genh, self._genc):
>              self._gen_type_cleanup(name)
>
> diff --git a/scripts/qapi/visit.py b/scripts/qapi/visit.py
> index c56ea4d724..725bfcef50 100644
> --- a/scripts/qapi/visit.py
> +++ b/scripts/qapi/visit.py
> @@ -28,6 +28,7 @@
>  )
>  from .schema import (
>      QAPISchema,
> +    QAPISchemaAlternatives,
>      QAPISchemaEnumMember,
>      QAPISchemaEnumType,
>      QAPISchemaFeature,
> @@ -222,7 +223,8 @@ def gen_visit_enum(name: str) -> str:
>                   c_name=c_name(name))
>
>
> -def gen_visit_alternate(name: str, variants: QAPISchemaVariants) -> str:
> +def gen_visit_alternate(name: str,
> +                        alternatives: QAPISchemaAlternatives) -> str:
>      ret = mcgen('''
>
>  bool visit_type_%(c_name)s(Visitor *v, const char *name,
> @@ -244,7 +246,7 @@ def gen_visit_alternate(name: str, variants: QAPISchemaVariants) -> str:
>  ''',
>                  c_name=c_name(name))
>
> -    for var in variants.variants:
> +    for var in alternatives.variants:
>          ret += var.ifcond.gen_if()
>          ret += mcgen('''
>      case %(case)s:
> @@ -414,10 +416,10 @@ def visit_alternate_type(self,
>                               info: Optional[QAPISourceInfo],
>                               ifcond: QAPISchemaIfCond,
>                               features: List[QAPISchemaFeature],
> -                             variants: QAPISchemaVariants) -> None:
> +                             alternatives: QAPISchemaAlternatives) -> None:
>          with ifcontext(ifcond, self._genh, self._genc):
>              self._genh.add(gen_visit_decl(name))
> -            self._genc.add(gen_visit_alternate(name, variants))
> +            self._genc.add(gen_visit_alternate(name, alternatives))
>
>
>  def gen_visit(schema: QAPISchema,
> diff --git a/tests/qapi-schema/test-qapi.py b/tests/qapi-schema/test-qapi.py
> index 14f7b62a44..b66ceb81b8 100755
> --- a/tests/qapi-schema/test-qapi.py
> +++ b/tests/qapi-schema/test-qapi.py
> @@ -61,9 +61,10 @@ def visit_object_type(self, name, info, ifcond, features,
>          self._print_if(ifcond)
>          self._print_features(features)
>
> -    def visit_alternate_type(self, name, info, ifcond, features, variants):
> +    def visit_alternate_type(self, name, info, ifcond, features,
> +                             alternatives):
>          print('alternate %s' % name)
> -        self._print_variants(variants)
> +        self._print_variants(alternatives)
>          self._print_if(ifcond)
>          self._print_features(features)
>
>

[PATCH 14/19] qapi/schema: assert QAPISchemaVariants are QAPISchemaObjectType

[John Snow]

I'm actually not too sure about this one, it seems to hold up at runtime
but instead of lying and coming up with an elaborate ruse as a commit
message I'm just going to admit I just cribbed my own notes from the
last time I typed schema.py and I no longer remember why or if this is
correct.

Cool!

With more seriousness, variants are only guaranteed to house a
QAPISchemaType as per the definition of QAPISchemaObjectTypeMember but
the only classes/types that have a check_clash method are descendents of
QAPISchemaMember and the QAPISchemaVariants class itself.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 476b19aed61..ce5b01b3182 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -717,6 +717,7 @@ def check_clash(self, info, seen):
         for v in self.variants:
             # Reset seen map for each variant, since qapi names from one
             # branch do not affect another branch
+            assert isinstance(v.type, QAPISchemaObjectType)  # I think, anyway?
             v.type.check_clash(info, dict(seen))
 
 
-- 
2.41.0

Re: [PATCH 14/19] qapi/schema: assert QAPISchemaVariants are QAPISchemaObjectType

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> I'm actually not too sure about this one, it seems to hold up at runtime
> but instead of lying and coming up with an elaborate ruse as a commit
> message I'm just going to admit I just cribbed my own notes from the
> last time I typed schema.py and I no longer remember why or if this is
> correct.
>
> Cool!
>
> With more seriousness, variants are only guaranteed to house a
> QAPISchemaType as per the definition of QAPISchemaObjectTypeMember but
> the only classes/types that have a check_clash method are descendents of
> QAPISchemaMember and the QAPISchemaVariants class itself.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/schema.py | 1 +
>  1 file changed, 1 insertion(+)
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 476b19aed61..ce5b01b3182 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -717,6 +717,7 @@ def check_clash(self, info, seen):
>          for v in self.variants:
>              # Reset seen map for each variant, since qapi names from one
>              # branch do not affect another branch
> +            assert isinstance(v.type, QAPISchemaObjectType)  # I think, anyway?
>              v.type.check_clash(info, dict(seen))

Have a look at .check() right above:

       def check(
           self, schema: QAPISchema, seen: Dict[str, QAPISchemaMember]
       ) -> None:
           [...]
           if not self.variants:
               raise QAPISemError(self.info, "union has no branches")
           for v in self.variants:
               v.check(schema)
               # Union names must match enum values; alternate names are
               # checked separately. Use 'seen' to tell the two apart.
               if seen:
                   if v.name not in self.tag_member.type.member_names():
                       raise QAPISemError(
                           self.info,
                           "branch '%s' is not a value of %s"
                           % (v.name, self.tag_member.type.describe()))
--->               if not isinstance(v.type, QAPISchemaObjectType):
--->                   raise QAPISemError(
                           self.info,
                           "%s cannot use %s"
                           % (v.describe(self.info), v.type.describe()))
                   v.type.check(schema)

Since .check() runs before .check_clash(), your assertion holds.

Clearer now?

Re: [PATCH 14/19] qapi/schema: assert QAPISchemaVariants are QAPISchemaObjectType

[John Snow]

On Thu, Nov 23, 2023 at 8:51 AM Markus Armbruster <armbru@redhat.com> wrote:
>
> John Snow <jsnow@redhat.com> writes:
>
> > I'm actually not too sure about this one, it seems to hold up at runtime
> > but instead of lying and coming up with an elaborate ruse as a commit
> > message I'm just going to admit I just cribbed my own notes from the
> > last time I typed schema.py and I no longer remember why or if this is
> > correct.
> >
> > Cool!
> >
> > With more seriousness, variants are only guaranteed to house a
> > QAPISchemaType as per the definition of QAPISchemaObjectTypeMember but
> > the only classes/types that have a check_clash method are descendents of
> > QAPISchemaMember and the QAPISchemaVariants class itself.
> >
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  scripts/qapi/schema.py | 1 +
> >  1 file changed, 1 insertion(+)
> >
> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> > index 476b19aed61..ce5b01b3182 100644
> > --- a/scripts/qapi/schema.py
> > +++ b/scripts/qapi/schema.py
> > @@ -717,6 +717,7 @@ def check_clash(self, info, seen):
> >          for v in self.variants:
> >              # Reset seen map for each variant, since qapi names from one
> >              # branch do not affect another branch
> > +            assert isinstance(v.type, QAPISchemaObjectType)  # I think, anyway?
> >              v.type.check_clash(info, dict(seen))
>
> Have a look at .check() right above:
>
>        def check(
>            self, schema: QAPISchema, seen: Dict[str, QAPISchemaMember]
>        ) -> None:
>            [...]
>            if not self.variants:
>                raise QAPISemError(self.info, "union has no branches")
>            for v in self.variants:
>                v.check(schema)
>                # Union names must match enum values; alternate names are
>                # checked separately. Use 'seen' to tell the two apart.
>                if seen:
>                    if v.name not in self.tag_member.type.member_names():
>                        raise QAPISemError(
>                            self.info,
>                            "branch '%s' is not a value of %s"
>                            % (v.name, self.tag_member.type.describe()))
> --->               if not isinstance(v.type, QAPISchemaObjectType):
> --->                   raise QAPISemError(
>                            self.info,
>                            "%s cannot use %s"
>                            % (v.describe(self.info), v.type.describe()))
>                    v.type.check(schema)
>
> Since .check() runs before .check_clash(), your assertion holds.
>
> Clearer now?
>

OK, I think this just needs a better commit message and comment, then.

--js

[PATCH 15/19] qapi/parser: demote QAPIExpression to Dict[str, Any]

[John Snow]

Dict[str, object] is a stricter type, but with the way that code is
currently arranged, it is infeasible to enforce this strictness.

In particular, although expr.py's entire raison d'être is normalization
and type-checking of QAPI Expressions, that type information is not
"remembered" in any meaningful way by mypy because each individual
expression is not downcast to a specific expression type that holds all
the details of each expression's unique form.

As a result, all of the code in schema.py that deals with actually
creating type-safe specialized structures has no guarantee (myopically)
that the data it is being passed is correct.

There are two ways to solve this:

(1) Re-assert that the incoming data is in the shape we expect it to be, or
(2) Disable type checking for this data.

(1) is appealing to my sense of strictness, but I gotta concede that it
is asinine to re-check the shape of a QAPIExpression in schema.py when
expr.py has just completed that work at length. The duplication of code
and the nightmare thought of needing to update both locations if and
when we change the shape of these structures makes me extremely
reluctant to go down this route.

(2) allows us the chance to miss updating types in the case that types
are updated in expr.py, but it *is* an awful lot simpler and,
importantly, gets us closer to type checking schema.py *at
all*. Something is better than nothing, I'd argue.

So, do the simpler dumber thing and worry about future strictness
improvements later.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/parser.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
index bf31018aef0..b7f08cf36f2 100644
--- a/scripts/qapi/parser.py
+++ b/scripts/qapi/parser.py
@@ -19,6 +19,7 @@
 import re
 from typing import (
     TYPE_CHECKING,
+    Any,
     Dict,
     List,
     Mapping,
@@ -43,7 +44,7 @@
 _ExprValue = Union[List[object], Dict[str, object], str, bool]
 
 
-class QAPIExpression(Dict[str, object]):
+class QAPIExpression(Dict[str, Any]):
     # pylint: disable=too-few-public-methods
     def __init__(self,
                  data: Mapping[str, object],
-- 
2.41.0

Re: [PATCH 15/19] qapi/parser: demote QAPIExpression to Dict[str, Any]

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> Dict[str, object] is a stricter type, but with the way that code is
> currently arranged, it is infeasible to enforce this strictness.
>
> In particular, although expr.py's entire raison d'être is normalization
> and type-checking of QAPI Expressions, that type information is not
> "remembered" in any meaningful way by mypy because each individual
> expression is not downcast to a specific expression type that holds all
> the details of each expression's unique form.
>
> As a result, all of the code in schema.py that deals with actually
> creating type-safe specialized structures has no guarantee (myopically)
> that the data it is being passed is correct.
>
> There are two ways to solve this:
>
> (1) Re-assert that the incoming data is in the shape we expect it to be, or
> (2) Disable type checking for this data.
>
> (1) is appealing to my sense of strictness, but I gotta concede that it
> is asinine to re-check the shape of a QAPIExpression in schema.py when
> expr.py has just completed that work at length. The duplication of code
> and the nightmare thought of needing to update both locations if and
> when we change the shape of these structures makes me extremely
> reluctant to go down this route.
>
> (2) allows us the chance to miss updating types in the case that types
> are updated in expr.py, but it *is* an awful lot simpler and,
> importantly, gets us closer to type checking schema.py *at
> all*. Something is better than nothing, I'd argue.
>
> So, do the simpler dumber thing and worry about future strictness
> improvements later.

Yes.

While Dict[str, object] is stricter than Dict[str, Any], both are miles
away from the actual, recursive type.

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/parser.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> index bf31018aef0..b7f08cf36f2 100644
> --- a/scripts/qapi/parser.py
> +++ b/scripts/qapi/parser.py
> @@ -19,6 +19,7 @@
>  import re
>  from typing import (
>      TYPE_CHECKING,
> +    Any,
>      Dict,
>      List,
>      Mapping,
> @@ -43,7 +44,7 @@
>  _ExprValue = Union[List[object], Dict[str, object], str, bool]
>  
>  
> -class QAPIExpression(Dict[str, object]):
> +class QAPIExpression(Dict[str, Any]):
>      # pylint: disable=too-few-public-methods
>      def __init__(self,
>                   data: Mapping[str, object],

There are several occurences of Dict[str, object] elsewhere.  Would your
argument for dumbing down QAPIExpression apply to (some of) them, too?

Skimming them, I found this in introspect.py:

    # These types are based on structures defined in QEMU's schema, so we
    # lack precise types for them here. Python 3.6 does not offer
    # TypedDict constructs, so they are broadly typed here as simple
    # Python Dicts.
    SchemaInfo = Dict[str, object]
    SchemaInfoEnumMember = Dict[str, object]
    SchemaInfoObject = Dict[str, object]
    SchemaInfoObjectVariant = Dict[str, object]
    SchemaInfoObjectMember = Dict[str, object]
    SchemaInfoCommand = Dict[str, object]

Can we do better now we have 3.8?

Re: [PATCH 15/19] qapi/parser: demote QAPIExpression to Dict[str, Any]

[John Snow]

On Thu, Nov 23, 2023 at 9:12 AM Markus Armbruster <armbru@redhat.com> wrote:
>
> John Snow <jsnow@redhat.com> writes:
>
> > Dict[str, object] is a stricter type, but with the way that code is
> > currently arranged, it is infeasible to enforce this strictness.
> >
> > In particular, although expr.py's entire raison d'être is normalization
> > and type-checking of QAPI Expressions, that type information is not
> > "remembered" in any meaningful way by mypy because each individual
> > expression is not downcast to a specific expression type that holds all
> > the details of each expression's unique form.
> >
> > As a result, all of the code in schema.py that deals with actually
> > creating type-safe specialized structures has no guarantee (myopically)
> > that the data it is being passed is correct.
> >
> > There are two ways to solve this:
> >
> > (1) Re-assert that the incoming data is in the shape we expect it to be, or
> > (2) Disable type checking for this data.
> >
> > (1) is appealing to my sense of strictness, but I gotta concede that it
> > is asinine to re-check the shape of a QAPIExpression in schema.py when
> > expr.py has just completed that work at length. The duplication of code
> > and the nightmare thought of needing to update both locations if and
> > when we change the shape of these structures makes me extremely
> > reluctant to go down this route.
> >
> > (2) allows us the chance to miss updating types in the case that types
> > are updated in expr.py, but it *is* an awful lot simpler and,
> > importantly, gets us closer to type checking schema.py *at
> > all*. Something is better than nothing, I'd argue.
> >
> > So, do the simpler dumber thing and worry about future strictness
> > improvements later.
>
> Yes.
>

(You were right, again.)

> While Dict[str, object] is stricter than Dict[str, Any], both are miles
> away from the actual, recursive type.
>
> > Signed-off-by: John Snow <jsnow@redhat.com>
> > ---
> >  scripts/qapi/parser.py | 3 ++-
> >  1 file changed, 2 insertions(+), 1 deletion(-)
> >
> > diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
> > index bf31018aef0..b7f08cf36f2 100644
> > --- a/scripts/qapi/parser.py
> > +++ b/scripts/qapi/parser.py
> > @@ -19,6 +19,7 @@
> >  import re
> >  from typing import (
> >      TYPE_CHECKING,
> > +    Any,
> >      Dict,
> >      List,
> >      Mapping,
> > @@ -43,7 +44,7 @@
> >  _ExprValue = Union[List[object], Dict[str, object], str, bool]
> >
> >
> > -class QAPIExpression(Dict[str, object]):
> > +class QAPIExpression(Dict[str, Any]):
> >      # pylint: disable=too-few-public-methods
> >      def __init__(self,
> >                   data: Mapping[str, object],
>
> There are several occurences of Dict[str, object] elsewhere.  Would your
> argument for dumbing down QAPIExpression apply to (some of) them, too?

When and if they piss me off, sure. I'm just wary of making the types
too permissive because it can obscure typing errors; by using Any, you
really disable any further checks and might lead to false confidence
in the static checker. I still have a weird grudge against Any and
would like to fully eliminate it from any statically checked Python
code, but it's just not always feasible and I have to admit that "good
enough" is good enough. Doesn't have me running to lessen the
strictness in areas that didn't cause me pain, though...

> Skimming them, I found this in introspect.py:
>
>     # These types are based on structures defined in QEMU's schema, so we
>     # lack precise types for them here. Python 3.6 does not offer
>     # TypedDict constructs, so they are broadly typed here as simple
>     # Python Dicts.
>     SchemaInfo = Dict[str, object]
>     SchemaInfoEnumMember = Dict[str, object]
>     SchemaInfoObject = Dict[str, object]
>     SchemaInfoObjectVariant = Dict[str, object]
>     SchemaInfoObjectMember = Dict[str, object]
>     SchemaInfoCommand = Dict[str, object]
>
> Can we do better now we have 3.8?

A little bit, but it involves reproducing these types -- which are
ultimately meant to represent QAPI types defined in introspect.json --
with "redundant" type info. i.e. I have to reproduce the existing type
definitions in Python-ese, and then we have the maintenance burden of
making sure they match.

Maybe too much work to come up with a crazy dynamic definition thing
where we take the QAPI definition and build Python types from them ...
without some pretty interesting work to avoid the Ouroboros that'd
result. introspection.py wants static types based on types defined
dynamically by the schema definition; but we are not guaranteed to
have a suitable schema with these types at all. I'm not sure how to
express this kind of dependency without some interesting re-work. This
is a rare circumstance of the QAPI generator relying on the contents
of the Schema to provide static type assistance.

Now, I COULD do it statically, since I don't expect these types to
change much, but I'm wary of how quickly it might get out of hand
trying to achieve better type specificity.

General impression: "Not worth the hassle for this series, but we can
discuss proposals for future improvements"

Re: [PATCH 15/19] qapi/parser: demote QAPIExpression to Dict[str, Any]

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On Thu, Nov 23, 2023 at 9:12 AM Markus Armbruster <armbru@redhat.com> wrote:
>>
>> John Snow <jsnow@redhat.com> writes:
>>
>> > Dict[str, object] is a stricter type, but with the way that code is
>> > currently arranged, it is infeasible to enforce this strictness.
>> >
>> > In particular, although expr.py's entire raison d'être is normalization
>> > and type-checking of QAPI Expressions, that type information is not
>> > "remembered" in any meaningful way by mypy because each individual
>> > expression is not downcast to a specific expression type that holds all
>> > the details of each expression's unique form.
>> >
>> > As a result, all of the code in schema.py that deals with actually
>> > creating type-safe specialized structures has no guarantee (myopically)
>> > that the data it is being passed is correct.
>> >
>> > There are two ways to solve this:
>> >
>> > (1) Re-assert that the incoming data is in the shape we expect it to be, or
>> > (2) Disable type checking for this data.
>> >
>> > (1) is appealing to my sense of strictness, but I gotta concede that it
>> > is asinine to re-check the shape of a QAPIExpression in schema.py when
>> > expr.py has just completed that work at length. The duplication of code
>> > and the nightmare thought of needing to update both locations if and
>> > when we change the shape of these structures makes me extremely
>> > reluctant to go down this route.
>> >
>> > (2) allows us the chance to miss updating types in the case that types
>> > are updated in expr.py, but it *is* an awful lot simpler and,
>> > importantly, gets us closer to type checking schema.py *at
>> > all*. Something is better than nothing, I'd argue.
>> >
>> > So, do the simpler dumber thing and worry about future strictness
>> > improvements later.
>>
>> Yes.
>
> (You were right, again.)

Occasionally happens ;)

>> While Dict[str, object] is stricter than Dict[str, Any], both are miles
>> away from the actual, recursive type.
>>
>> > Signed-off-by: John Snow <jsnow@redhat.com>
>> > ---
>> >  scripts/qapi/parser.py | 3 ++-
>> >  1 file changed, 2 insertions(+), 1 deletion(-)
>> >
>> > diff --git a/scripts/qapi/parser.py b/scripts/qapi/parser.py
>> > index bf31018aef0..b7f08cf36f2 100644
>> > --- a/scripts/qapi/parser.py
>> > +++ b/scripts/qapi/parser.py
>> > @@ -19,6 +19,7 @@
>> >  import re
>> >  from typing import (
>> >      TYPE_CHECKING,
>> > +    Any,
>> >      Dict,
>> >      List,
>> >      Mapping,
>> > @@ -43,7 +44,7 @@
>> >  _ExprValue = Union[List[object], Dict[str, object], str, bool]
>> >
>> >
>> > -class QAPIExpression(Dict[str, object]):
>> > +class QAPIExpression(Dict[str, Any]):
>> >      # pylint: disable=too-few-public-methods
>> >      def __init__(self,
>> >                   data: Mapping[str, object],
>>
>> There are several occurences of Dict[str, object] elsewhere.  Would your
>> argument for dumbing down QAPIExpression apply to (some of) them, too?
>
> When and if they piss me off, sure. I'm just wary of making the types
> too permissive because it can obscure typing errors; by using Any, you
> really disable any further checks and might lead to false confidence
> in the static checker. I still have a weird grudge against Any and
> would like to fully eliminate it from any statically checked Python
> code, but it's just not always feasible and I have to admit that "good
> enough" is good enough. Doesn't have me running to lessen the
> strictness in areas that didn't cause me pain, though...
>
>> Skimming them, I found this in introspect.py:
>>
>>     # These types are based on structures defined in QEMU's schema, so we
>>     # lack precise types for them here. Python 3.6 does not offer
>>     # TypedDict constructs, so they are broadly typed here as simple
>>     # Python Dicts.
>>     SchemaInfo = Dict[str, object]
>>     SchemaInfoEnumMember = Dict[str, object]
>>     SchemaInfoObject = Dict[str, object]
>>     SchemaInfoObjectVariant = Dict[str, object]
>>     SchemaInfoObjectMember = Dict[str, object]
>>     SchemaInfoCommand = Dict[str, object]
>>
>> Can we do better now we have 3.8?
>
> A little bit, but it involves reproducing these types -- which are
> ultimately meant to represent QAPI types defined in introspect.json --
> with "redundant" type info. i.e. I have to reproduce the existing type
> definitions in Python-ese, and then we have the maintenance burden of
> making sure they match.
>
> Maybe too much work to come up with a crazy dynamic definition thing
> where we take the QAPI definition and build Python types from them ...
> without some pretty interesting work to avoid the Ouroboros that'd
> result. introspection.py wants static types based on types defined
> dynamically by the schema definition; but we are not guaranteed to
> have a suitable schema with these types at all. I'm not sure how to
> express this kind of dependency without some interesting re-work. This
> is a rare circumstance of the QAPI generator relying on the contents
> of the Schema to provide static type assistance.
>
> Now, I COULD do it statically, since I don't expect these types to
> change much, but I'm wary of how quickly it might get out of hand
> trying to achieve better type specificity.
>
> General impression: "Not worth the hassle for this series, but we can
> discuss proposals for future improvements"

Fair enough.

Thanks!

[PATCH 16/19] qapi/schema: add type hints

[John Snow]

This patch only adds type hints, which aren't utilized at runtime and
don't change the behavior of this module in any way.

In a scant few locations, type hints are removed where no longer
necessary due to inference power from typing all of the rest of
creation; and any type hints that no longer need string quotes are
changed.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 554 +++++++++++++++++++++++++++++------------
 1 file changed, 389 insertions(+), 165 deletions(-)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index ce5b01b3182..5d19b59def0 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -15,10 +15,20 @@
 # TODO catching name collisions in generated code would be nice
 # pylint: disable=too-many-lines
 
+from __future__ import annotations
+
 from collections import OrderedDict
 import os
 import re
-from typing import List, Optional, cast
+from typing import (
+    Any,
+    Callable,
+    Dict,
+    List,
+    Optional,
+    Union,
+    cast,
+)
 
 from .common import (
     POINTER_SUFFIX,
@@ -30,39 +40,50 @@
 )
 from .error import QAPIError, QAPISemError, QAPISourceError
 from .expr import check_exprs
-from .parser import QAPIExpression, QAPISchemaParser
+from .parser import QAPIDoc, QAPIExpression, QAPISchemaParser
+from .source import QAPISourceInfo
 
 
 class QAPISchemaIfCond:
-    def __init__(self, ifcond=None):
+    def __init__(
+        self,
+        ifcond: Optional[Union[str, Dict[str, object]]] = None,
+    ) -> None:
         self.ifcond = ifcond
 
-    def _cgen(self):
+    def _cgen(self) -> str:
         return cgen_ifcond(self.ifcond)
 
-    def gen_if(self):
+    def gen_if(self) -> str:
         return gen_if(self._cgen())
 
-    def gen_endif(self):
+    def gen_endif(self) -> str:
         return gen_endif(self._cgen())
 
-    def docgen(self):
+    def docgen(self) -> str:
         return docgen_ifcond(self.ifcond)
 
-    def is_present(self):
+    def is_present(self) -> bool:
         return bool(self.ifcond)
 
 
 class QAPISchemaEntity:
-    meta: Optional[str] = None
+    meta: str
 
-    def __init__(self, name: str, info, doc, ifcond=None, features=None):
+    def __init__(
+        self,
+        name: str,
+        info: Optional[QAPISourceInfo],
+        doc: Optional[QAPIDoc],
+        ifcond: Optional[QAPISchemaIfCond] = None,
+        features: Optional[List[QAPISchemaFeature]] = None,
+    ):
         assert name is None or isinstance(name, str)
         for f in features or []:
             assert isinstance(f, QAPISchemaFeature)
             f.set_defined_in(name)
         self.name = name
-        self._module = None
+        self._module: Optional[QAPISchemaModule] = None
         # For explicitly defined entities, info points to the (explicit)
         # definition.  For builtins (and their arrays), info is None.
         # For implicitly defined entities, info points to a place that
@@ -74,102 +95,162 @@ def __init__(self, name: str, info, doc, ifcond=None, features=None):
         self.features = features or []
         self._checked = False
 
-    def __repr__(self):
+    def __repr__(self) -> str:
         if self.name is None:
             return "<%s at 0x%x>" % (type(self).__name__, id(self))
         return "<%s:%s at 0x%x>" % (type(self).__name__, self.name, id(self))
 
-    def c_name(self):
+    def c_name(self) -> str:
         return c_name(self.name)
 
-    def check(self, schema):
+    def check(self, schema: QAPISchema) -> None:
         # pylint: disable=unused-argument
         assert not self._checked
-        seen = {}
+        seen: Dict[str, QAPISchemaMember] = {}
         for f in self.features:
             f.check_clash(self.info, seen)
         self._checked = True
 
-    def connect_doc(self, doc=None):
+    def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
         doc = doc or self.doc
         if doc:
             for f in self.features:
                 doc.connect_feature(f)
 
-    def check_doc(self):
+    def check_doc(self) -> None:
         if self.doc:
             self.doc.check()
 
-    def _set_module(self, schema, info):
+    def _set_module(
+        self, schema: QAPISchema, info: Optional[QAPISourceInfo]
+    ) -> None:
         assert self._checked
         fname = info.fname if info else QAPISchemaModule.BUILTIN_MODULE_NAME
         self._module = schema.module_by_fname(fname)
         self._module.add_entity(self)
 
-    def set_module(self, schema):
+    def set_module(self, schema: QAPISchema) -> None:
         self._set_module(schema, self.info)
 
     @property
-    def ifcond(self):
+    def ifcond(self) -> QAPISchemaIfCond:
         assert self._checked
         return self._ifcond
 
-    def is_implicit(self):
+    def is_implicit(self) -> bool:
         return not self.info
 
-    def visit(self, visitor):
+    def visit(self, visitor: QAPISchemaVisitor) -> None:
         # pylint: disable=unused-argument
         assert self._checked
 
-    def describe(self):
+    def describe(self) -> str:
         assert self.meta
         return "%s '%s'" % (self.meta, self.name)
 
 
 class QAPISchemaVisitor:
-    def visit_begin(self, schema):
+    def visit_begin(self, schema: QAPISchema) -> None:
         pass
 
-    def visit_end(self):
+    def visit_end(self) -> None:
         pass
 
-    def visit_module(self, name):
+    def visit_module(self, name: str) -> None:
         pass
 
-    def visit_needed(self, entity):
+    def visit_needed(self, entity: QAPISchemaEntity) -> bool:
         # pylint: disable=unused-argument
         # Default to visiting everything
         return True
 
-    def visit_include(self, name, info):
+    def visit_include(self, name: str, info: Optional[QAPISourceInfo]) -> None:
         pass
 
-    def visit_builtin_type(self, name, info, json_type):
+    def visit_builtin_type(
+        self, name: str, info: Optional[QAPISourceInfo], json_type: str
+    ) -> None:
         pass
 
-    def visit_enum_type(self, name, info, ifcond, features, members, prefix):
+    def visit_enum_type(
+        self,
+        name: str,
+        info: Optional[QAPISourceInfo],
+        ifcond: QAPISchemaIfCond,
+        features: List[QAPISchemaFeature],
+        members: List[QAPISchemaEnumMember],
+        prefix: Optional[str],
+    ) -> None:
         pass
 
-    def visit_array_type(self, name, info, ifcond, element_type):
+    def visit_array_type(
+        self,
+        name: str,
+        info: Optional[QAPISourceInfo],
+        ifcond: QAPISchemaIfCond,
+        element_type: QAPISchemaType,
+    ) -> None:
         pass
 
-    def visit_object_type(self, name, info, ifcond, features,
-                          base, members, variants):
+    def visit_object_type(
+        self,
+        name: str,
+        info: Optional[QAPISourceInfo],
+        ifcond: QAPISchemaIfCond,
+        features: List[QAPISchemaFeature],
+        base: Optional[QAPISchemaObjectType],
+        members: List[QAPISchemaObjectTypeMember],
+        variants: Optional[QAPISchemaVariants],
+    ) -> None:
         pass
 
-    def visit_object_type_flat(self, name, info, ifcond, features,
-                               members, variants):
+    def visit_object_type_flat(
+        self,
+        name: str,
+        info: Optional[QAPISourceInfo],
+        ifcond: QAPISchemaIfCond,
+        features: List[QAPISchemaFeature],
+        members: List[QAPISchemaObjectTypeMember],
+        variants: Optional[QAPISchemaVariants],
+    ) -> None:
         pass
 
-    def visit_alternate_type(self, name, info, ifcond, features, variants):
+    def visit_alternate_type(
+        self,
+        name: str,
+        info: Optional[QAPISourceInfo],
+        ifcond: QAPISchemaIfCond,
+        features: List[QAPISchemaFeature],
+        variants: QAPISchemaVariants,
+    ) -> None:
         pass
 
-    def visit_command(self, name, info, ifcond, features,
-                      arg_type, ret_type, gen, success_response, boxed,
-                      allow_oob, allow_preconfig, coroutine):
+    def visit_command(
+        self,
+        name: str,
+        info: Optional[QAPISourceInfo],
+        ifcond: QAPISchemaIfCond,
+        features: List[QAPISchemaFeature],
+        arg_type: Optional[QAPISchemaObjectType],
+        ret_type: Optional[QAPISchemaType],
+        gen: bool,
+        success_response: bool,
+        boxed: bool,
+        allow_oob: bool,
+        allow_preconfig: bool,
+        coroutine: bool,
+    ) -> None:
         pass
 
-    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
+    def visit_event(
+        self,
+        name: str,
+        info: Optional[QAPISourceInfo],
+        ifcond: QAPISchemaIfCond,
+        features: List[QAPISchemaFeature],
+        arg_type: Optional[QAPISchemaObjectType],
+        boxed: bool,
+    ) -> None:
         pass
 
 
@@ -177,9 +258,9 @@ class QAPISchemaModule:
 
     BUILTIN_MODULE_NAME = './builtin'
 
-    def __init__(self, name):
+    def __init__(self, name: str):
         self.name = name
-        self._entity_list = []
+        self._entity_list: List[QAPISchemaEntity] = []
 
     @staticmethod
     def is_system_module(name: str) -> bool:
@@ -208,10 +289,10 @@ def is_builtin_module(cls, name: str) -> bool:
         """
         return name == cls.BUILTIN_MODULE_NAME
 
-    def add_entity(self, ent):
+    def add_entity(self, ent: QAPISchemaEntity) -> None:
         self._entity_list.append(ent)
 
-    def visit(self, visitor):
+    def visit(self, visitor: QAPISchemaVisitor) -> None:
         visitor.visit_module(self.name)
         for entity in self._entity_list:
             if visitor.visit_needed(entity):
@@ -219,13 +300,13 @@ def visit(self, visitor):
 
 
 class QAPISchemaInclude(QAPISchemaEntity):
-    def __init__(self, sub_module, info):
+    def __init__(self, sub_module: QAPISchemaModule, info: QAPISourceInfo):
         # Includes are internal entity objects; and may occur multiple times
         name = f"q_include_{info.fname}:{info.line}"
         super().__init__(name, info, None)
         self._sub_module = sub_module
 
-    def visit(self, visitor):
+    def visit(self, visitor: QAPISchemaVisitor) -> None:
         super().visit(visitor)
         visitor.visit_include(self._sub_module.name, self.info)
 
@@ -237,17 +318,17 @@ def c_type(self) -> str:
         raise NotImplementedError()
 
     # Return the C type to be used in a parameter list.
-    def c_param_type(self):
+    def c_param_type(self) -> str:
         return self.c_type()
 
     # Return the C type to be used where we suppress boxing.
-    def c_unboxed_type(self):
+    def c_unboxed_type(self) -> str:
         return self.c_type()
 
     def json_type(self) -> str:
         raise NotImplementedError()
 
-    def alternate_qtype(self):
+    def alternate_qtype(self) -> Optional[str]:
         json2qtype = {
             'null':    'QTYPE_QNULL',
             'string':  'QTYPE_QSTRING',
@@ -259,17 +340,17 @@ def alternate_qtype(self):
         }
         return json2qtype.get(self.json_type())
 
-    def doc_type(self):
+    def doc_type(self) -> Optional[str]:
         if self.is_implicit():
             return None
         return self.name
 
-    def need_has_if_optional(self):
+    def need_has_if_optional(self) -> bool:
         # When FOO is a pointer, has_FOO == !!FOO, i.e. has_FOO is redundant.
         # Except for arrays; see QAPISchemaArrayType.need_has_if_optional().
         return not self.c_type().endswith(POINTER_SUFFIX)
 
-    def check(self, schema):
+    def check(self, schema: QAPISchema) -> None:
         super().check(schema)
         for feat in self.features:
             if feat.is_special():
@@ -277,7 +358,7 @@ def check(self, schema):
                     self.info,
                     f"feature '{feat.name}' is not supported for types")
 
-    def describe(self):
+    def describe(self) -> str:
         assert self.meta
         return "%s type '%s'" % (self.meta, self.name)
 
@@ -285,7 +366,7 @@ def describe(self):
 class QAPISchemaBuiltinType(QAPISchemaType):
     meta = 'built-in'
 
-    def __init__(self, name, json_type, c_type):
+    def __init__(self, name: str, json_type: str, c_type: str):
         super().__init__(name, None, None)
         assert not c_type or isinstance(c_type, str)
         assert json_type in ('string', 'number', 'int', 'boolean', 'null',
@@ -293,24 +374,24 @@ def __init__(self, name, json_type, c_type):
         self._json_type_name = json_type
         self._c_type_name = c_type
 
-    def c_name(self):
+    def c_name(self) -> str:
         return self.name
 
-    def c_type(self):
+    def c_type(self) -> str:
         return self._c_type_name
 
-    def c_param_type(self):
+    def c_param_type(self) -> str:
         if self.name == 'str':
             return 'const ' + self._c_type_name
         return self._c_type_name
 
-    def json_type(self):
+    def json_type(self) -> str:
         return self._json_type_name
 
-    def doc_type(self):
+    def doc_type(self) -> str:
         return self.json_type()
 
-    def visit(self, visitor):
+    def visit(self, visitor: QAPISchemaVisitor) -> None:
         super().visit(visitor)
         visitor.visit_builtin_type(self.name, self.info, self.json_type())
 
@@ -318,7 +399,16 @@ def visit(self, visitor):
 class QAPISchemaEnumType(QAPISchemaType):
     meta = 'enum'
 
-    def __init__(self, name, info, doc, ifcond, features, members, prefix):
+    def __init__(
+        self,
+        name: str,
+        info: Optional[QAPISourceInfo],
+        doc: Optional[QAPIDoc],
+        ifcond: Optional[QAPISchemaIfCond],
+        features: Optional[List[QAPISchemaFeature]],
+        members: List[QAPISchemaEnumMember],
+        prefix: Optional[str],
+    ):
         super().__init__(name, info, doc, ifcond, features)
         for m in members:
             assert isinstance(m, QAPISchemaEnumMember)
@@ -327,32 +417,32 @@ def __init__(self, name, info, doc, ifcond, features, members, prefix):
         self.members = members
         self.prefix = prefix
 
-    def check(self, schema):
+    def check(self, schema: QAPISchema) -> None:
         super().check(schema)
-        seen = {}
+        seen: Dict[str, QAPISchemaMember] = {}
         for m in self.members:
             m.check_clash(self.info, seen)
 
-    def connect_doc(self, doc=None):
+    def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
         super().connect_doc(doc)
         doc = doc or self.doc
         for m in self.members:
             m.connect_doc(doc)
 
-    def is_implicit(self):
+    def is_implicit(self) -> bool:
         # See QAPISchema._def_predefineds()
         return self.name == 'QType'
 
-    def c_type(self):
+    def c_type(self) -> str:
         return c_name(self.name)
 
-    def member_names(self):
+    def member_names(self) -> List[str]:
         return [m.name for m in self.members]
 
-    def json_type(self):
+    def json_type(self) -> str:
         return 'string'
 
-    def visit(self, visitor):
+    def visit(self, visitor: QAPISchemaVisitor) -> None:
         super().visit(visitor)
         visitor.visit_enum_type(
             self.name, self.info, self.ifcond, self.features,
@@ -362,7 +452,9 @@ def visit(self, visitor):
 class QAPISchemaArrayType(QAPISchemaType):
     meta = 'array'
 
-    def __init__(self, name, info, element_type):
+    def __init__(
+        self, name: str, info: Optional[QAPISourceInfo], element_type: str
+    ):
         super().__init__(name, info, None)
         assert isinstance(element_type, str)
         self._element_type_name = element_type
@@ -377,12 +469,12 @@ def element_type(self) -> QAPISchemaType:
             )
         return self._element_type
 
-    def need_has_if_optional(self):
+    def need_has_if_optional(self) -> bool:
         # When FOO is an array, we still need has_FOO to distinguish
         # absent (!has_FOO) from present and empty (has_FOO && !FOO).
         return True
 
-    def check(self, schema):
+    def check(self, schema: QAPISchema) -> None:
         super().check(schema)
 
         if self.info:
@@ -396,42 +488,51 @@ def check(self, schema):
         )
         assert not isinstance(self.element_type, QAPISchemaArrayType)
 
-    def set_module(self, schema):
+    def set_module(self, schema: QAPISchema) -> None:
         self._set_module(schema, self.element_type.info)
 
     @property
-    def ifcond(self):
+    def ifcond(self) -> QAPISchemaIfCond:
         assert self._checked
         return self.element_type.ifcond
 
-    def is_implicit(self):
+    def is_implicit(self) -> bool:
         return True
 
-    def c_type(self):
+    def c_type(self) -> str:
         return c_name(self.name) + POINTER_SUFFIX
 
-    def json_type(self):
+    def json_type(self) -> str:
         return 'array'
 
-    def doc_type(self):
+    def doc_type(self) -> Optional[str]:
         elt_doc_type = self.element_type.doc_type()
         if not elt_doc_type:
             return None
         return 'array of ' + elt_doc_type
 
-    def visit(self, visitor):
+    def visit(self, visitor: QAPISchemaVisitor) -> None:
         super().visit(visitor)
         visitor.visit_array_type(self.name, self.info, self.ifcond,
                                  self.element_type)
 
-    def describe(self):
+    def describe(self) -> str:
         assert self.meta
         return "%s type ['%s']" % (self.meta, self._element_type_name)
 
 
 class QAPISchemaObjectType(QAPISchemaType):
-    def __init__(self, name, info, doc, ifcond, features,
-                 base, local_members, variants):
+    def __init__(
+        self,
+        name: str,
+        info: Optional[QAPISourceInfo],
+        doc: Optional[QAPIDoc],
+        ifcond: Optional[QAPISchemaIfCond],
+        features: Optional[List[QAPISchemaFeature]],
+        base: Optional[str],
+        local_members: List[QAPISchemaObjectTypeMember],
+        variants: Optional[QAPISchemaVariants],
+    ):
         # struct has local_members, optional base, and no variants
         # union has base, variants, and no local_members
         super().__init__(name, info, doc, ifcond, features)
@@ -450,7 +551,7 @@ def __init__(self, name, info, doc, ifcond, features,
         self.members: List[QAPISchemaObjectTypeMember] = []
         self._checking = False
 
-    def check(self, schema):
+    def check(self, schema: QAPISchema) -> None:
         # This calls another type T's .check() exactly when the C
         # struct emitted by gen_object() contains that T's C struct
         # (pointers don't count).
@@ -496,14 +597,18 @@ def check(self, schema):
     # Check that the members of this type do not cause duplicate JSON members,
     # and update seen to track the members seen so far. Report any errors
     # on behalf of info, which is not necessarily self.info
-    def check_clash(self, info, seen):
+    def check_clash(
+        self,
+        info: Optional[QAPISourceInfo],
+        seen: Dict[str, QAPISchemaMember],
+    ) -> None:
         assert self._checked
         for m in self.members:
             m.check_clash(info, seen)
         if self.variants:
             self.variants.check_clash(info, seen)
 
-    def connect_doc(self, doc=None):
+    def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
         super().connect_doc(doc)
         doc = doc or self.doc
         if self.base and self.base.is_implicit():
@@ -511,34 +616,34 @@ def connect_doc(self, doc=None):
         for m in self.local_members:
             m.connect_doc(doc)
 
-    def is_implicit(self):
+    def is_implicit(self) -> bool:
         # See QAPISchema._make_implicit_object_type(), as well as
         # _def_predefineds()
         return self.name.startswith('q_')
 
-    def is_empty(self):
+    def is_empty(self) -> bool:
         assert self.members is not None
         return not self.members and not self.variants
 
-    def has_conditional_members(self):
+    def has_conditional_members(self) -> bool:
         assert self.members is not None
         return any(m.ifcond.is_present() for m in self.members)
 
-    def c_name(self):
+    def c_name(self) -> str:
         assert self.name != 'q_empty'
         return super().c_name()
 
-    def c_type(self):
+    def c_type(self) -> str:
         assert not self.is_implicit()
         return c_name(self.name) + POINTER_SUFFIX
 
-    def c_unboxed_type(self):
+    def c_unboxed_type(self) -> str:
         return c_name(self.name)
 
-    def json_type(self):
+    def json_type(self) -> str:
         return 'object'
 
-    def visit(self, visitor):
+    def visit(self, visitor: QAPISchemaVisitor) -> None:
         super().visit(visitor)
         visitor.visit_object_type(
             self.name, self.info, self.ifcond, self.features,
@@ -551,7 +656,15 @@ def visit(self, visitor):
 class QAPISchemaAlternateType(QAPISchemaType):
     meta = 'alternate'
 
-    def __init__(self, name, info, doc, ifcond, features, variants):
+    def __init__(
+        self,
+        name: str,
+        info: QAPISourceInfo,
+        doc: Optional[QAPIDoc],
+        ifcond: Optional[QAPISchemaIfCond],
+        features: List[QAPISchemaFeature],
+        variants: QAPISchemaVariants,
+    ):
         super().__init__(name, info, doc, ifcond, features)
         assert isinstance(variants, QAPISchemaVariants)
         assert variants.tag_member
@@ -559,7 +672,7 @@ def __init__(self, name, info, doc, ifcond, features, variants):
         variants.tag_member.set_defined_in(self.name)
         self.variants = variants
 
-    def check(self, schema):
+    def check(self, schema: QAPISchema) -> None:
         super().check(schema)
         self.variants.tag_member.check(schema)
         # Not calling self.variants.check_clash(), because there's nothing
@@ -567,8 +680,8 @@ def check(self, schema):
         self.variants.check(schema, {})
         # Alternate branch names have no relation to the tag enum values;
         # so we have to check for potential name collisions ourselves.
-        seen = {}
-        types_seen = {}
+        seen: Dict[str, QAPISchemaMember] = {}
+        types_seen: Dict[str, str] = {}
         for v in self.variants.variants:
             v.check_clash(self.info, seen)
             qtype = v.type.alternate_qtype()
@@ -597,26 +710,32 @@ def check(self, schema):
                         % (v.describe(self.info), types_seen[qt]))
                 types_seen[qt] = v.name
 
-    def connect_doc(self, doc=None):
+    def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
         super().connect_doc(doc)
         doc = doc or self.doc
         for v in self.variants.variants:
             v.connect_doc(doc)
 
-    def c_type(self):
+    def c_type(self) -> str:
         return c_name(self.name) + POINTER_SUFFIX
 
-    def json_type(self):
+    def json_type(self) -> str:
         return 'value'
 
-    def visit(self, visitor):
+    def visit(self, visitor: QAPISchemaVisitor) -> None:
         super().visit(visitor)
         visitor.visit_alternate_type(
             self.name, self.info, self.ifcond, self.features, self.variants)
 
 
 class QAPISchemaVariants:
-    def __init__(self, tag_name, info, tag_member, variants):
+    def __init__(
+        self,
+        tag_name: Optional[str],
+        info: QAPISourceInfo,
+        tag_member: Optional[QAPISchemaObjectTypeMember],
+        variants: List[QAPISchemaVariant],
+    ):
         # Unions pass tag_name but not tag_member.
         # Alternates pass tag_member but not tag_name.
         # After check(), tag_member is always set.
@@ -627,11 +746,11 @@ def __init__(self, tag_name, info, tag_member, variants):
             assert isinstance(v, QAPISchemaVariant)
         self._tag_name = tag_name
         self.info = info
-        self._tag_member: Optional[QAPISchemaObjectTypeMember] = tag_member
+        self._tag_member = tag_member
         self.variants = variants
 
     @property
-    def tag_member(self) -> 'QAPISchemaObjectTypeMember':
+    def tag_member(self) -> QAPISchemaObjectTypeMember:
         if self._tag_member is None:
             raise RuntimeError(
                 "QAPISchemaVariants has no tag_member property until "
@@ -639,11 +758,13 @@ def tag_member(self) -> 'QAPISchemaObjectTypeMember':
             )
         return self._tag_member
 
-    def set_defined_in(self, name):
+    def set_defined_in(self, name: str) -> None:
         for v in self.variants:
             v.set_defined_in(name)
 
-    def check(self, schema, seen):
+    def check(
+        self, schema: QAPISchema, seen: Dict[str, QAPISchemaMember]
+    ) -> None:
         if self._tag_name:      # union
             # We need to narrow the member type:
             tmp = seen.get(c_name(self._tag_name))
@@ -713,7 +834,11 @@ def check(self, schema, seen):
                         % (v.describe(self.info), v.type.describe()))
                 v.type.check(schema)
 
-    def check_clash(self, info, seen):
+    def check_clash(
+        self,
+        info: Optional[QAPISourceInfo],
+        seen: Dict[str, QAPISchemaMember],
+    ) -> None:
         for v in self.variants:
             # Reset seen map for each variant, since qapi names from one
             # branch do not affect another branch
@@ -725,18 +850,27 @@ class QAPISchemaMember:
     """ Represents object members, enum members and features """
     role = 'member'
 
-    def __init__(self, name, info, ifcond=None):
+    def __init__(
+        self,
+        name: str,
+        info: Optional[QAPISourceInfo],
+        ifcond: Optional[QAPISchemaIfCond] = None,
+    ):
         assert isinstance(name, str)
         self.name = name
         self.info = info
         self.ifcond = ifcond or QAPISchemaIfCond()
-        self.defined_in = None
+        self.defined_in: Optional[str] = None
 
-    def set_defined_in(self, name):
+    def set_defined_in(self, name: str) -> None:
         assert not self.defined_in
         self.defined_in = name
 
-    def check_clash(self, info, seen):
+    def check_clash(
+        self,
+        info: Optional[QAPISourceInfo],
+        seen: Dict[str, QAPISchemaMember],
+    ) -> None:
         cname = c_name(self.name)
         if cname in seen:
             raise QAPISemError(
@@ -745,11 +879,11 @@ def check_clash(self, info, seen):
                 % (self.describe(info), seen[cname].describe(info)))
         seen[cname] = self
 
-    def connect_doc(self, doc):
+    def connect_doc(self, doc: Optional[QAPIDoc]) -> None:
         if doc:
             doc.connect_member(self)
 
-    def describe(self, info):
+    def describe(self, info: Optional[QAPISourceInfo]) -> str:
         role = self.role
         meta = 'type'
         defined_in = self.defined_in
@@ -781,14 +915,20 @@ def describe(self, info):
 class QAPISchemaEnumMember(QAPISchemaMember):
     role = 'value'
 
-    def __init__(self, name, info, ifcond=None, features=None):
+    def __init__(
+        self,
+        name: str,
+        info: Optional[QAPISourceInfo],
+        ifcond: Optional[QAPISchemaIfCond] = None,
+        features: Optional[List[QAPISchemaFeature]] = None,
+    ):
         super().__init__(name, info, ifcond)
         for f in features or []:
             assert isinstance(f, QAPISchemaFeature)
             f.set_defined_in(name)
         self.features = features or []
 
-    def connect_doc(self, doc):
+    def connect_doc(self, doc: Optional[QAPIDoc]) -> None:
         super().connect_doc(doc)
         if doc:
             for f in self.features:
@@ -798,12 +938,20 @@ def connect_doc(self, doc):
 class QAPISchemaFeature(QAPISchemaMember):
     role = 'feature'
 
-    def is_special(self):
+    def is_special(self) -> bool:
         return self.name in ('deprecated', 'unstable')
 
 
 class QAPISchemaObjectTypeMember(QAPISchemaMember):
-    def __init__(self, name, info, typ, optional, ifcond=None, features=None):
+    def __init__(
+        self,
+        name: str,
+        info: QAPISourceInfo,
+        typ: str,
+        optional: bool,
+        ifcond: Optional[QAPISchemaIfCond] = None,
+        features: Optional[List[QAPISchemaFeature]] = None,
+    ):
         super().__init__(name, info, ifcond)
         assert isinstance(typ, str)
         assert isinstance(optional, bool)
@@ -815,19 +963,19 @@ def __init__(self, name, info, typ, optional, ifcond=None, features=None):
         self.optional = optional
         self.features = features or []
 
-    def need_has(self):
+    def need_has(self) -> bool:
         assert self.type
         return self.optional and self.type.need_has_if_optional()
 
-    def check(self, schema):
+    def check(self, schema: QAPISchema) -> None:
         assert self.defined_in
         self.type = schema.resolve_type(self._type_name, self.info,
                                         self.describe)
-        seen = {}
+        seen: Dict[str, QAPISchemaMember] = {}
         for f in self.features:
             f.check_clash(self.info, seen)
 
-    def connect_doc(self, doc):
+    def connect_doc(self, doc: Optional[QAPIDoc]) -> None:
         super().connect_doc(doc)
         if doc:
             for f in self.features:
@@ -837,24 +985,42 @@ def connect_doc(self, doc):
 class QAPISchemaVariant(QAPISchemaObjectTypeMember):
     role = 'branch'
 
-    def __init__(self, name, info, typ, ifcond=None):
+    def __init__(
+        self,
+        name: str,
+        info: QAPISourceInfo,
+        typ: str,
+        ifcond: QAPISchemaIfCond,
+    ):
         super().__init__(name, info, typ, False, ifcond)
 
 
 class QAPISchemaCommand(QAPISchemaEntity):
     meta = 'command'
 
-    def __init__(self, name, info, doc, ifcond, features,
-                 arg_type, ret_type,
-                 gen, success_response, boxed, allow_oob, allow_preconfig,
-                 coroutine):
+    def __init__(
+        self,
+        name: str,
+        info: QAPISourceInfo,
+        doc: Optional[QAPIDoc],
+        ifcond: QAPISchemaIfCond,
+        features: List[QAPISchemaFeature],
+        arg_type: Optional[str],
+        ret_type: Optional[str],
+        gen: bool,
+        success_response: bool,
+        boxed: bool,
+        allow_oob: bool,
+        allow_preconfig: bool,
+        coroutine: bool,
+    ):
         super().__init__(name, info, doc, ifcond, features)
         assert not arg_type or isinstance(arg_type, str)
         assert not ret_type or isinstance(ret_type, str)
         self._arg_type_name = arg_type
-        self.arg_type = None
+        self.arg_type: Optional[QAPISchemaObjectType] = None
         self._ret_type_name = ret_type
-        self.ret_type = None
+        self.ret_type: Optional[QAPISchemaType] = None
         self.gen = gen
         self.success_response = success_response
         self.boxed = boxed
@@ -862,7 +1028,7 @@ def __init__(self, name, info, doc, ifcond, features,
         self.allow_preconfig = allow_preconfig
         self.coroutine = coroutine
 
-    def check(self, schema):
+    def check(self, schema: QAPISchema) -> None:
         assert self.info is not None
         super().check(schema)
         if self._arg_type_name:
@@ -897,14 +1063,14 @@ def check(self, schema):
                         "command's 'returns' cannot take %s"
                         % self.ret_type.describe())
 
-    def connect_doc(self, doc=None):
+    def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
         super().connect_doc(doc)
         doc = doc or self.doc
         if doc:
             if self.arg_type and self.arg_type.is_implicit():
                 self.arg_type.connect_doc(doc)
 
-    def visit(self, visitor):
+    def visit(self, visitor: QAPISchemaVisitor) -> None:
         super().visit(visitor)
         visitor.visit_command(
             self.name, self.info, self.ifcond, self.features,
@@ -916,14 +1082,23 @@ def visit(self, visitor):
 class QAPISchemaEvent(QAPISchemaEntity):
     meta = 'event'
 
-    def __init__(self, name, info, doc, ifcond, features, arg_type, boxed):
+    def __init__(
+        self,
+        name: str,
+        info: QAPISourceInfo,
+        doc: Optional[QAPIDoc],
+        ifcond: QAPISchemaIfCond,
+        features: List[QAPISchemaFeature],
+        arg_type: Optional[str],
+        boxed: bool,
+    ):
         super().__init__(name, info, doc, ifcond, features)
         assert not arg_type or isinstance(arg_type, str)
         self._arg_type_name = arg_type
-        self.arg_type = None
+        self.arg_type: Optional[QAPISchemaObjectType] = None
         self.boxed = boxed
 
-    def check(self, schema):
+    def check(self, schema: QAPISchema) -> None:
         super().check(schema)
         if self._arg_type_name:
             typ = schema.resolve_type(
@@ -945,14 +1120,14 @@ def check(self, schema):
                     self.info,
                     "conditional event arguments require 'boxed': true")
 
-    def connect_doc(self, doc=None):
+    def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
         super().connect_doc(doc)
         doc = doc or self.doc
         if doc:
             if self.arg_type and self.arg_type.is_implicit():
                 self.arg_type.connect_doc(doc)
 
-    def visit(self, visitor):
+    def visit(self, visitor: QAPISchemaVisitor) -> None:
         super().visit(visitor)
         visitor.visit_event(
             self.name, self.info, self.ifcond, self.features,
@@ -960,7 +1135,7 @@ def visit(self, visitor):
 
 
 class QAPISchema:
-    def __init__(self, fname):
+    def __init__(self, fname: str):
         self.fname = fname
 
         try:
@@ -972,9 +1147,9 @@ def __init__(self, fname):
 
         exprs = check_exprs(parser.exprs)
         self.docs = parser.docs
-        self._entity_list = []
-        self._entity_dict = {}
-        self._module_dict = OrderedDict()
+        self._entity_list: List[QAPISchemaEntity] = []
+        self._entity_dict: Dict[str, QAPISchemaEntity] = {}
+        self._module_dict: Dict[str, QAPISchemaModule] = OrderedDict()
         self._schema_dir = os.path.dirname(fname)
         self._make_module(QAPISchemaModule.BUILTIN_MODULE_NAME)
         self._make_module(fname)
@@ -984,7 +1159,7 @@ def __init__(self, fname):
         self._def_exprs(exprs)
         self.check()
 
-    def _def_entity(self, ent):
+    def _def_entity(self, ent: QAPISchemaEntity) -> None:
         # Only the predefined types are allowed to not have info
         assert ent.info or self._predefining
         self._entity_list.append(ent)
@@ -1003,7 +1178,11 @@ def _def_entity(self, ent):
                 ent.info, "%s is already defined" % other_ent.describe())
         self._entity_dict[ent.name] = ent
 
-    def lookup_entity(self, name, typ=None):
+    def lookup_entity(
+        self,
+        name: str,
+        typ: Optional[type] = None,
+    ) -> Optional[QAPISchemaEntity]:
         ent = self._entity_dict.get(name)
         if typ and not isinstance(ent, typ):
             return None
@@ -1016,7 +1195,12 @@ def lookup_type(self, name: str) -> Optional[QAPISchemaType]:
         assert isinstance(typ, QAPISchemaType)
         return typ
 
-    def resolve_type(self, name, info, what):
+    def resolve_type(
+        self,
+        name: str,
+        info: Optional[QAPISourceInfo],
+        what: Union[str, Callable[[Optional[QAPISourceInfo]], str]],
+    ) -> QAPISchemaType:
         typ = self.lookup_type(name)
         if not typ:
             if callable(what):
@@ -1030,23 +1214,25 @@ def _module_name(self, fname: str) -> str:
             return fname
         return os.path.relpath(fname, self._schema_dir)
 
-    def _make_module(self, fname):
+    def _make_module(self, fname: str) -> QAPISchemaModule:
         name = self._module_name(fname)
         if name not in self._module_dict:
             self._module_dict[name] = QAPISchemaModule(name)
         return self._module_dict[name]
 
-    def module_by_fname(self, fname):
+    def module_by_fname(self, fname: str) -> QAPISchemaModule:
         name = self._module_name(fname)
         return self._module_dict[name]
 
-    def _def_include(self, expr: QAPIExpression):
+    def _def_include(self, expr: QAPIExpression) -> None:
         include = expr['include']
         assert expr.doc is None
         self._def_entity(
             QAPISchemaInclude(self._make_module(include), expr.info))
 
-    def _def_builtin_type(self, name, json_type, c_type):
+    def _def_builtin_type(
+        self, name: str, json_type: str, c_type: str
+    ) -> None:
         self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type))
         # Instantiating only the arrays that are actually used would
         # be nice, but we can't as long as their generated code
@@ -1054,7 +1240,7 @@ def _def_builtin_type(self, name, json_type, c_type):
         # schema.
         self._make_array_type(name, None)
 
-    def _def_predefineds(self):
+    def _def_predefineds(self) -> None:
         for t in [('str',    'string',  'char' + POINTER_SUFFIX),
                   ('number', 'number',  'double'),
                   ('int',    'int',     'int64_t'),
@@ -1083,30 +1269,51 @@ def _def_predefineds(self):
         self._def_entity(QAPISchemaEnumType('QType', None, None, None, None,
                                             qtype_values, 'QTYPE'))
 
-    def _make_features(self, features, info):
+    def _make_features(
+        self,
+        features: Optional[List[Dict[str, Any]]],
+        info: Optional[QAPISourceInfo],
+    ) -> List[QAPISchemaFeature]:
         if features is None:
             return []
         return [QAPISchemaFeature(f['name'], info,
                                   QAPISchemaIfCond(f.get('if')))
                 for f in features]
 
-    def _make_enum_member(self, name, ifcond, features, info):
+    def _make_enum_member(
+        self,
+        name: str,
+        ifcond: Optional[Union[str, Dict[str, Any]]],
+        features: Optional[List[Dict[str, Any]]],
+        info: Optional[QAPISourceInfo],
+    ) -> QAPISchemaEnumMember:
         return QAPISchemaEnumMember(name, info,
                                     QAPISchemaIfCond(ifcond),
                                     self._make_features(features, info))
 
-    def _make_enum_members(self, values, info):
+    def _make_enum_members(
+        self, values: List[Dict[str, Any]], info: Optional[QAPISourceInfo]
+    ) -> List[QAPISchemaEnumMember]:
         return [self._make_enum_member(v['name'], v.get('if'),
                                        v.get('features'), info)
                 for v in values]
 
-    def _make_array_type(self, element_type, info):
+    def _make_array_type(
+        self, element_type: str, info: Optional[QAPISourceInfo]
+    ) -> str:
         name = element_type + 'List'    # reserved by check_defn_name_str()
         if not self.lookup_type(name):
             self._def_entity(QAPISchemaArrayType(name, info, element_type))
         return name
 
-    def _make_implicit_object_type(self, name, info, ifcond, role, members):
+    def _make_implicit_object_type(
+        self,
+        name: str,
+        info: QAPISourceInfo,
+        ifcond: QAPISchemaIfCond,
+        role: str,
+        members: List[QAPISchemaObjectTypeMember],
+    ) -> Optional[str]:
         if not members:
             return None
         # See also QAPISchemaObjectTypeMember.describe()
@@ -1122,7 +1329,7 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members):
                 name, info, None, ifcond, None, None, members, None))
         return name
 
-    def _def_enum_type(self, expr: QAPIExpression):
+    def _def_enum_type(self, expr: QAPIExpression) -> None:
         name = expr['enum']
         data = expr['data']
         prefix = expr.get('prefix')
@@ -1133,7 +1340,14 @@ def _def_enum_type(self, expr: QAPIExpression):
             name, info, expr.doc, ifcond, features,
             self._make_enum_members(data, info), prefix))
 
-    def _make_member(self, name, typ, ifcond, features, info):
+    def _make_member(
+        self,
+        name: str,
+        typ: Union[List[str], str],
+        ifcond: QAPISchemaIfCond,
+        features: Optional[List[Dict[str, Any]]],
+        info: QAPISourceInfo,
+    ) -> QAPISchemaObjectTypeMember:
         optional = False
         if name.startswith('*'):
             name = name[1:]
@@ -1144,13 +1358,17 @@ def _make_member(self, name, typ, ifcond, features, info):
         return QAPISchemaObjectTypeMember(name, info, typ, optional, ifcond,
                                           self._make_features(features, info))
 
-    def _make_members(self, data, info):
+    def _make_members(
+        self,
+        data: Dict[str, Any],
+        info: QAPISourceInfo,
+    ) -> List[QAPISchemaObjectTypeMember]:
         return [self._make_member(key, value['type'],
                                   QAPISchemaIfCond(value.get('if')),
                                   value.get('features'), info)
                 for (key, value) in data.items()]
 
-    def _def_struct_type(self, expr: QAPIExpression):
+    def _def_struct_type(self, expr: QAPIExpression) -> None:
         name = expr['struct']
         base = expr.get('base')
         data = expr['data']
@@ -1162,13 +1380,19 @@ def _def_struct_type(self, expr: QAPIExpression):
             self._make_members(data, info),
             None))
 
-    def _make_variant(self, case, typ, ifcond, info):
+    def _make_variant(
+        self,
+        case: str,
+        typ: str,
+        ifcond: QAPISchemaIfCond,
+        info: QAPISourceInfo,
+    ) -> QAPISchemaVariant:
         if isinstance(typ, list):
             assert len(typ) == 1
             typ = self._make_array_type(typ[0], info)
         return QAPISchemaVariant(case, info, typ, ifcond)
 
-    def _def_union_type(self, expr: QAPIExpression):
+    def _def_union_type(self, expr: QAPIExpression) -> None:
         name = expr['union']
         base = expr['base']
         tag_name = expr['discriminator']
@@ -1193,7 +1417,7 @@ def _def_union_type(self, expr: QAPIExpression):
                                  QAPISchemaVariants(
                                      tag_name, info, None, variants)))
 
-    def _def_alternate_type(self, expr: QAPIExpression):
+    def _def_alternate_type(self, expr: QAPIExpression) -> None:
         name = expr['alternate']
         data = expr['data']
         assert isinstance(data, dict)
@@ -1211,7 +1435,7 @@ def _def_alternate_type(self, expr: QAPIExpression):
                 name, info, expr.doc, ifcond, features,
                 QAPISchemaVariants(None, info, tag_member, variants)))
 
-    def _def_command(self, expr: QAPIExpression):
+    def _def_command(self, expr: QAPIExpression) -> None:
         name = expr['command']
         data = expr.get('data')
         rets = expr.get('returns')
@@ -1237,7 +1461,7 @@ def _def_command(self, expr: QAPIExpression):
                                            boxed, allow_oob, allow_preconfig,
                                            coroutine))
 
-    def _def_event(self, expr: QAPIExpression):
+    def _def_event(self, expr: QAPIExpression) -> None:
         name = expr['event']
         data = expr.get('data')
         boxed = expr.get('boxed', False)
@@ -1251,7 +1475,7 @@ def _def_event(self, expr: QAPIExpression):
         self._def_entity(QAPISchemaEvent(name, info, expr.doc, ifcond,
                                          features, data, boxed))
 
-    def _def_exprs(self, exprs):
+    def _def_exprs(self, exprs: List[QAPIExpression]) -> None:
         for expr in exprs:
             if 'enum' in expr:
                 self._def_enum_type(expr)
@@ -1270,7 +1494,7 @@ def _def_exprs(self, exprs):
             else:
                 assert False
 
-    def check(self):
+    def check(self) -> None:
         for ent in self._entity_list:
             ent.check(self)
             ent.connect_doc()
@@ -1278,7 +1502,7 @@ def check(self):
         for ent in self._entity_list:
             ent.set_module(self)
 
-    def visit(self, visitor):
+    def visit(self, visitor: QAPISchemaVisitor) -> None:
         visitor.visit_begin(self)
         for mod in self._module_dict.values():
             mod.visit(visitor)
-- 
2.41.0

Re: [PATCH 16/19] qapi/schema: add type hints

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> This patch only adds type hints, which aren't utilized at runtime and
> don't change the behavior of this module in any way.
>
> In a scant few locations, type hints are removed where no longer
> necessary due to inference power from typing all of the rest of
> creation; and any type hints that no longer need string quotes are
> changed.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/schema.py | 554 +++++++++++++++++++++++++++++------------
>  1 file changed, 389 insertions(+), 165 deletions(-)
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index ce5b01b3182..5d19b59def0 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -15,10 +15,20 @@
>  # TODO catching name collisions in generated code would be nice
>  # pylint: disable=too-many-lines
>  
> +from __future__ import annotations
> +
>  from collections import OrderedDict
>  import os
>  import re
> -from typing import List, Optional, cast
> +from typing import (
> +    Any,
> +    Callable,
> +    Dict,
> +    List,
> +    Optional,
> +    Union,
> +    cast,
> +)
>  
>  from .common import (
>      POINTER_SUFFIX,
> @@ -30,39 +40,50 @@
>  )
>  from .error import QAPIError, QAPISemError, QAPISourceError
>  from .expr import check_exprs
> -from .parser import QAPIExpression, QAPISchemaParser
> +from .parser import QAPIDoc, QAPIExpression, QAPISchemaParser
> +from .source import QAPISourceInfo
>  
>  
>  class QAPISchemaIfCond:
> -    def __init__(self, ifcond=None):
> +    def __init__(
> +        self,
> +        ifcond: Optional[Union[str, Dict[str, object]]] = None,

Once again, we approximate a recursive type with Dict[str, object].

> +    ) -> None:
>          self.ifcond = ifcond
>  
> -    def _cgen(self):
> +    def _cgen(self) -> str:
>          return cgen_ifcond(self.ifcond)
>  
> -    def gen_if(self):
> +    def gen_if(self) -> str:
>          return gen_if(self._cgen())
>  
> -    def gen_endif(self):
> +    def gen_endif(self) -> str:
>          return gen_endif(self._cgen())
>  
> -    def docgen(self):
> +    def docgen(self) -> str:
>          return docgen_ifcond(self.ifcond)
>  
> -    def is_present(self):
> +    def is_present(self) -> bool:
>          return bool(self.ifcond)
>  
>  
>  class QAPISchemaEntity:
> -    meta: Optional[str] = None
> +    meta: str

This is more than just a type hint, you also drop the initial value.
Which is fine; QAPISchemaEntity is abstract, and the concrete subtypes
all initialize it.

Separate patch so this patch's commit message doesn't lie.

>  
> -    def __init__(self, name: str, info, doc, ifcond=None, features=None):
> +    def __init__(
> +        self,
> +        name: str,
> +        info: Optional[QAPISourceInfo],
> +        doc: Optional[QAPIDoc],
> +        ifcond: Optional[QAPISchemaIfCond] = None,
> +        features: Optional[List[QAPISchemaFeature]] = None,
> +    ):
>          assert name is None or isinstance(name, str)
>          for f in features or []:
>              assert isinstance(f, QAPISchemaFeature)
>              f.set_defined_in(name)
>          self.name = name
> -        self._module = None
> +        self._module: Optional[QAPISchemaModule] = None
>          # For explicitly defined entities, info points to the (explicit)
>          # definition.  For builtins (and their arrays), info is None.
>          # For implicitly defined entities, info points to a place that
> @@ -74,102 +95,162 @@ def __init__(self, name: str, info, doc, ifcond=None, features=None):
>          self.features = features or []
>          self._checked = False
>  
> -    def __repr__(self):
> +    def __repr__(self) -> str:
>          if self.name is None:
>              return "<%s at 0x%x>" % (type(self).__name__, id(self))
>          return "<%s:%s at 0x%x>" % (type(self).__name__, self.name, id(self))
>  
> -    def c_name(self):
> +    def c_name(self) -> str:
>          return c_name(self.name)
>  
> -    def check(self, schema):
> +    def check(self, schema: QAPISchema) -> None:
>          # pylint: disable=unused-argument
>          assert not self._checked
> -        seen = {}
> +        seen: Dict[str, QAPISchemaMember] = {}
>          for f in self.features:
>              f.check_clash(self.info, seen)
>          self._checked = True
>  
> -    def connect_doc(self, doc=None):
> +    def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
>          doc = doc or self.doc
>          if doc:
>              for f in self.features:
>                  doc.connect_feature(f)
>  
> -    def check_doc(self):
> +    def check_doc(self) -> None:
>          if self.doc:
>              self.doc.check()
>  
> -    def _set_module(self, schema, info):
> +    def _set_module(
> +        self, schema: QAPISchema, info: Optional[QAPISourceInfo]
> +    ) -> None:
>          assert self._checked
>          fname = info.fname if info else QAPISchemaModule.BUILTIN_MODULE_NAME
>          self._module = schema.module_by_fname(fname)
>          self._module.add_entity(self)
>  
> -    def set_module(self, schema):
> +    def set_module(self, schema: QAPISchema) -> None:
>          self._set_module(schema, self.info)
>  
>      @property
> -    def ifcond(self):
> +    def ifcond(self) -> QAPISchemaIfCond:
>          assert self._checked
>          return self._ifcond
>  
> -    def is_implicit(self):
> +    def is_implicit(self) -> bool:
>          return not self.info
>  
> -    def visit(self, visitor):
> +    def visit(self, visitor: QAPISchemaVisitor) -> None:
>          # pylint: disable=unused-argument
>          assert self._checked
>  
> -    def describe(self):
> +    def describe(self) -> str:
>          assert self.meta
>          return "%s '%s'" % (self.meta, self.name)
>  
>  
>  class QAPISchemaVisitor:
> -    def visit_begin(self, schema):
> +    def visit_begin(self, schema: QAPISchema) -> None:
>          pass
>  
> -    def visit_end(self):
> +    def visit_end(self) -> None:
>          pass
>  
> -    def visit_module(self, name):
> +    def visit_module(self, name: str) -> None:
>          pass
>  
> -    def visit_needed(self, entity):
> +    def visit_needed(self, entity: QAPISchemaEntity) -> bool:
>          # pylint: disable=unused-argument
>          # Default to visiting everything
>          return True
>  
> -    def visit_include(self, name, info):
> +    def visit_include(self, name: str, info: Optional[QAPISourceInfo]) -> None:

Long line.

>          pass
>  
> -    def visit_builtin_type(self, name, info, json_type):
> +    def visit_builtin_type(
> +        self, name: str, info: Optional[QAPISourceInfo], json_type: str
> +    ) -> None:
>          pass
>  
> -    def visit_enum_type(self, name, info, ifcond, features, members, prefix):
> +    def visit_enum_type(
> +        self,
> +        name: str,
> +        info: Optional[QAPISourceInfo],
> +        ifcond: QAPISchemaIfCond,
> +        features: List[QAPISchemaFeature],
> +        members: List[QAPISchemaEnumMember],
> +        prefix: Optional[str],
> +    ) -> None:
>          pass
>  
> -    def visit_array_type(self, name, info, ifcond, element_type):
> +    def visit_array_type(
> +        self,
> +        name: str,
> +        info: Optional[QAPISourceInfo],
> +        ifcond: QAPISchemaIfCond,
> +        element_type: QAPISchemaType,
> +    ) -> None:
>          pass
>  
> -    def visit_object_type(self, name, info, ifcond, features,
> -                          base, members, variants):
> +    def visit_object_type(
> +        self,
> +        name: str,
> +        info: Optional[QAPISourceInfo],
> +        ifcond: QAPISchemaIfCond,
> +        features: List[QAPISchemaFeature],
> +        base: Optional[QAPISchemaObjectType],
> +        members: List[QAPISchemaObjectTypeMember],
> +        variants: Optional[QAPISchemaVariants],
> +    ) -> None:
>          pass
>  
> -    def visit_object_type_flat(self, name, info, ifcond, features,
> -                               members, variants):
> +    def visit_object_type_flat(
> +        self,
> +        name: str,
> +        info: Optional[QAPISourceInfo],
> +        ifcond: QAPISchemaIfCond,
> +        features: List[QAPISchemaFeature],
> +        members: List[QAPISchemaObjectTypeMember],
> +        variants: Optional[QAPISchemaVariants],
> +    ) -> None:
>          pass
>  
> -    def visit_alternate_type(self, name, info, ifcond, features, variants):
> +    def visit_alternate_type(
> +        self,
> +        name: str,
> +        info: Optional[QAPISourceInfo],
> +        ifcond: QAPISchemaIfCond,
> +        features: List[QAPISchemaFeature],
> +        variants: QAPISchemaVariants,
> +    ) -> None:
>          pass
>  
> -    def visit_command(self, name, info, ifcond, features,
> -                      arg_type, ret_type, gen, success_response, boxed,
> -                      allow_oob, allow_preconfig, coroutine):
> +    def visit_command(
> +        self,
> +        name: str,
> +        info: Optional[QAPISourceInfo],
> +        ifcond: QAPISchemaIfCond,
> +        features: List[QAPISchemaFeature],
> +        arg_type: Optional[QAPISchemaObjectType],
> +        ret_type: Optional[QAPISchemaType],
> +        gen: bool,
> +        success_response: bool,
> +        boxed: bool,
> +        allow_oob: bool,
> +        allow_preconfig: bool,
> +        coroutine: bool,
> +    ) -> None:
>          pass
>  
> -    def visit_event(self, name, info, ifcond, features, arg_type, boxed):
> +    def visit_event(
> +        self,
> +        name: str,
> +        info: Optional[QAPISourceInfo],
> +        ifcond: QAPISchemaIfCond,
> +        features: List[QAPISchemaFeature],
> +        arg_type: Optional[QAPISchemaObjectType],
> +        boxed: bool,
> +    ) -> None:
>          pass
>  
>  
> @@ -177,9 +258,9 @@ class QAPISchemaModule:
>  
>      BUILTIN_MODULE_NAME = './builtin'
>  
> -    def __init__(self, name):
> +    def __init__(self, name: str):
>          self.name = name
> -        self._entity_list = []
> +        self._entity_list: List[QAPISchemaEntity] = []
>  
>      @staticmethod
>      def is_system_module(name: str) -> bool:
> @@ -208,10 +289,10 @@ def is_builtin_module(cls, name: str) -> bool:
>          """
>          return name == cls.BUILTIN_MODULE_NAME
>  
> -    def add_entity(self, ent):
> +    def add_entity(self, ent: QAPISchemaEntity) -> None:
>          self._entity_list.append(ent)
>  
> -    def visit(self, visitor):
> +    def visit(self, visitor: QAPISchemaVisitor) -> None:
>          visitor.visit_module(self.name)
>          for entity in self._entity_list:
>              if visitor.visit_needed(entity):
> @@ -219,13 +300,13 @@ def visit(self, visitor):
>  
>  
>  class QAPISchemaInclude(QAPISchemaEntity):
> -    def __init__(self, sub_module, info):
> +    def __init__(self, sub_module: QAPISchemaModule, info: QAPISourceInfo):
>          # Includes are internal entity objects; and may occur multiple times
>          name = f"q_include_{info.fname}:{info.line}"
>          super().__init__(name, info, None)
>          self._sub_module = sub_module
>  
> -    def visit(self, visitor):
> +    def visit(self, visitor: QAPISchemaVisitor) -> None:
>          super().visit(visitor)
>          visitor.visit_include(self._sub_module.name, self.info)
>  
> @@ -237,17 +318,17 @@ def c_type(self) -> str:
>          raise NotImplementedError()
>  
>      # Return the C type to be used in a parameter list.
> -    def c_param_type(self):
> +    def c_param_type(self) -> str:
>          return self.c_type()
>  
>      # Return the C type to be used where we suppress boxing.
> -    def c_unboxed_type(self):
> +    def c_unboxed_type(self) -> str:
>          return self.c_type()
>  
>      def json_type(self) -> str:
>          raise NotImplementedError()
>  
> -    def alternate_qtype(self):
> +    def alternate_qtype(self) -> Optional[str]:
>          json2qtype = {
>              'null':    'QTYPE_QNULL',
>              'string':  'QTYPE_QSTRING',
> @@ -259,17 +340,17 @@ def alternate_qtype(self):
>          }
>          return json2qtype.get(self.json_type())
>  
> -    def doc_type(self):
> +    def doc_type(self) -> Optional[str]:
>          if self.is_implicit():
>              return None
>          return self.name
>  
> -    def need_has_if_optional(self):
> +    def need_has_if_optional(self) -> bool:
>          # When FOO is a pointer, has_FOO == !!FOO, i.e. has_FOO is redundant.
>          # Except for arrays; see QAPISchemaArrayType.need_has_if_optional().
>          return not self.c_type().endswith(POINTER_SUFFIX)
>  
> -    def check(self, schema):
> +    def check(self, schema: QAPISchema) -> None:
>          super().check(schema)
>          for feat in self.features:
>              if feat.is_special():
> @@ -277,7 +358,7 @@ def check(self, schema):
>                      self.info,
>                      f"feature '{feat.name}' is not supported for types")
>  
> -    def describe(self):
> +    def describe(self) -> str:
>          assert self.meta
>          return "%s type '%s'" % (self.meta, self.name)
>  
> @@ -285,7 +366,7 @@ def describe(self):
>  class QAPISchemaBuiltinType(QAPISchemaType):
>      meta = 'built-in'
>  
> -    def __init__(self, name, json_type, c_type):
> +    def __init__(self, name: str, json_type: str, c_type: str):
>          super().__init__(name, None, None)
>          assert not c_type or isinstance(c_type, str)
>          assert json_type in ('string', 'number', 'int', 'boolean', 'null',
> @@ -293,24 +374,24 @@ def __init__(self, name, json_type, c_type):
>          self._json_type_name = json_type
>          self._c_type_name = c_type
>  
> -    def c_name(self):
> +    def c_name(self) -> str:
>          return self.name
>  
> -    def c_type(self):
> +    def c_type(self) -> str:
>          return self._c_type_name
>  
> -    def c_param_type(self):
> +    def c_param_type(self) -> str:
>          if self.name == 'str':
>              return 'const ' + self._c_type_name
>          return self._c_type_name
>  
> -    def json_type(self):
> +    def json_type(self) -> str:
>          return self._json_type_name
>  
> -    def doc_type(self):
> +    def doc_type(self) -> str:
>          return self.json_type()
>  
> -    def visit(self, visitor):
> +    def visit(self, visitor: QAPISchemaVisitor) -> None:
>          super().visit(visitor)
>          visitor.visit_builtin_type(self.name, self.info, self.json_type())
>  
> @@ -318,7 +399,16 @@ def visit(self, visitor):
>  class QAPISchemaEnumType(QAPISchemaType):
>      meta = 'enum'
>  
> -    def __init__(self, name, info, doc, ifcond, features, members, prefix):
> +    def __init__(
> +        self,
> +        name: str,
> +        info: Optional[QAPISourceInfo],
> +        doc: Optional[QAPIDoc],
> +        ifcond: Optional[QAPISchemaIfCond],
> +        features: Optional[List[QAPISchemaFeature]],
> +        members: List[QAPISchemaEnumMember],
> +        prefix: Optional[str],
> +    ):
>          super().__init__(name, info, doc, ifcond, features)
>          for m in members:
>              assert isinstance(m, QAPISchemaEnumMember)
> @@ -327,32 +417,32 @@ def __init__(self, name, info, doc, ifcond, features, members, prefix):
>          self.members = members
>          self.prefix = prefix
>  
> -    def check(self, schema):
> +    def check(self, schema: QAPISchema) -> None:
>          super().check(schema)
> -        seen = {}
> +        seen: Dict[str, QAPISchemaMember] = {}
>          for m in self.members:
>              m.check_clash(self.info, seen)
>  
> -    def connect_doc(self, doc=None):
> +    def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
>          super().connect_doc(doc)
>          doc = doc or self.doc
>          for m in self.members:
>              m.connect_doc(doc)
>  
> -    def is_implicit(self):
> +    def is_implicit(self) -> bool:
>          # See QAPISchema._def_predefineds()
>          return self.name == 'QType'
>  
> -    def c_type(self):
> +    def c_type(self) -> str:
>          return c_name(self.name)
>  
> -    def member_names(self):
> +    def member_names(self) -> List[str]:
>          return [m.name for m in self.members]
>  
> -    def json_type(self):
> +    def json_type(self) -> str:
>          return 'string'
>  
> -    def visit(self, visitor):
> +    def visit(self, visitor: QAPISchemaVisitor) -> None:
>          super().visit(visitor)
>          visitor.visit_enum_type(
>              self.name, self.info, self.ifcond, self.features,
> @@ -362,7 +452,9 @@ def visit(self, visitor):
>  class QAPISchemaArrayType(QAPISchemaType):
>      meta = 'array'
>  
> -    def __init__(self, name, info, element_type):
> +    def __init__(
> +        self, name: str, info: Optional[QAPISourceInfo], element_type: str
> +    ):
>          super().__init__(name, info, None)
>          assert isinstance(element_type, str)
>          self._element_type_name = element_type
> @@ -377,12 +469,12 @@ def element_type(self) -> QAPISchemaType:
>              )
>          return self._element_type
>  
> -    def need_has_if_optional(self):
> +    def need_has_if_optional(self) -> bool:
>          # When FOO is an array, we still need has_FOO to distinguish
>          # absent (!has_FOO) from present and empty (has_FOO && !FOO).
>          return True
>  
> -    def check(self, schema):
> +    def check(self, schema: QAPISchema) -> None:
>          super().check(schema)
>  
>          if self.info:
> @@ -396,42 +488,51 @@ def check(self, schema):
>          )
>          assert not isinstance(self.element_type, QAPISchemaArrayType)
>  
> -    def set_module(self, schema):
> +    def set_module(self, schema: QAPISchema) -> None:
>          self._set_module(schema, self.element_type.info)
>  
>      @property
> -    def ifcond(self):
> +    def ifcond(self) -> QAPISchemaIfCond:
>          assert self._checked
>          return self.element_type.ifcond
>  
> -    def is_implicit(self):
> +    def is_implicit(self) -> bool:
>          return True
>  
> -    def c_type(self):
> +    def c_type(self) -> str:
>          return c_name(self.name) + POINTER_SUFFIX
>  
> -    def json_type(self):
> +    def json_type(self) -> str:
>          return 'array'
>  
> -    def doc_type(self):
> +    def doc_type(self) -> Optional[str]:
>          elt_doc_type = self.element_type.doc_type()
>          if not elt_doc_type:
>              return None
>          return 'array of ' + elt_doc_type
>  
> -    def visit(self, visitor):
> +    def visit(self, visitor: QAPISchemaVisitor) -> None:
>          super().visit(visitor)
>          visitor.visit_array_type(self.name, self.info, self.ifcond,
>                                   self.element_type)
>  
> -    def describe(self):
> +    def describe(self) -> str:
>          assert self.meta
>          return "%s type ['%s']" % (self.meta, self._element_type_name)
>  
>  
>  class QAPISchemaObjectType(QAPISchemaType):
> -    def __init__(self, name, info, doc, ifcond, features,
> -                 base, local_members, variants):
> +    def __init__(
> +        self,
> +        name: str,
> +        info: Optional[QAPISourceInfo],
> +        doc: Optional[QAPIDoc],
> +        ifcond: Optional[QAPISchemaIfCond],
> +        features: Optional[List[QAPISchemaFeature]],
> +        base: Optional[str],
> +        local_members: List[QAPISchemaObjectTypeMember],
> +        variants: Optional[QAPISchemaVariants],
> +    ):
>          # struct has local_members, optional base, and no variants
>          # union has base, variants, and no local_members
>          super().__init__(name, info, doc, ifcond, features)
> @@ -450,7 +551,7 @@ def __init__(self, name, info, doc, ifcond, features,
>          self.members: List[QAPISchemaObjectTypeMember] = []
>          self._checking = False
>  
> -    def check(self, schema):
> +    def check(self, schema: QAPISchema) -> None:
>          # This calls another type T's .check() exactly when the C
>          # struct emitted by gen_object() contains that T's C struct
>          # (pointers don't count).
> @@ -496,14 +597,18 @@ def check(self, schema):
>      # Check that the members of this type do not cause duplicate JSON members,
>      # and update seen to track the members seen so far. Report any errors
>      # on behalf of info, which is not necessarily self.info
> -    def check_clash(self, info, seen):
> +    def check_clash(
> +        self,
> +        info: Optional[QAPISourceInfo],
> +        seen: Dict[str, QAPISchemaMember],
> +    ) -> None:
>          assert self._checked
>          for m in self.members:
>              m.check_clash(info, seen)
>          if self.variants:
>              self.variants.check_clash(info, seen)
>  
> -    def connect_doc(self, doc=None):
> +    def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
>          super().connect_doc(doc)
>          doc = doc or self.doc
>          if self.base and self.base.is_implicit():
> @@ -511,34 +616,34 @@ def connect_doc(self, doc=None):
>          for m in self.local_members:
>              m.connect_doc(doc)
>  
> -    def is_implicit(self):
> +    def is_implicit(self) -> bool:
>          # See QAPISchema._make_implicit_object_type(), as well as
>          # _def_predefineds()
>          return self.name.startswith('q_')
>  
> -    def is_empty(self):
> +    def is_empty(self) -> bool:
>          assert self.members is not None
>          return not self.members and not self.variants
>  
> -    def has_conditional_members(self):
> +    def has_conditional_members(self) -> bool:
>          assert self.members is not None
>          return any(m.ifcond.is_present() for m in self.members)
>  
> -    def c_name(self):
> +    def c_name(self) -> str:
>          assert self.name != 'q_empty'
>          return super().c_name()
>  
> -    def c_type(self):
> +    def c_type(self) -> str:
>          assert not self.is_implicit()
>          return c_name(self.name) + POINTER_SUFFIX
>  
> -    def c_unboxed_type(self):
> +    def c_unboxed_type(self) -> str:
>          return c_name(self.name)
>  
> -    def json_type(self):
> +    def json_type(self) -> str:
>          return 'object'
>  
> -    def visit(self, visitor):
> +    def visit(self, visitor: QAPISchemaVisitor) -> None:
>          super().visit(visitor)
>          visitor.visit_object_type(
>              self.name, self.info, self.ifcond, self.features,
> @@ -551,7 +656,15 @@ def visit(self, visitor):
>  class QAPISchemaAlternateType(QAPISchemaType):
>      meta = 'alternate'
>  
> -    def __init__(self, name, info, doc, ifcond, features, variants):
> +    def __init__(
> +        self,
> +        name: str,
> +        info: QAPISourceInfo,
> +        doc: Optional[QAPIDoc],
> +        ifcond: Optional[QAPISchemaIfCond],
> +        features: List[QAPISchemaFeature],
> +        variants: QAPISchemaVariants,
> +    ):
>          super().__init__(name, info, doc, ifcond, features)
>          assert isinstance(variants, QAPISchemaVariants)
>          assert variants.tag_member
> @@ -559,7 +672,7 @@ def __init__(self, name, info, doc, ifcond, features, variants):
>          variants.tag_member.set_defined_in(self.name)
>          self.variants = variants
>  
> -    def check(self, schema):
> +    def check(self, schema: QAPISchema) -> None:
>          super().check(schema)
>          self.variants.tag_member.check(schema)
>          # Not calling self.variants.check_clash(), because there's nothing
> @@ -567,8 +680,8 @@ def check(self, schema):
>          self.variants.check(schema, {})
>          # Alternate branch names have no relation to the tag enum values;
>          # so we have to check for potential name collisions ourselves.
> -        seen = {}
> -        types_seen = {}
> +        seen: Dict[str, QAPISchemaMember] = {}
> +        types_seen: Dict[str, str] = {}
>          for v in self.variants.variants:
>              v.check_clash(self.info, seen)
>              qtype = v.type.alternate_qtype()
> @@ -597,26 +710,32 @@ def check(self, schema):
>                          % (v.describe(self.info), types_seen[qt]))
>                  types_seen[qt] = v.name
>  
> -    def connect_doc(self, doc=None):
> +    def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
>          super().connect_doc(doc)
>          doc = doc or self.doc
>          for v in self.variants.variants:
>              v.connect_doc(doc)
>  
> -    def c_type(self):
> +    def c_type(self) -> str:
>          return c_name(self.name) + POINTER_SUFFIX
>  
> -    def json_type(self):
> +    def json_type(self) -> str:
>          return 'value'
>  
> -    def visit(self, visitor):
> +    def visit(self, visitor: QAPISchemaVisitor) -> None:
>          super().visit(visitor)
>          visitor.visit_alternate_type(
>              self.name, self.info, self.ifcond, self.features, self.variants)
>  
>  
>  class QAPISchemaVariants:
> -    def __init__(self, tag_name, info, tag_member, variants):
> +    def __init__(
> +        self,
> +        tag_name: Optional[str],
> +        info: QAPISourceInfo,
> +        tag_member: Optional[QAPISchemaObjectTypeMember],
> +        variants: List[QAPISchemaVariant],
> +    ):
>          # Unions pass tag_name but not tag_member.
>          # Alternates pass tag_member but not tag_name.
>          # After check(), tag_member is always set.
> @@ -627,11 +746,11 @@ def __init__(self, tag_name, info, tag_member, variants):
>              assert isinstance(v, QAPISchemaVariant)
>          self._tag_name = tag_name
>          self.info = info
> -        self._tag_member: Optional[QAPISchemaObjectTypeMember] = tag_member
> +        self._tag_member = tag_member

Appreciate your removal of now unnecessary type hints!

>          self.variants = variants
>  
>      @property
> -    def tag_member(self) -> 'QAPISchemaObjectTypeMember':
> +    def tag_member(self) -> QAPISchemaObjectTypeMember:

Is this a cleanup enabled by leaving Python 3.6 behind?

>          if self._tag_member is None:
>              raise RuntimeError(
>                  "QAPISchemaVariants has no tag_member property until "
> @@ -639,11 +758,13 @@ def tag_member(self) -> 'QAPISchemaObjectTypeMember':
>              )
>          return self._tag_member
>  
> -    def set_defined_in(self, name):
> +    def set_defined_in(self, name: str) -> None:
>          for v in self.variants:
>              v.set_defined_in(name)
>  
> -    def check(self, schema, seen):
> +    def check(
> +        self, schema: QAPISchema, seen: Dict[str, QAPISchemaMember]
> +    ) -> None:
>          if self._tag_name:      # union
>              # We need to narrow the member type:
>              tmp = seen.get(c_name(self._tag_name))
> @@ -713,7 +834,11 @@ def check(self, schema, seen):
>                          % (v.describe(self.info), v.type.describe()))
>                  v.type.check(schema)
>  
> -    def check_clash(self, info, seen):
> +    def check_clash(
> +        self,
> +        info: Optional[QAPISourceInfo],
> +        seen: Dict[str, QAPISchemaMember],
> +    ) -> None:
>          for v in self.variants:
>              # Reset seen map for each variant, since qapi names from one
>              # branch do not affect another branch
> @@ -725,18 +850,27 @@ class QAPISchemaMember:
>      """ Represents object members, enum members and features """
>      role = 'member'
>  
> -    def __init__(self, name, info, ifcond=None):
> +    def __init__(
> +        self,
> +        name: str,
> +        info: Optional[QAPISourceInfo],
> +        ifcond: Optional[QAPISchemaIfCond] = None,
> +    ):
>          assert isinstance(name, str)
>          self.name = name
>          self.info = info
>          self.ifcond = ifcond or QAPISchemaIfCond()
> -        self.defined_in = None
> +        self.defined_in: Optional[str] = None
>  
> -    def set_defined_in(self, name):
> +    def set_defined_in(self, name: str) -> None:
>          assert not self.defined_in
>          self.defined_in = name
>  
> -    def check_clash(self, info, seen):
> +    def check_clash(
> +        self,
> +        info: Optional[QAPISourceInfo],
> +        seen: Dict[str, QAPISchemaMember],
> +    ) -> None:
>          cname = c_name(self.name)
>          if cname in seen:
>              raise QAPISemError(
> @@ -745,11 +879,11 @@ def check_clash(self, info, seen):
>                  % (self.describe(info), seen[cname].describe(info)))
>          seen[cname] = self
>  
> -    def connect_doc(self, doc):
> +    def connect_doc(self, doc: Optional[QAPIDoc]) -> None:
>          if doc:
>              doc.connect_member(self)
>  
> -    def describe(self, info):
> +    def describe(self, info: Optional[QAPISourceInfo]) -> str:
>          role = self.role
>          meta = 'type'
>          defined_in = self.defined_in
> @@ -781,14 +915,20 @@ def describe(self, info):
>  class QAPISchemaEnumMember(QAPISchemaMember):
>      role = 'value'
>  
> -    def __init__(self, name, info, ifcond=None, features=None):
> +    def __init__(
> +        self,
> +        name: str,
> +        info: Optional[QAPISourceInfo],
> +        ifcond: Optional[QAPISchemaIfCond] = None,
> +        features: Optional[List[QAPISchemaFeature]] = None,
> +    ):
>          super().__init__(name, info, ifcond)
>          for f in features or []:
>              assert isinstance(f, QAPISchemaFeature)
>              f.set_defined_in(name)
>          self.features = features or []
>  
> -    def connect_doc(self, doc):
> +    def connect_doc(self, doc: Optional[QAPIDoc]) -> None:
>          super().connect_doc(doc)
>          if doc:
>              for f in self.features:
> @@ -798,12 +938,20 @@ def connect_doc(self, doc):
>  class QAPISchemaFeature(QAPISchemaMember):
>      role = 'feature'
>  
> -    def is_special(self):
> +    def is_special(self) -> bool:
>          return self.name in ('deprecated', 'unstable')
>  
>  
>  class QAPISchemaObjectTypeMember(QAPISchemaMember):
> -    def __init__(self, name, info, typ, optional, ifcond=None, features=None):
> +    def __init__(
> +        self,
> +        name: str,
> +        info: QAPISourceInfo,
> +        typ: str,
> +        optional: bool,
> +        ifcond: Optional[QAPISchemaIfCond] = None,
> +        features: Optional[List[QAPISchemaFeature]] = None,
> +    ):
>          super().__init__(name, info, ifcond)
>          assert isinstance(typ, str)
>          assert isinstance(optional, bool)
> @@ -815,19 +963,19 @@ def __init__(self, name, info, typ, optional, ifcond=None, features=None):
>          self.optional = optional
>          self.features = features or []
>  
> -    def need_has(self):
> +    def need_has(self) -> bool:
>          assert self.type
>          return self.optional and self.type.need_has_if_optional()
>  
> -    def check(self, schema):
> +    def check(self, schema: QAPISchema) -> None:
>          assert self.defined_in
>          self.type = schema.resolve_type(self._type_name, self.info,
>                                          self.describe)
> -        seen = {}
> +        seen: Dict[str, QAPISchemaMember] = {}
>          for f in self.features:
>              f.check_clash(self.info, seen)
>  
> -    def connect_doc(self, doc):
> +    def connect_doc(self, doc: Optional[QAPIDoc]) -> None:
>          super().connect_doc(doc)
>          if doc:
>              for f in self.features:
> @@ -837,24 +985,42 @@ def connect_doc(self, doc):
>  class QAPISchemaVariant(QAPISchemaObjectTypeMember):
>      role = 'branch'
>  
> -    def __init__(self, name, info, typ, ifcond=None):
> +    def __init__(
> +        self,
> +        name: str,
> +        info: QAPISourceInfo,
> +        typ: str,
> +        ifcond: QAPISchemaIfCond,
> +    ):
>          super().__init__(name, info, typ, False, ifcond)
>  
>  
>  class QAPISchemaCommand(QAPISchemaEntity):
>      meta = 'command'
>  
> -    def __init__(self, name, info, doc, ifcond, features,
> -                 arg_type, ret_type,
> -                 gen, success_response, boxed, allow_oob, allow_preconfig,
> -                 coroutine):
> +    def __init__(
> +        self,
> +        name: str,
> +        info: QAPISourceInfo,
> +        doc: Optional[QAPIDoc],
> +        ifcond: QAPISchemaIfCond,
> +        features: List[QAPISchemaFeature],
> +        arg_type: Optional[str],
> +        ret_type: Optional[str],
> +        gen: bool,
> +        success_response: bool,
> +        boxed: bool,
> +        allow_oob: bool,
> +        allow_preconfig: bool,
> +        coroutine: bool,
> +    ):
>          super().__init__(name, info, doc, ifcond, features)
>          assert not arg_type or isinstance(arg_type, str)
>          assert not ret_type or isinstance(ret_type, str)
>          self._arg_type_name = arg_type
> -        self.arg_type = None
> +        self.arg_type: Optional[QAPISchemaObjectType] = None
>          self._ret_type_name = ret_type
> -        self.ret_type = None
> +        self.ret_type: Optional[QAPISchemaType] = None
>          self.gen = gen
>          self.success_response = success_response
>          self.boxed = boxed
> @@ -862,7 +1028,7 @@ def __init__(self, name, info, doc, ifcond, features,
>          self.allow_preconfig = allow_preconfig
>          self.coroutine = coroutine
>  
> -    def check(self, schema):
> +    def check(self, schema: QAPISchema) -> None:
>          assert self.info is not None
>          super().check(schema)
>          if self._arg_type_name:
> @@ -897,14 +1063,14 @@ def check(self, schema):
>                          "command's 'returns' cannot take %s"
>                          % self.ret_type.describe())
>  
> -    def connect_doc(self, doc=None):
> +    def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
>          super().connect_doc(doc)
>          doc = doc or self.doc
>          if doc:
>              if self.arg_type and self.arg_type.is_implicit():
>                  self.arg_type.connect_doc(doc)
>  
> -    def visit(self, visitor):
> +    def visit(self, visitor: QAPISchemaVisitor) -> None:
>          super().visit(visitor)
>          visitor.visit_command(
>              self.name, self.info, self.ifcond, self.features,
> @@ -916,14 +1082,23 @@ def visit(self, visitor):
>  class QAPISchemaEvent(QAPISchemaEntity):
>      meta = 'event'
>  
> -    def __init__(self, name, info, doc, ifcond, features, arg_type, boxed):
> +    def __init__(
> +        self,
> +        name: str,
> +        info: QAPISourceInfo,
> +        doc: Optional[QAPIDoc],
> +        ifcond: QAPISchemaIfCond,
> +        features: List[QAPISchemaFeature],
> +        arg_type: Optional[str],
> +        boxed: bool,
> +    ):
>          super().__init__(name, info, doc, ifcond, features)
>          assert not arg_type or isinstance(arg_type, str)
>          self._arg_type_name = arg_type
> -        self.arg_type = None
> +        self.arg_type: Optional[QAPISchemaObjectType] = None
>          self.boxed = boxed
>  
> -    def check(self, schema):
> +    def check(self, schema: QAPISchema) -> None:
>          super().check(schema)
>          if self._arg_type_name:
>              typ = schema.resolve_type(
> @@ -945,14 +1120,14 @@ def check(self, schema):
>                      self.info,
>                      "conditional event arguments require 'boxed': true")
>  
> -    def connect_doc(self, doc=None):
> +    def connect_doc(self, doc: Optional[QAPIDoc] = None) -> None:
>          super().connect_doc(doc)
>          doc = doc or self.doc
>          if doc:
>              if self.arg_type and self.arg_type.is_implicit():
>                  self.arg_type.connect_doc(doc)
>  
> -    def visit(self, visitor):
> +    def visit(self, visitor: QAPISchemaVisitor) -> None:
>          super().visit(visitor)
>          visitor.visit_event(
>              self.name, self.info, self.ifcond, self.features,
> @@ -960,7 +1135,7 @@ def visit(self, visitor):
>  
>  
>  class QAPISchema:
> -    def __init__(self, fname):
> +    def __init__(self, fname: str):
>          self.fname = fname
>  
>          try:
> @@ -972,9 +1147,9 @@ def __init__(self, fname):
>  
>          exprs = check_exprs(parser.exprs)
>          self.docs = parser.docs
> -        self._entity_list = []
> -        self._entity_dict = {}
> -        self._module_dict = OrderedDict()
> +        self._entity_list: List[QAPISchemaEntity] = []
> +        self._entity_dict: Dict[str, QAPISchemaEntity] = {}
> +        self._module_dict: Dict[str, QAPISchemaModule] = OrderedDict()

Apropos OrderedDict: plain dict guarantees insertion order since Python
3.7.  We should look into switching from OrderedDict to dict.  Not in
this patch, of course, and no need to do it in this series.

>          self._schema_dir = os.path.dirname(fname)
>          self._make_module(QAPISchemaModule.BUILTIN_MODULE_NAME)
>          self._make_module(fname)
> @@ -984,7 +1159,7 @@ def __init__(self, fname):
>          self._def_exprs(exprs)
>          self.check()
>  
> -    def _def_entity(self, ent):
> +    def _def_entity(self, ent: QAPISchemaEntity) -> None:
>          # Only the predefined types are allowed to not have info
>          assert ent.info or self._predefining
>          self._entity_list.append(ent)
> @@ -1003,7 +1178,11 @@ def _def_entity(self, ent):
>                  ent.info, "%s is already defined" % other_ent.describe())
>          self._entity_dict[ent.name] = ent
>  
> -    def lookup_entity(self, name, typ=None):
> +    def lookup_entity(
> +        self,
> +        name: str,
> +        typ: Optional[type] = None,
> +    ) -> Optional[QAPISchemaEntity]:
>          ent = self._entity_dict.get(name)
>          if typ and not isinstance(ent, typ):
>              return None
> @@ -1016,7 +1195,12 @@ def lookup_type(self, name: str) -> Optional[QAPISchemaType]:
>          assert isinstance(typ, QAPISchemaType)
>          return typ
>  
> -    def resolve_type(self, name, info, what):
> +    def resolve_type(
> +        self,
> +        name: str,
> +        info: Optional[QAPISourceInfo],
> +        what: Union[str, Callable[[Optional[QAPISourceInfo]], str]],
> +    ) -> QAPISchemaType:
>          typ = self.lookup_type(name)
>          if not typ:
>              if callable(what):
> @@ -1030,23 +1214,25 @@ def _module_name(self, fname: str) -> str:
>              return fname
>          return os.path.relpath(fname, self._schema_dir)
>  
> -    def _make_module(self, fname):
> +    def _make_module(self, fname: str) -> QAPISchemaModule:
>          name = self._module_name(fname)
>          if name not in self._module_dict:
>              self._module_dict[name] = QAPISchemaModule(name)
>          return self._module_dict[name]
>  
> -    def module_by_fname(self, fname):
> +    def module_by_fname(self, fname: str) -> QAPISchemaModule:
>          name = self._module_name(fname)
>          return self._module_dict[name]
>  
> -    def _def_include(self, expr: QAPIExpression):
> +    def _def_include(self, expr: QAPIExpression) -> None:
>          include = expr['include']
>          assert expr.doc is None
>          self._def_entity(
>              QAPISchemaInclude(self._make_module(include), expr.info))
>  
> -    def _def_builtin_type(self, name, json_type, c_type):
> +    def _def_builtin_type(
> +        self, name: str, json_type: str, c_type: str
> +    ) -> None:
>          self._def_entity(QAPISchemaBuiltinType(name, json_type, c_type))
>          # Instantiating only the arrays that are actually used would
>          # be nice, but we can't as long as their generated code
> @@ -1054,7 +1240,7 @@ def _def_builtin_type(self, name, json_type, c_type):
>          # schema.
>          self._make_array_type(name, None)
>  
> -    def _def_predefineds(self):
> +    def _def_predefineds(self) -> None:
>          for t in [('str',    'string',  'char' + POINTER_SUFFIX),
>                    ('number', 'number',  'double'),
>                    ('int',    'int',     'int64_t'),
> @@ -1083,30 +1269,51 @@ def _def_predefineds(self):
>          self._def_entity(QAPISchemaEnumType('QType', None, None, None, None,
>                                              qtype_values, 'QTYPE'))
>  
> -    def _make_features(self, features, info):
> +    def _make_features(
> +        self,
> +        features: Optional[List[Dict[str, Any]]],
> +        info: Optional[QAPISourceInfo],
> +    ) -> List[QAPISchemaFeature]:
>          if features is None:
>              return []
>          return [QAPISchemaFeature(f['name'], info,
>                                    QAPISchemaIfCond(f.get('if')))
>                  for f in features]
>  
> -    def _make_enum_member(self, name, ifcond, features, info):
> +    def _make_enum_member(
> +        self,
> +        name: str,
> +        ifcond: Optional[Union[str, Dict[str, Any]]],

Hmm.  In QAPISchemaIfCond.__init__(), you used Dict[str, object].  Any
particular reason for using Any here, and object there?

Out of curiosity: is there a practical difference between
Optional[Union[A, B]] and Union[None, A, B]?

> +        features: Optional[List[Dict[str, Any]]],
> +        info: Optional[QAPISourceInfo],
> +    ) -> QAPISchemaEnumMember:
>          return QAPISchemaEnumMember(name, info,
>                                      QAPISchemaIfCond(ifcond),
>                                      self._make_features(features, info))
>  
> -    def _make_enum_members(self, values, info):
> +    def _make_enum_members(
> +        self, values: List[Dict[str, Any]], info: Optional[QAPISourceInfo]
> +    ) -> List[QAPISchemaEnumMember]:
>          return [self._make_enum_member(v['name'], v.get('if'),
>                                         v.get('features'), info)
>                  for v in values]
>  
> -    def _make_array_type(self, element_type, info):
> +    def _make_array_type(
> +        self, element_type: str, info: Optional[QAPISourceInfo]
> +    ) -> str:
>          name = element_type + 'List'    # reserved by check_defn_name_str()
>          if not self.lookup_type(name):
>              self._def_entity(QAPISchemaArrayType(name, info, element_type))
>          return name
>  
> -    def _make_implicit_object_type(self, name, info, ifcond, role, members):
> +    def _make_implicit_object_type(
> +        self,
> +        name: str,
> +        info: QAPISourceInfo,
> +        ifcond: QAPISchemaIfCond,
> +        role: str,
> +        members: List[QAPISchemaObjectTypeMember],
> +    ) -> Optional[str]:
>          if not members:
>              return None
>          # See also QAPISchemaObjectTypeMember.describe()
> @@ -1122,7 +1329,7 @@ def _make_implicit_object_type(self, name, info, ifcond, role, members):
>                  name, info, None, ifcond, None, None, members, None))
>          return name
>  
> -    def _def_enum_type(self, expr: QAPIExpression):
> +    def _def_enum_type(self, expr: QAPIExpression) -> None:
>          name = expr['enum']
>          data = expr['data']
>          prefix = expr.get('prefix')
> @@ -1133,7 +1340,14 @@ def _def_enum_type(self, expr: QAPIExpression):
>              name, info, expr.doc, ifcond, features,
>              self._make_enum_members(data, info), prefix))
>  
> -    def _make_member(self, name, typ, ifcond, features, info):
> +    def _make_member(
> +        self,
> +        name: str,
> +        typ: Union[List[str], str],
> +        ifcond: QAPISchemaIfCond,
> +        features: Optional[List[Dict[str, Any]]],
> +        info: QAPISourceInfo,
> +    ) -> QAPISchemaObjectTypeMember:
>          optional = False
>          if name.startswith('*'):
>              name = name[1:]
> @@ -1144,13 +1358,17 @@ def _make_member(self, name, typ, ifcond, features, info):
>          return QAPISchemaObjectTypeMember(name, info, typ, optional, ifcond,
>                                            self._make_features(features, info))
>  
> -    def _make_members(self, data, info):
> +    def _make_members(
> +        self,
> +        data: Dict[str, Any],
> +        info: QAPISourceInfo,
> +    ) -> List[QAPISchemaObjectTypeMember]:
>          return [self._make_member(key, value['type'],
>                                    QAPISchemaIfCond(value.get('if')),
>                                    value.get('features'), info)
>                  for (key, value) in data.items()]
>  
> -    def _def_struct_type(self, expr: QAPIExpression):
> +    def _def_struct_type(self, expr: QAPIExpression) -> None:
>          name = expr['struct']
>          base = expr.get('base')
>          data = expr['data']
> @@ -1162,13 +1380,19 @@ def _def_struct_type(self, expr: QAPIExpression):
>              self._make_members(data, info),
>              None))
>  
> -    def _make_variant(self, case, typ, ifcond, info):
> +    def _make_variant(
> +        self,
> +        case: str,
> +        typ: str,
> +        ifcond: QAPISchemaIfCond,
> +        info: QAPISourceInfo,
> +    ) -> QAPISchemaVariant:
>          if isinstance(typ, list):
>              assert len(typ) == 1
>              typ = self._make_array_type(typ[0], info)
>          return QAPISchemaVariant(case, info, typ, ifcond)
>  
> -    def _def_union_type(self, expr: QAPIExpression):
> +    def _def_union_type(self, expr: QAPIExpression) -> None:
>          name = expr['union']
>          base = expr['base']
>          tag_name = expr['discriminator']
> @@ -1193,7 +1417,7 @@ def _def_union_type(self, expr: QAPIExpression):
>                                   QAPISchemaVariants(
>                                       tag_name, info, None, variants)))
>  
> -    def _def_alternate_type(self, expr: QAPIExpression):
> +    def _def_alternate_type(self, expr: QAPIExpression) -> None:
>          name = expr['alternate']
>          data = expr['data']
>          assert isinstance(data, dict)
> @@ -1211,7 +1435,7 @@ def _def_alternate_type(self, expr: QAPIExpression):
>                  name, info, expr.doc, ifcond, features,
>                  QAPISchemaVariants(None, info, tag_member, variants)))
>  
> -    def _def_command(self, expr: QAPIExpression):
> +    def _def_command(self, expr: QAPIExpression) -> None:
>          name = expr['command']
>          data = expr.get('data')
>          rets = expr.get('returns')
> @@ -1237,7 +1461,7 @@ def _def_command(self, expr: QAPIExpression):
>                                             boxed, allow_oob, allow_preconfig,
>                                             coroutine))
>  
> -    def _def_event(self, expr: QAPIExpression):
> +    def _def_event(self, expr: QAPIExpression) -> None:
>          name = expr['event']
>          data = expr.get('data')
>          boxed = expr.get('boxed', False)
> @@ -1251,7 +1475,7 @@ def _def_event(self, expr: QAPIExpression):
>          self._def_entity(QAPISchemaEvent(name, info, expr.doc, ifcond,
>                                           features, data, boxed))
>  
> -    def _def_exprs(self, exprs):
> +    def _def_exprs(self, exprs: List[QAPIExpression]) -> None:
>          for expr in exprs:
>              if 'enum' in expr:
>                  self._def_enum_type(expr)
> @@ -1270,7 +1494,7 @@ def _def_exprs(self, exprs):
>              else:
>                  assert False
>  
> -    def check(self):
> +    def check(self) -> None:
>          for ent in self._entity_list:
>              ent.check(self)
>              ent.connect_doc()
> @@ -1278,7 +1502,7 @@ def check(self):
>          for ent in self._entity_list:
>              ent.set_module(self)
>  
> -    def visit(self, visitor):
> +    def visit(self, visitor: QAPISchemaVisitor) -> None:
>          visitor.visit_begin(self)
>          for mod in self._module_dict.values():
>              mod.visit(visitor)

Big patch, impractical to review thoroughly, but I did what I could, and
since mypy is content, that should be good enough.

[PATCH 17/19] qapi/schema: turn on mypy strictness

[John Snow]

This patch can be rolled in with the previous one once the series is
ready for merge, but for work-in-progress' sake, it's separate here.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/mypy.ini | 5 -----
 1 file changed, 5 deletions(-)

diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
index 56e0dfb1327..8109470a031 100644
--- a/scripts/qapi/mypy.ini
+++ b/scripts/qapi/mypy.ini
@@ -2,8 +2,3 @@
 strict = True
 disallow_untyped_calls = False
 python_version = 3.8
-
-[mypy-qapi.schema]
-disallow_untyped_defs = False
-disallow_incomplete_defs = False
-check_untyped_defs = False
-- 
2.41.0

[PATCH 18/19] qapi/schema: remove unnecessary asserts

[John Snow]

With strict typing enabled, these runtime statements aren't necessary
anymore.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/qapi/schema.py | 23 -----------------------
 1 file changed, 23 deletions(-)

diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index 5d19b59def0..b5f377e68b8 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -78,9 +78,7 @@ def __init__(
         ifcond: Optional[QAPISchemaIfCond] = None,
         features: Optional[List[QAPISchemaFeature]] = None,
     ):
-        assert name is None or isinstance(name, str)
         for f in features or []:
-            assert isinstance(f, QAPISchemaFeature)
             f.set_defined_in(name)
         self.name = name
         self._module: Optional[QAPISchemaModule] = None
@@ -145,7 +143,6 @@ def visit(self, visitor: QAPISchemaVisitor) -> None:
         assert self._checked
 
     def describe(self) -> str:
-        assert self.meta
         return "%s '%s'" % (self.meta, self.name)
 
 
@@ -359,7 +356,6 @@ def check(self, schema: QAPISchema) -> None:
                     f"feature '{feat.name}' is not supported for types")
 
     def describe(self) -> str:
-        assert self.meta
         return "%s type '%s'" % (self.meta, self.name)
 
 
@@ -368,7 +364,6 @@ class QAPISchemaBuiltinType(QAPISchemaType):
 
     def __init__(self, name: str, json_type: str, c_type: str):
         super().__init__(name, None, None)
-        assert not c_type or isinstance(c_type, str)
         assert json_type in ('string', 'number', 'int', 'boolean', 'null',
                              'value')
         self._json_type_name = json_type
@@ -411,9 +406,7 @@ def __init__(
     ):
         super().__init__(name, info, doc, ifcond, features)
         for m in members:
-            assert isinstance(m, QAPISchemaEnumMember)
             m.set_defined_in(name)
-        assert prefix is None or isinstance(prefix, str)
         self.members = members
         self.prefix = prefix
 
@@ -456,7 +449,6 @@ def __init__(
         self, name: str, info: Optional[QAPISourceInfo], element_type: str
     ):
         super().__init__(name, info, None)
-        assert isinstance(element_type, str)
         self._element_type_name = element_type
         self._element_type: Optional[QAPISchemaType] = None
 
@@ -517,7 +509,6 @@ def visit(self, visitor: QAPISchemaVisitor) -> None:
                                  self.element_type)
 
     def describe(self) -> str:
-        assert self.meta
         return "%s type ['%s']" % (self.meta, self._element_type_name)
 
 
@@ -537,12 +528,9 @@ def __init__(
         # union has base, variants, and no local_members
         super().__init__(name, info, doc, ifcond, features)
         self.meta = 'union' if variants else 'struct'
-        assert base is None or isinstance(base, str)
         for m in local_members:
-            assert isinstance(m, QAPISchemaObjectTypeMember)
             m.set_defined_in(name)
         if variants is not None:
-            assert isinstance(variants, QAPISchemaVariants)
             variants.set_defined_in(name)
         self._base_name = base
         self.base = None
@@ -666,7 +654,6 @@ def __init__(
         variants: QAPISchemaVariants,
     ):
         super().__init__(name, info, doc, ifcond, features)
-        assert isinstance(variants, QAPISchemaVariants)
         assert variants.tag_member
         variants.set_defined_in(name)
         variants.tag_member.set_defined_in(self.name)
@@ -742,8 +729,6 @@ def __init__(
         assert bool(tag_member) != bool(tag_name)
         assert (isinstance(tag_name, str) or
                 isinstance(tag_member, QAPISchemaObjectTypeMember))
-        for v in variants:
-            assert isinstance(v, QAPISchemaVariant)
         self._tag_name = tag_name
         self.info = info
         self._tag_member = tag_member
@@ -856,7 +841,6 @@ def __init__(
         info: Optional[QAPISourceInfo],
         ifcond: Optional[QAPISchemaIfCond] = None,
     ):
-        assert isinstance(name, str)
         self.name = name
         self.info = info
         self.ifcond = ifcond or QAPISchemaIfCond()
@@ -924,7 +908,6 @@ def __init__(
     ):
         super().__init__(name, info, ifcond)
         for f in features or []:
-            assert isinstance(f, QAPISchemaFeature)
             f.set_defined_in(name)
         self.features = features or []
 
@@ -953,10 +936,7 @@ def __init__(
         features: Optional[List[QAPISchemaFeature]] = None,
     ):
         super().__init__(name, info, ifcond)
-        assert isinstance(typ, str)
-        assert isinstance(optional, bool)
         for f in features or []:
-            assert isinstance(f, QAPISchemaFeature)
             f.set_defined_in(name)
         self._type_name = typ
         self.type: QAPISchemaType  # set during check(). Kind of hokey.
@@ -1015,8 +995,6 @@ def __init__(
         coroutine: bool,
     ):
         super().__init__(name, info, doc, ifcond, features)
-        assert not arg_type or isinstance(arg_type, str)
-        assert not ret_type or isinstance(ret_type, str)
         self._arg_type_name = arg_type
         self.arg_type: Optional[QAPISchemaObjectType] = None
         self._ret_type_name = ret_type
@@ -1093,7 +1071,6 @@ def __init__(
         boxed: bool,
     ):
         super().__init__(name, info, doc, ifcond, features)
-        assert not arg_type or isinstance(arg_type, str)
         self._arg_type_name = arg_type
         self.arg_type: Optional[QAPISchemaObjectType] = None
         self.boxed = boxed
-- 
2.41.0

Re: [PATCH 18/19] qapi/schema: remove unnecessary asserts

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> With strict typing enabled, these runtime statements aren't necessary
> anymore.
>
> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  scripts/qapi/schema.py | 23 -----------------------
>  1 file changed, 23 deletions(-)
>
> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index 5d19b59def0..b5f377e68b8 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -78,9 +78,7 @@ def __init__(
   class QAPISchemaEntity:
       meta: str

       def __init__(
           self,
           name: str,
           info: Optional[QAPISourceInfo],
           doc: Optional[QAPIDoc],
>          ifcond: Optional[QAPISchemaIfCond] = None,
>          features: Optional[List[QAPISchemaFeature]] = None,
>      ):
> -        assert name is None or isinstance(name, str)

Yup, because name: str.

>          for f in features or []:
> -            assert isinstance(f, QAPISchemaFeature)

Yup, because features: Optional[List[QAPISchemaFeature]].

>              f.set_defined_in(name)
>          self.name = name
>          self._module: Optional[QAPISchemaModule] = None
> @@ -145,7 +143,6 @@ def visit(self, visitor: QAPISchemaVisitor) -> None:
>          assert self._checked
>  
>      def describe(self) -> str:
> -        assert self.meta

Yup, because QAPISchemaEntity has meta: str.

>          return "%s '%s'" % (self.meta, self.name)
>  
>  
> @@ -359,7 +356,6 @@ def check(self, schema: QAPISchema) -> None:
>                      f"feature '{feat.name}' is not supported for types")
>  
>      def describe(self) -> str:
> -        assert self.meta

Likewise.

>          return "%s type '%s'" % (self.meta, self.name)
>  
>  
> @@ -368,7 +364,6 @@ class QAPISchemaBuiltinType(QAPISchemaType):
   class QAPISchemaBuiltinType(QAPISchemaType):
       meta = 'built-in'
>  
>      def __init__(self, name: str, json_type: str, c_type: str):
>          super().__init__(name, None, None)
> -        assert not c_type or isinstance(c_type, str)

Yup, because c_type: str.

Odd: the assertion accepts None, but the type doesn't.  Turns out None
was possible until commit 2d21291ae64 (qapi: Pseudo-type '**' is now
unused, drop it).  The assertion should have been adjusted then.

Probably not worth a commit message mention now.

>          assert json_type in ('string', 'number', 'int', 'boolean', 'null',
>                               'value')
>          self._json_type_name = json_type
> @@ -411,9 +406,7 @@ def __init__(
   class QAPISchemaEnumType(QAPISchemaType):
       meta = 'enum'

       def __init__(
           self,
           name: str,
           info: Optional[QAPISourceInfo],
           doc: Optional[QAPIDoc],
           ifcond: Optional[QAPISchemaIfCond],
           features: Optional[List[QAPISchemaFeature]],
           members: List[QAPISchemaEnumMember],
           prefix: Optional[str],
>      ):
>          super().__init__(name, info, doc, ifcond, features)
>          for m in members:
> -            assert isinstance(m, QAPISchemaEnumMember)

Yup, because members: List[QAPISchemaEnumMember].

>              m.set_defined_in(name)
> -        assert prefix is None or isinstance(prefix, str)

Yup, because prefix: Optional[str].

>          self.members = members
>          self.prefix = prefix
>  
> @@ -456,7 +449,6 @@ def __init__(
   class QAPISchemaArrayType(QAPISchemaType):
       meta = 'array'

       def __init__(
>          self, name: str, info: Optional[QAPISourceInfo], element_type: str
>      ):
>          super().__init__(name, info, None)
> -        assert isinstance(element_type, str)

Yup, because element_type: str.

>          self._element_type_name = element_type
>          self._element_type: Optional[QAPISchemaType] = None
>  
> @@ -517,7 +509,6 @@ def visit(self, visitor: QAPISchemaVisitor) -> None:
>                                   self.element_type)
>  
>      def describe(self) -> str:
> -        assert self.meta

Yup, because QAPISchemaEntity has meta: str.

>          return "%s type ['%s']" % (self.meta, self._element_type_name)
>  
>  
> @@ -537,12 +528,9 @@ def __init__(
   class QAPISchemaObjectType(QAPISchemaType):
       def __init__(
           self,
           name: str,
           info: Optional[QAPISourceInfo],
           doc: Optional[QAPIDoc],
           ifcond: Optional[QAPISchemaIfCond],
           features: Optional[List[QAPISchemaFeature]],
           base: Optional[str],
           local_members: List[QAPISchemaObjectTypeMember],
           variants: Optional[QAPISchemaVariants],
       ):
           # struct has local_members, optional base, and no variants
>          # union has base, variants, and no local_members
>          super().__init__(name, info, doc, ifcond, features)
>          self.meta = 'union' if variants else 'struct'
> -        assert base is None or isinstance(base, str)

Yup, because base: Optional[str].

>          for m in local_members:
> -            assert isinstance(m, QAPISchemaObjectTypeMember)

Yup, because local_members: List[QAPISchemaObjectTypeMember].

>              m.set_defined_in(name)
>          if variants is not None:
> -            assert isinstance(variants, QAPISchemaVariants)

Yup, because variants: Optional[QAPISchemaVariants]

>              variants.set_defined_in(name)
>          self._base_name = base
>          self.base = None
> @@ -666,7 +654,6 @@ def __init__(
   class QAPISchemaAlternateType(QAPISchemaType):
       meta = 'alternate'

       def __init__(
           self,
           name: str,
           info: QAPISourceInfo,
           doc: Optional[QAPIDoc],
           ifcond: Optional[QAPISchemaIfCond],
           features: List[QAPISchemaFeature],
>          variants: QAPISchemaVariants,
>      ):
>          super().__init__(name, info, doc, ifcond, features)
> -        assert isinstance(variants, QAPISchemaVariants)

Yup, because variants: QAPISchemaVariants.

>          assert variants.tag_member
>          variants.set_defined_in(name)
>          variants.tag_member.set_defined_in(self.name)
> @@ -742,8 +729,6 @@ def __init__(
   class QAPISchemaVariants:
       def __init__(
           self,
           tag_name: Optional[str],
           info: QAPISourceInfo,
           tag_member: Optional[QAPISchemaObjectTypeMember],
           variants: List[QAPISchemaVariant],
       ):
           # Unions pass tag_name but not tag_member.
           # Alternates pass tag_member but not tag_name.
           # After check(), tag_member is always set.
>          assert bool(tag_member) != bool(tag_name)
>          assert (isinstance(tag_name, str) or
>                  isinstance(tag_member, QAPISchemaObjectTypeMember))
> -        for v in variants:
> -            assert isinstance(v, QAPISchemaVariant)

Yup, because variants: List[QAPISchemaVariant].

>          self._tag_name = tag_name
>          self.info = info
>          self._tag_member = tag_member
> @@ -856,7 +841,6 @@ def __init__(
   class QAPISchemaMember:
       """ Represents object members, enum members and features """
       role = 'member'

       def __init__(
           self,
           name: str,
>          info: Optional[QAPISourceInfo],
>          ifcond: Optional[QAPISchemaIfCond] = None,
>      ):
> -        assert isinstance(name, str)

Yup, because name: str.

>          self.name = name
>          self.info = info
>          self.ifcond = ifcond or QAPISchemaIfCond()
> @@ -924,7 +908,6 @@ def __init__(
   class QAPISchemaEnumMember(QAPISchemaMember):
       role = 'value'

       def __init__(
           self,
           name: str,
           info: QAPISourceInfo,
           typ: str,
           optional: bool,
           ifcond: Optional[QAPISchemaIfCond] = None,
           features: Optional[List[QAPISchemaFeature]] = None,
>      ):
>          super().__init__(name, info, ifcond)
>          for f in features or []:
> -            assert isinstance(f, QAPISchemaFeature)

Yup, because features: Optional[List[QAPISchemaFeature]].

>              f.set_defined_in(name)
>          self.features = features or []
>  
> @@ -953,10 +936,7 @@ def __init__(
   class QAPISchemaObjectTypeMember(QAPISchemaMember):
       def __init__(
           self,
           name: str,
           info: QAPISourceInfo,
           typ: str,
           optional: bool,
           ifcond: Optional[QAPISchemaIfCond] = None,
>          features: Optional[List[QAPISchemaFeature]] = None,
>      ):
>          super().__init__(name, info, ifcond)
> -        assert isinstance(typ, str)

Yup, because typ: str.

> -        assert isinstance(optional, bool)

Yup, because optional: bool.

>          for f in features or []:
> -            assert isinstance(f, QAPISchemaFeature)

Yup, because features: Optional[List[QAPISchemaFeature]].

>              f.set_defined_in(name)
>          self._type_name = typ
>          self.type: QAPISchemaType  # set during check(). Kind of hokey.
> @@ -1015,8 +995,6 @@ def __init__(
   class QAPISchemaCommand(QAPISchemaEntity):
       meta = 'command'

       def __init__(
           self,
           name: str,
           info: QAPISourceInfo,
           doc: Optional[QAPIDoc],
           ifcond: QAPISchemaIfCond,
           features: List[QAPISchemaFeature],
           arg_type: Optional[str],
           ret_type: Optional[str],
           gen: bool,
           success_response: bool,
           boxed: bool,
           allow_oob: bool,
           allow_preconfig: bool,
>          coroutine: bool,
>      ):
>          super().__init__(name, info, doc, ifcond, features)
> -        assert not arg_type or isinstance(arg_type, str)

Yup, because arg_type: Optional[str].

> -        assert not ret_type or isinstance(ret_type, str)

Yup, because ret_type: Optional[str].

>          self._arg_type_name = arg_type
>          self.arg_type: Optional[QAPISchemaObjectType] = None
>          self._ret_type_name = ret_type
> @@ -1093,7 +1071,6 @@ def __init__(
   class QAPISchemaEvent(QAPISchemaEntity):
       meta = 'event'

       def __init__(
           self,
           name: str,
           info: QAPISourceInfo,
           doc: Optional[QAPIDoc],
           ifcond: QAPISchemaIfCond,
           features: List[QAPISchemaFeature],
           arg_type: Optional[str],
           boxed: bool,
       ):
           super().__init__(name, info, doc, ifcond, features)
           assert not arg_type or isinstance(arg_type, str)
           self._arg_type_name = arg_type
           self.arg_type: Optional[QAPISchemaObjectType] = None
>          boxed: bool,
>      ):
>          super().__init__(name, info, doc, ifcond, features)
> -        assert not arg_type or isinstance(arg_type, str)

Yup, because arg_type: Optional[str].

>          self._arg_type_name = arg_type
>          self.arg_type: Optional[QAPISchemaObjectType] = None
>          self.boxed = boxed

Reviewed-by: Markus Armbruster <armbru@redhat.com>

[PATCH 19/19] qapi/schema: refactor entity lookup helpers

[John Snow]

This is not a clear win, but I was confused and I couldn't help myself.

Before:

lookup_entity(self, name: str, typ: Optional[type] = None
              ) -> Optional[QAPISchemaEntity]: ...

lookup_type(self, name: str) -> Optional[QAPISchemaType]: ...

resolve_type(self, name: str, info: Optional[QAPISourceInfo],
             what: Union[str, Callable[[Optional[QAPISourceInfo]], str]]
             ) -> QAPISchemaType: ...

After:

get_entity(self, name: str) -> Optional[QAPISchemaEntity]: ...
get_typed_entity(self, name: str, typ: Type[_EntityType]
                 ) -> Optional[_EntityType]: ...
lookup_type(self, name: str) -> QAPISchemaType: ...
resolve_type(self, name: str, info: Optional[QAPISourceInfo],
             what: Union[str, Callable[[Optional[QAPISourceInfo]], str]]
             ) -> QAPISchemaType: ...

In essence, any function that can return a None value becomes "get ..."
to encourage association with the dict.get() function which has the same
behavior. Any function named "lookup" or "resolve" by contrast is no
longer allowed to return a None value.

This means that any callers to resolve_type or lookup_type don't have to
check that the function worked, they can just assume it did.

Callers to resolve_type will be greeted with a QAPISemError if something
has gone wrong, as they have in the past. Callers to lookup_type will be
greeted with a KeyError if the entity does not exist, or a TypeError if
it does, but is the wrong type.

get_entity and get_typed_entity remain for any callers who are
specifically interested in the negative case. These functions have only
a single caller each.

Signed-off-by: John Snow <jsnow@redhat.com>
---
 docs/sphinx/qapidoc.py     |  2 +-
 scripts/qapi/introspect.py |  8 ++----
 scripts/qapi/schema.py     | 52 ++++++++++++++++++++++++--------------
 3 files changed, 36 insertions(+), 26 deletions(-)

diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index 8f3b9997a15..96deadbf7fc 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -508,7 +508,7 @@ def run(self):
             vis.visit_begin(schema)
             for doc in schema.docs:
                 if doc.symbol:
-                    vis.symbol(doc, schema.lookup_entity(doc.symbol))
+                    vis.symbol(doc, schema.get_entity(doc.symbol))
                 else:
                     vis.freeform(doc)
             return vis.get_document_nodes()
diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
index 42981bce163..67c7d89aae0 100644
--- a/scripts/qapi/introspect.py
+++ b/scripts/qapi/introspect.py
@@ -227,14 +227,10 @@ def _use_type(self, typ: QAPISchemaType) -> str:
 
         # Map the various integer types to plain int
         if typ.json_type() == 'int':
-            tmp = self._schema.lookup_type('int')
-            assert tmp is not None
-            typ = tmp
+            typ = self._schema.lookup_type('int')
         elif (isinstance(typ, QAPISchemaArrayType) and
               typ.element_type.json_type() == 'int'):
-            tmp = self._schema.lookup_type('intList')
-            assert tmp is not None
-            typ = tmp
+            typ = self._schema.lookup_type('intList')
         # Add type to work queue if new
         if typ not in self._used_types:
             self._used_types.append(typ)
diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
index b5f377e68b8..5813136e78b 100644
--- a/scripts/qapi/schema.py
+++ b/scripts/qapi/schema.py
@@ -26,6 +26,8 @@
     Dict,
     List,
     Optional,
+    Type,
+    TypeVar,
     Union,
     cast,
 )
@@ -767,7 +769,6 @@ def check(
             # Here we do:
             assert self.tag_member.defined_in
             base_type = schema.lookup_type(self.tag_member.defined_in)
-            assert base_type
             if not base_type.is_implicit():
                 base = "base type '%s'" % self.tag_member.defined_in
             if not isinstance(self.tag_member.type, QAPISchemaEnumType):
@@ -1111,6 +1112,12 @@ def visit(self, visitor: QAPISchemaVisitor) -> None:
             self.arg_type, self.boxed)
 
 
+# Used for type-dependent type lookup return values.
+_EntityType = TypeVar(   # pylint: disable=invalid-name
+    '_EntityType', bound=QAPISchemaEntity
+)
+
+
 class QAPISchema:
     def __init__(self, fname: str):
         self.fname = fname
@@ -1155,22 +1162,28 @@ def _def_entity(self, ent: QAPISchemaEntity) -> None:
                 ent.info, "%s is already defined" % other_ent.describe())
         self._entity_dict[ent.name] = ent
 
-    def lookup_entity(
+    def get_entity(self, name: str) -> Optional[QAPISchemaEntity]:
+        return self._entity_dict.get(name)
+
+    def get_typed_entity(
         self,
         name: str,
-        typ: Optional[type] = None,
-    ) -> Optional[QAPISchemaEntity]:
-        ent = self._entity_dict.get(name)
-        if typ and not isinstance(ent, typ):
-            return None
+        typ: Type[_EntityType]
+    ) -> Optional[_EntityType]:
+        ent = self.get_entity(name)
+        if ent is not None and not isinstance(ent, typ):
+            etype = type(ent).__name__
+            ttype = typ.__name__
+            raise TypeError(
+                f"Entity '{name}' is of type '{etype}', not '{ttype}'."
+            )
         return ent
 
-    def lookup_type(self, name: str) -> Optional[QAPISchemaType]:
-        typ = self.lookup_entity(name, QAPISchemaType)
-        if typ is None:
-            return None
-        assert isinstance(typ, QAPISchemaType)
-        return typ
+    def lookup_type(self, name: str) -> QAPISchemaType:
+        ent = self.get_typed_entity(name, QAPISchemaType)
+        if ent is None:
+            raise KeyError(f"Entity '{name}' is not defined.")
+        return ent
 
     def resolve_type(
         self,
@@ -1178,13 +1191,14 @@ def resolve_type(
         info: Optional[QAPISourceInfo],
         what: Union[str, Callable[[Optional[QAPISourceInfo]], str]],
     ) -> QAPISchemaType:
-        typ = self.lookup_type(name)
-        if not typ:
+        try:
+            return self.lookup_type(name)
+        except (KeyError, TypeError) as err:
             if callable(what):
                 what = what(info)
             raise QAPISemError(
-                info, "%s uses unknown type '%s'" % (what, name))
-        return typ
+                info, "%s uses unknown type '%s'" % (what, name)
+            ) from err
 
     def _module_name(self, fname: str) -> str:
         if QAPISchemaModule.is_system_module(fname):
@@ -1279,7 +1293,7 @@ def _make_array_type(
         self, element_type: str, info: Optional[QAPISourceInfo]
     ) -> str:
         name = element_type + 'List'    # reserved by check_defn_name_str()
-        if not self.lookup_type(name):
+        if not self.get_entity(name):
             self._def_entity(QAPISchemaArrayType(name, info, element_type))
         return name
 
@@ -1295,7 +1309,7 @@ def _make_implicit_object_type(
             return None
         # See also QAPISchemaObjectTypeMember.describe()
         name = 'q_obj_%s-%s' % (name, role)
-        typ = self.lookup_entity(name, QAPISchemaObjectType)
+        typ = self.get_typed_entity(name, QAPISchemaObjectType)
         if typ:
             # The implicit object type has multiple users.  This can
             # only be a duplicate definition, which will be flagged
-- 
2.41.0

Re: [PATCH 19/19] qapi/schema: refactor entity lookup helpers

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> This is not a clear win, but I was confused and I couldn't help myself.
>
> Before:
>
> lookup_entity(self, name: str, typ: Optional[type] = None
>               ) -> Optional[QAPISchemaEntity]: ...
>
> lookup_type(self, name: str) -> Optional[QAPISchemaType]: ...
>
> resolve_type(self, name: str, info: Optional[QAPISourceInfo],
>              what: Union[str, Callable[[Optional[QAPISourceInfo]], str]]
>              ) -> QAPISchemaType: ...
>
> After:
>
> get_entity(self, name: str) -> Optional[QAPISchemaEntity]: ...
> get_typed_entity(self, name: str, typ: Type[_EntityType]
>                  ) -> Optional[_EntityType]: ...
> lookup_type(self, name: str) -> QAPISchemaType: ...
> resolve_type(self, name: str, info: Optional[QAPISourceInfo],
>              what: Union[str, Callable[[Optional[QAPISourceInfo]], str]]
>              ) -> QAPISchemaType: ...

.resolve_type()'s type remains the same.

> In essence, any function that can return a None value becomes "get ..."
> to encourage association with the dict.get() function which has the same
> behavior. Any function named "lookup" or "resolve" by contrast is no
> longer allowed to return a None value.

.resolve_type() doesn't before the patch.

> This means that any callers to resolve_type or lookup_type don't have to
> check that the function worked, they can just assume it did.
>
> Callers to resolve_type will be greeted with a QAPISemError if something
> has gone wrong, as they have in the past. Callers to lookup_type will be
> greeted with a KeyError if the entity does not exist, or a TypeError if
> it does, but is the wrong type.

Talking about .resolve_type() so much suggests you're changing it.
You're not.

Here's my own summary of the change, just to make sure I got it:

1. Split .lookup_entity() into .get_entity() and .get_typed_entity().

   schema.lookup_entity(name) and schema.lookup_entity(name, None)
   become schema.get_entity(name).

   schema.lookup_entity(name, typ) where typ is not None becomes
   schema.get_typed_entity().

2. Tighten .get_typed_entity()'s type from Optional[QAPISchemaEntity] to
   Optional[_EntityType], where Entity is argument @typ.

3. Change .lookup_type()'s behavior for "not found" from "return None"
   to "throw KeyError if doesn't exist, throw TypeError if exists, but
   not a type".

Correct?

> get_entity and get_typed_entity remain for any callers who are
> specifically interested in the negative case. These functions have only
> a single caller each.

.get_entity()'s single caller being QAPIDocDirective.run(), and its
other single caller being QAPISchema._make_implicit_object_type() ;-P

> Signed-off-by: John Snow <jsnow@redhat.com>
> ---
>  docs/sphinx/qapidoc.py     |  2 +-
>  scripts/qapi/introspect.py |  8 ++----
>  scripts/qapi/schema.py     | 52 ++++++++++++++++++++++++--------------
>  3 files changed, 36 insertions(+), 26 deletions(-)
>
> diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
> index 8f3b9997a15..96deadbf7fc 100644
> --- a/docs/sphinx/qapidoc.py
> +++ b/docs/sphinx/qapidoc.py
> @@ -508,7 +508,7 @@ def run(self):
               vis = QAPISchemaGenRSTVisitor(self)
>              vis.visit_begin(schema)
>              for doc in schema.docs:
>                  if doc.symbol:
> -                    vis.symbol(doc, schema.lookup_entity(doc.symbol))
> +                    vis.symbol(doc, schema.get_entity(doc.symbol))

@vis is a QAPISchemaGenRSTVisitor, and vis.symbol is

    def symbol(self, doc, entity):
        [...]
        self._cur_doc = doc
        entity.visit(self)
        self._cur_doc = None

When you add type hints to qapidoc.py, parameter @entity will be
QAPISchemaEntity.  Type error since .get_entity() returns
Optional[QAPISchemaEntity].  I'm fine with addressing that when adding
types to qapidoc.py.

>                  else:
>                      vis.freeform(doc)
>              return vis.get_document_nodes()
> diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py
> index 42981bce163..67c7d89aae0 100644
> --- a/scripts/qapi/introspect.py
> +++ b/scripts/qapi/introspect.py
> @@ -227,14 +227,10 @@ def _use_type(self, typ: QAPISchemaType) -> str:
>  
>          # Map the various integer types to plain int
>          if typ.json_type() == 'int':
> -            tmp = self._schema.lookup_type('int')
> -            assert tmp is not None
> -            typ = tmp
> +            typ = self._schema.lookup_type('int')
>          elif (isinstance(typ, QAPISchemaArrayType) and
>                typ.element_type.json_type() == 'int'):
> -            tmp = self._schema.lookup_type('intList')
> -            assert tmp is not None
> -            typ = tmp
> +            typ = self._schema.lookup_type('intList')
>          # Add type to work queue if new
>          if typ not in self._used_types:
>              self._used_types.append(typ)

Readability improvement here, due to tighter typing of .lookup_type():
it now returns QAPISchemaType instead of Optional[QAPISchemaType].

Before, lookup failure results in AssertionError.  Afterwards, it
results in KeyError or TypeError.  Fine.  Is it worth mentioning in the
commit message?  Genuine question!

> diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
> index b5f377e68b8..5813136e78b 100644
> --- a/scripts/qapi/schema.py
> +++ b/scripts/qapi/schema.py
> @@ -26,6 +26,8 @@
>      Dict,
>      List,
>      Optional,
> +    Type,
> +    TypeVar,
>      Union,
>      cast,
>  )
> @@ -767,7 +769,6 @@ def check(
>              # Here we do:
>              assert self.tag_member.defined_in
>              base_type = schema.lookup_type(self.tag_member.defined_in)
> -            assert base_type

Same change of errors as above.

>              if not base_type.is_implicit():
>                  base = "base type '%s'" % self.tag_member.defined_in
>              if not isinstance(self.tag_member.type, QAPISchemaEnumType):
> @@ -1111,6 +1112,12 @@ def visit(self, visitor: QAPISchemaVisitor) -> None:
>              self.arg_type, self.boxed)
>  
>  
> +# Used for type-dependent type lookup return values.
> +_EntityType = TypeVar(   # pylint: disable=invalid-name
> +    '_EntityType', bound=QAPISchemaEntity
> +)

Oh, the fanciness!

> +
> +
>  class QAPISchema:
>      def __init__(self, fname: str):
>          self.fname = fname
> @@ -1155,22 +1162,28 @@ def _def_entity(self, ent: QAPISchemaEntity) -> None:
>                  ent.info, "%s is already defined" % other_ent.describe())
>          self._entity_dict[ent.name] = ent
>  
> -    def lookup_entity(
> +    def get_entity(self, name: str) -> Optional[QAPISchemaEntity]:
> +        return self._entity_dict.get(name)
> +
> +    def get_typed_entity(
>          self,
>          name: str,
> -        typ: Optional[type] = None,
> -    ) -> Optional[QAPISchemaEntity]:
> -        ent = self._entity_dict.get(name)
> -        if typ and not isinstance(ent, typ):
> -            return None
> +        typ: Type[_EntityType]
> +    ) -> Optional[_EntityType]:
> +        ent = self.get_entity(name)
> +        if ent is not None and not isinstance(ent, typ):
> +            etype = type(ent).__name__
> +            ttype = typ.__name__
> +            raise TypeError(
> +                f"Entity '{name}' is of type '{etype}', not '{ttype}'."
> +            )
>          return ent
>  
> -    def lookup_type(self, name: str) -> Optional[QAPISchemaType]:
> -        typ = self.lookup_entity(name, QAPISchemaType)
> -        if typ is None:
> -            return None
> -        assert isinstance(typ, QAPISchemaType)
> -        return typ
> +    def lookup_type(self, name: str) -> QAPISchemaType:
> +        ent = self.get_typed_entity(name, QAPISchemaType)
> +        if ent is None:
> +            raise KeyError(f"Entity '{name}' is not defined.")
> +        return ent
>  
>      def resolve_type(
>          self,
> @@ -1178,13 +1191,14 @@ def resolve_type(
>          info: Optional[QAPISourceInfo],
>          what: Union[str, Callable[[Optional[QAPISourceInfo]], str]],
>      ) -> QAPISchemaType:
> -        typ = self.lookup_type(name)
> -        if not typ:
> +        try:
> +            return self.lookup_type(name)
> +        except (KeyError, TypeError) as err:
>              if callable(what):
>                  what = what(info)
>              raise QAPISemError(
> -                info, "%s uses unknown type '%s'" % (what, name))
> -        return typ
> +                info, "%s uses unknown type '%s'" % (what, name)
> +            ) from err

This is at best a wash for readability.

When something throws KeyError or TypeError unexpectedly, we
misinterpret the programming error as a semantic error in the schema.

>  
>      def _module_name(self, fname: str) -> str:
>          if QAPISchemaModule.is_system_module(fname):
> @@ -1279,7 +1293,7 @@ def _make_array_type(
>          self, element_type: str, info: Optional[QAPISourceInfo]
>      ) -> str:
>          name = element_type + 'List'    # reserved by check_defn_name_str()
> -        if not self.lookup_type(name):
> +        if not self.get_entity(name):
>              self._def_entity(QAPISchemaArrayType(name, info, element_type))
>          return name
>  
> @@ -1295,7 +1309,7 @@ def _make_implicit_object_type(
>              return None
>          # See also QAPISchemaObjectTypeMember.describe()
>          name = 'q_obj_%s-%s' % (name, role)
> -        typ = self.lookup_entity(name, QAPISchemaObjectType)
> +        typ = self.get_typed_entity(name, QAPISchemaObjectType)
>          if typ:
>              # The implicit object type has multiple users.  This can
>              # only be a duplicate definition, which will be flagged