Re: [PATCH v2 4/4] qapi: expose all schema features to code

[Markus Armbruster <armbru@redhat.com>]

Daniel P. Berrangé <berrange@redhat.com> writes:

> On Thu, Nov 14, 2024 at 01:48:28PM +0100, Markus Armbruster wrote:
>> Daniel P. Berrangé <berrange@redhat.com> writes:
>> 
>> > This replaces use of the constants from the QapiSpecialFeatures
>> > enum, with constants from the auto-generate QapiFeatures enum
>> > in qapi-features.h
>> >
>> > The 'deprecated' and 'unstable' features still have a little bit of
>> > special handling, being force defined to be the 1st + 2nd features
>> > in the enum, regardless of whether they're used in the schema. This
>> > retains compatibility with common code that references the features
>> > via the QapiSpecialFeatures constants.
>> >
>> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>> > ---
>> >  meson.build              |  1 +
>> >  scripts/qapi/commands.py |  1 +
>> >  scripts/qapi/features.py | 62 ++++++++++++++++++++++++++++++++++++++++
>> >  scripts/qapi/gen.py      |  4 +--
>> >  scripts/qapi/main.py     |  2 ++
>> >  scripts/qapi/schema.py   | 19 +++++++++++-
>> >  scripts/qapi/types.py    |  6 ++--
>> >  scripts/qapi/visit.py    |  3 +-
>> >  8 files changed, 92 insertions(+), 6 deletions(-)
>> >  create mode 100644 scripts/qapi/features.py
>> >
>> > diff --git a/meson.build b/meson.build
>> > index d26690ce20..b9d58be66f 100644
>> > --- a/meson.build
>> > +++ b/meson.build
>> > @@ -3332,6 +3332,7 @@ qapi_gen_depends = [ meson.current_source_dir() / 'scripts/qapi/__init__.py',
>> >                       meson.current_source_dir() / 'scripts/qapi/schema.py',
>> >                       meson.current_source_dir() / 'scripts/qapi/source.py',
>> >                       meson.current_source_dir() / 'scripts/qapi/types.py',
>> > +                     meson.current_source_dir() / 'scripts/qapi/features.py',
>> >                       meson.current_source_dir() / 'scripts/qapi/visit.py',
>> >                       meson.current_source_dir() / 'scripts/qapi-gen.py'
>> >  ]
>> > diff --git a/scripts/qapi/commands.py b/scripts/qapi/commands.py
>> > index d629d2d97e..bf88bfc442 100644
>> > --- a/scripts/qapi/commands.py
>> > +++ b/scripts/qapi/commands.py
>> > @@ -355,6 +355,7 @@ def visit_begin(self, schema: QAPISchema) -> None:
>> >  #include "qemu/osdep.h"
>> >  #include "%(prefix)sqapi-commands.h"
>> >  #include "%(prefix)sqapi-init-commands.h"
>> > +#include "%(prefix)sqapi-features.h"
>> >  
>> >  void %(c_prefix)sqmp_init_marshal(QmpCommandList *cmds)
>> >  {
>> > diff --git a/scripts/qapi/features.py b/scripts/qapi/features.py
>> > new file mode 100644
>> > index 0000000000..dc10c7cea0
>> > --- /dev/null
>> > +++ b/scripts/qapi/features.py
>> > @@ -0,0 +1,62 @@
>> > +"""
>> > +QAPI types generator
>> 
>> QAPI features generator
>> 
>> > +
>> > +Copyright 2024 Red Hat
>> > +
>> > +This work is licensed under the terms of the GNU GPL, version 2.
>> > +# See the COPYING file in the top-level directory.
>> > +"""
>> > +
>> > +from typing import List, Optional
>> 
>> pylint gripes
>
> Sigh, I really wish we had pylint/mypy/pycodestyle being run as
> part of 'make check' by default. I don't like making the mistake
> of sending patches which fail extra non-default checks maintainers
> need compliance with.

We've made some progress towards running these checkers, but we're not
there, yet.

>>     scripts/qapi/features.py:10:0: W0611: Unused List imported from typing (unused-
>>     import)
>>     scripts/qapi/features.py:10:0: W0611: Unused Optional imported from typing (unused-import)
>> 
>> > +
>> > +from .common import c_enum_const, mcgen, c_name
>> 
>>     scripts/qapi/features.py:12:0: W0611: Unused mcgen imported from common (unused-import)
>> 
>> > +from .gen import QAPISchemaMonolithicCVisitor
>> > +from .schema import (
>> > +    QAPISchema,
>> > +    QAPISchemaFeature,
>> > +)
>> > +from .source import QAPISourceInfo
>> 
>>     scripts/qapi/features.py:18:0: W0611: Unused QAPISourceInfo imported from source (unused-import)
>> 
>> > +
>> > +
>> > +class QAPISchemaGenFeatureVisitor(QAPISchemaMonolithicCVisitor):
>> > +
>> > +    def __init__(self, prefix: str):
>> > +        super().__init__(
>> > +            prefix, 'qapi-features',
>> > +            ' * Schema-defined QAPI features',
>> > +            __doc__)
>> > +
>> > +        self.features = {}
>> 
>> mypy gripes
>> 
>>     scripts/qapi/features.py:29: error: Need type annotation for "features" (hint: "features: Dict[<type>, <type>] = ...")  [var-annotated]
>> 
>> Elsewhere, we avoid rummaging in QAPISchema's innards by defining
>> suitable visit.  If that's too much trouble for you now, I can take this
>> as is an clean up on top.
>> 
>> > +
>> > +    def visit_begin(self, schema: QAPISchema):
>> 
>> mypy gripes
>> 
>>     scripts/qapi/features.py:31: error: Function is missing a return type annotation  [no-untyped-def]
>> 
>> > +        self.features = schema._feature_dict
>> 
>> pylint gripes
>> 
>>     scripts/qapi/features.py:32:24: W0212: Access to a protected member _feature_dict of a client class (protected-access)
>> 
>> > +
>> > +    def visit_end(self) -> None:
>> > +        features = [
>> > +            self.features[f]
>> > +            for f in QAPISchemaFeature.SPECIAL_NAMES
>> > +        ]
>> > +
>> > +        features.extend(
>> > +            sorted(
>> > +                filter(lambda f: not f.is_special(),
>> > +                       self.features.values()),
>> > +                key=lambda f: f.name)
>> > +        )
>> 
>> @features is schema._feature_dict.values() sorted by name in a certain
>> way, namely first the .SPECIAL_NAMES in order, then all the others in
>> alphabetical order.
>> 
>> Why you do this is not immediately obvious.  I guess it's to make the
>> generated enum a compatible extension of enum QapiSpecialFeature.  That
>> one exists for use by schema-independent support code such
>> compat_policy_input_ok() and qobject_output_policy_skip().
>
> Yes, I wanted the overlapping enums vaules to match.

Needs a comment.

>> I further guess you sort the non-special features just to make the
>> generated code easier for humans to navigate.
>> 
>> Correct?
>
> The remaining sort was just to give a predictable stable output,
> should QAPI usage of features be reordered.

We don't do that for enum QAPIEvent, and it hasn't inconvenienced us as
far as I can tell.  No big deal, I just like consistency.

>> > +
>> > +        self._genh.add("typedef enum {\n")
>> > +        for f in features:
>> > +            self._genh.add(f"    {c_enum_const('qapi_feature', f.name)}")
>> > +            if f.name in QAPISchemaFeature.SPECIAL_NAMES:
>> > +                self._genh.add(f" = {c_enum_const('qapi', f.name)},\n" )
>> 
>> pycodestyle gripes
>> 
>>     scripts/qapi/features.py:51:71: E202 whitespace before ')'
>> 
>> > +            else:
>> > +                self._genh.add(",\n")
>> > +
>> > +        self._genh.add("} " + c_name('QapiFeature') + ";\n")
>> > +
>> 
>> pycodestyle gripes
>> 
>>     scripts/qapi/features.py:57:1: E302 expected 2 blank lines, found 1
>> 
>> This part generates a C enum.  It's similar to gen_enum() from types.py,
>> except we work with a list of QAPISchemaFeature here, and a list of
>> QAPISchemaEnumMember there.
>> 
>> To reuse gen_enum() here, we'd have to make up a member list, like we do
>> in events.py for enum QAPIEvent.
>
> I'll have a look at that.

Reuse it only if it's easy for you.  We can always improve on top.

>> > diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py
>> > index e97c978d38..5e14b1829b 100644
>> > --- a/scripts/qapi/schema.py
>> > +++ b/scripts/qapi/schema.py
>> > @@ -933,8 +933,11 @@ def connect_doc(self, doc: Optional[QAPIDoc]) -> None:
>> >  class QAPISchemaFeature(QAPISchemaMember):
>> >      role = 'feature'
>> >  
>> > +    # Features which are standardized across all schemas
>> > +    SPECIAL_NAMES = ['deprecated', 'unstable']
>> > +
>> >      def is_special(self) -> bool:
>> > -        return self.name in ('deprecated', 'unstable')
>> > +        return self.name in QAPISchemaFeature.SPECIAL_NAMES
>> >  
>> >  
>> >  class QAPISchemaObjectTypeMember(QAPISchemaMember):
>> > @@ -1138,6 +1141,11 @@ def __init__(self, fname: str):
>> >          self._entity_list: List[QAPISchemaEntity] = []
>> >          self._entity_dict: Dict[str, QAPISchemaDefinition] = {}
>> >          self._module_dict: Dict[str, QAPISchemaModule] = OrderedDict()
>> > +        self._feature_dict: Dict[str, QAPISchemaFeature] = {}
>> > +
>> > +        for f in QAPISchemaFeature.SPECIAL_NAMES:
>> > +            self._feature_dict[f] = QAPISchemaFeature(f, "special feature")
>> 
>> mypy gripes
>> 
>>     scripts/qapi/schema.py:1147: error: Argument 2 to "QAPISchemaFeature" has incompatible type "str"; expected "Optional[QAPISourceInfo]"  [arg-type]
>> 
>> We commonly use None as info value for built-in stuff, and that's why
>> it's Optional[QAPISourceInfo], not just QAPISourceInfo.
>
> Yeah, not sure what I was thinking here, looking again I
> should have passed "None"
>
>> But do we really need to make up some QAPISchemaFeature?  Hmm.  The
>> appended patch dumbs down ._feature_dict to a set.

See below for a possible reason to keep .feature_dict.

> I was following the same pattern as self._entity_dict and
> self._module_dict, rather than dumbing down to the bare
> minimum needed by my current use case. I don't mind which
> strategy we take.

.entity_dict maps the name to the one entity.  Likewise .module_dict.
.feature_dict, however, maps it to the first of possibly many.  That's
not wrong, just peculiar and possibly less than obvious to readers who
aren't familiar with how we represent features internally.  Worth a
comment?

>> > +
>> >          self._schema_dir = os.path.dirname(fname)
>> >          self._make_module(QAPISchemaModule.BUILTIN_MODULE_NAME)
>> >          self._make_module(fname)
>> > @@ -1258,6 +1266,15 @@ def _make_features(
>> >      ) -> List[QAPISchemaFeature]:
>> >          if features is None:
>> >              return []
>> > +
>> > +        for f in features:
>> > +            feat = QAPISchemaFeature(f['name'], info)
>> > +            if feat.name not in self._feature_dict:
>> > +                if len(self._feature_dict) == 64:
>> > +                    raise Exception("Maximum of 64 schema features is permitted")
>> 
>> The limit is an implementation restriction.  Okay, we can lift it when
>> it bites us.
>> 
>> However, the reporting is less than nice:
>> 
>>     $ python scripts/qapi-gen.py -o $$ tests/qapi-schema/features-too-many.json 
>>     Traceback (most recent call last):
>>       File "/work/armbru/qemu/scripts/qapi-gen.py", line 19, in <module>
>>         sys.exit(main.main())
>>                  ^^^^^^^^^^^
>>       File "/work/armbru/qemu/scripts/qapi/main.py", line 96, in main
>>         generate(args.schema,
>>       File "/work/armbru/qemu/scripts/qapi/main.py", line 51, in generate
>>         schema = QAPISchema(schema_file)
>>                  ^^^^^^^^^^^^^^^^^^^^^^^
>>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 1155, in __init__
>>         self._def_exprs(exprs)
>>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 1482, in _def_exprs
>>         self._def_struct_type(expr)
>>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 1377, in _def_struct_type
>>         features = self._make_features(expr.get('features'), info)
>>                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>       File "/work/armbru/qemu/scripts/qapi/schema.py", line 1274, in _make_features
>>         raise Exception("Maximum of 64 schema features is permitted")
>>     Exception: Maximum of 64 schema features is permitted
>
> Is there any better way to approach this error reporting ?

Raise QAPISemError in .check().

Hmm, then you need a QAPISourceInfo to pass.  .feature_dict will give
you one: the .info of the feature added last.