From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0068C433E0 for ; Fri, 19 Feb 2021 15:38:00 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 52D3264E60 for ; Fri, 19 Feb 2021 15:38:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 52D3264E60 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Received: from localhost ([::1]:47156 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1lD7qx-0006C5-Du for qemu-devel@archiver.kernel.org; Fri, 19 Feb 2021 10:37:59 -0500 Received: from eggs.gnu.org ([2001:470:142:3::10]:35342) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lD76c-00056C-F6 for qemu-devel@nongnu.org; Fri, 19 Feb 2021 09:50:06 -0500 Received: from us-smtp-delivery-124.mimecast.com ([63.128.21.124]:31791) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.90_1) (envelope-from ) id 1lD76U-0004mP-Sb for qemu-devel@nongnu.org; Fri, 19 Feb 2021 09:50:06 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1613746198; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=QIR+F/RXcDUT3PnedrRlQ10I5Cz+44aTve4QC0Z4tr8=; b=FrqD60jmoI9WL1t069pQ8ho6OuXREd/rr2OCglmJLPvxXKnV8hcA/qDKhaqTsmaBDIhNxl /WHdrDf9z6gD/rAbfEUZzfPeb7HqcLNPZKsJV329pYcDI6Kt5ulAfyiPtheO+1Ljj2x+rV zBHj0Z2sYjELZwEddNDLBqTC6luSFA0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-318-XDIwwFpIMEOYxRG-HMSylA-1; Fri, 19 Feb 2021 09:49:43 -0500 X-MC-Unique: XDIwwFpIMEOYxRG-HMSylA-1 Received: from smtp.corp.redhat.com (int-mx01.intmail.prod.int.phx2.redhat.com [10.5.11.11]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id E399D1935780; Fri, 19 Feb 2021 14:49:42 +0000 (UTC) Received: from blackfin.pond.sub.org (ovpn-115-79.ams2.redhat.com [10.36.115.79]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 680A318B4B; Fri, 19 Feb 2021 14:49:42 +0000 (UTC) Received: by blackfin.pond.sub.org (Postfix, from userid 1000) id 7A5A4112F05F; Fri, 19 Feb 2021 15:49:39 +0100 (CET) From: Markus Armbruster To: qemu-devel@nongnu.org Subject: [PULL 14/18] qapi/introspect.py: add type hint annotations Date: Fri, 19 Feb 2021 15:49:35 +0100 Message-Id: <20210219144939.604488-15-armbru@redhat.com> In-Reply-To: <20210219144939.604488-1-armbru@redhat.com> References: <20210219144939.604488-1-armbru@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.11 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=armbru@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII" Received-SPF: pass client-ip=63.128.21.124; envelope-from=armbru@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: peter.maydell@linaro.org, John Snow Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" From: John Snow NB: The type aliases (SchemaInfo et al) declare intent for some of the "dictly-typed" objects we pass around in introspect.py. They do not enforce the shape of those objects, and cannot, until Python 3.7 or later. (And even then, it may not be "worth it".) Annotations are also added to the QAPISchemaEntity __init__ method in schema.py to allow mypy to statically prove the type of typ.name, needed to prove the return type of QAPISchemaGenIntrospectVisitor._use_type(). Signed-off-by: John Snow Message-Id: <20210216021809.134886-15-jsnow@redhat.com> Reviewed-by: Markus Armbruster [Note on QAPISchemaEntity.__init__() squashed into commit message, Comment wrapped to conform to PEP 8] Signed-off-by: Markus Armbruster --- scripts/qapi/introspect.py | 125 +++++++++++++++++++++++++++---------- scripts/qapi/mypy.ini | 5 -- scripts/qapi/schema.py | 2 +- 3 files changed, 93 insertions(+), 39 deletions(-) diff --git a/scripts/qapi/introspect.py b/scripts/qapi/introspect.py index da7bc8883c..05c1a196e9 100644 --- a/scripts/qapi/introspect.py +++ b/scripts/qapi/introspect.py @@ -17,6 +17,7 @@ from typing import ( Iterable, List, Optional, + Sequence, Tuple, TypeVar, Union, @@ -30,10 +31,19 @@ from .common import ( ) from .gen import QAPISchemaMonolithicCVisitor from .schema import ( + QAPISchema, QAPISchemaArrayType, QAPISchemaBuiltinType, + QAPISchemaEntity, + QAPISchemaEnumMember, + QAPISchemaFeature, + QAPISchemaObjectType, + QAPISchemaObjectTypeMember, QAPISchemaType, + QAPISchemaVariant, + QAPISchemaVariants, ) +from .source import QAPISourceInfo # This module constructs a tree data structure that is used to @@ -58,6 +68,16 @@ _NonScalar = Union[Dict[str, _Stub], List[_Stub]] _Value = Union[_Scalar, _NonScalar] JSONValue = Union[_Value, 'Annotated[_Value]'] +# These types are based on structures defined in QEMU's schema, so we +# lack precise types for them here. Python 3.6 does not offer +# TypedDict constructs, so they are broadly typed here as simple +# Python Dicts. +SchemaInfo = Dict[str, object] +SchemaInfoObject = Dict[str, object] +SchemaInfoObjectVariant = Dict[str, object] +SchemaInfoObjectMember = Dict[str, object] +SchemaInfoCommand = Dict[str, object] + _ValueT = TypeVar('_ValueT', bound=_Value) @@ -77,9 +97,11 @@ class Annotated(Generic[_ValueT]): self.ifcond: Tuple[str, ...] = tuple(ifcond) -def _tree_to_qlit(obj, level=0, dict_value=False): +def _tree_to_qlit(obj: JSONValue, + level: int = 0, + dict_value: bool = False) -> str: - def indent(level): + def indent(level: int) -> str: return level * 4 * ' ' if isinstance(obj, Annotated): @@ -137,21 +159,21 @@ def _tree_to_qlit(obj, level=0, dict_value=False): return ret -def to_c_string(string): +def to_c_string(string: str) -> str: return '"' + string.replace('\\', r'\\').replace('"', r'\"') + '"' class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor): - def __init__(self, prefix, unmask): + def __init__(self, prefix: str, unmask: bool): super().__init__( prefix, 'qapi-introspect', ' * QAPI/QMP schema introspection', __doc__) self._unmask = unmask - self._schema = None - self._trees = [] - self._used_types = [] - self._name_map = {} + self._schema: Optional[QAPISchema] = None + self._trees: List[Annotated[SchemaInfo]] = [] + self._used_types: List[QAPISchemaType] = [] + self._name_map: Dict[str, str] = {} self._genc.add(mcgen(''' #include "qemu/osdep.h" #include "%(prefix)sqapi-introspect.h" @@ -159,10 +181,10 @@ class QAPISchemaGenIntrospectVisitor(QAPISchemaMonolithicCVisitor): ''', prefix=prefix)) - def visit_begin(self, schema): + def visit_begin(self, schema: QAPISchema) -> None: self._schema = schema - def visit_end(self): + def visit_end(self) -> None: # visit the types that are actually used for typ in self._used_types: typ.visit(self) @@ -184,18 +206,18 @@ const QLitObject %(c_name)s = %(c_string)s; self._used_types = [] self._name_map = {} - def visit_needed(self, entity): + def visit_needed(self, entity: QAPISchemaEntity) -> bool: # Ignore types on first pass; visit_end() will pick up used types return not isinstance(entity, QAPISchemaType) - def _name(self, name): + def _name(self, name: str) -> str: if self._unmask: return name if name not in self._name_map: self._name_map[name] = '%d' % len(self._name_map) return self._name_map[name] - def _use_type(self, typ): + def _use_type(self, typ: QAPISchemaType) -> str: assert self._schema is not None # Map the various integer types to plain int @@ -217,10 +239,13 @@ const QLitObject %(c_name)s = %(c_string)s; return self._name(typ.name) @staticmethod - def _gen_features(features): + def _gen_features(features: List[QAPISchemaFeature] + ) -> List[Annotated[str]]: return [Annotated(f.name, f.ifcond) for f in features] - def _gen_tree(self, name, mtype, obj, ifcond, features): + def _gen_tree(self, name: str, mtype: str, obj: Dict[str, object], + ifcond: Sequence[str], + features: Optional[List[QAPISchemaFeature]]) -> None: comment: Optional[str] = None if mtype not in ('command', 'event', 'builtin', 'array'): if not self._unmask: @@ -234,42 +259,65 @@ const QLitObject %(c_name)s = %(c_string)s; obj['features'] = self._gen_features(features) self._trees.append(Annotated(obj, ifcond, comment)) - def _gen_member(self, member): - obj = {'name': member.name, 'type': self._use_type(member.type)} + def _gen_member(self, member: QAPISchemaObjectTypeMember + ) -> Annotated[SchemaInfoObjectMember]: + obj: SchemaInfoObjectMember = { + 'name': member.name, + 'type': self._use_type(member.type) + } if member.optional: obj['default'] = None if member.features: obj['features'] = self._gen_features(member.features) return Annotated(obj, member.ifcond) - def _gen_variant(self, variant): - obj = {'case': variant.name, 'type': self._use_type(variant.type)} + def _gen_variant(self, variant: QAPISchemaVariant + ) -> Annotated[SchemaInfoObjectVariant]: + obj: SchemaInfoObjectVariant = { + 'case': variant.name, + 'type': self._use_type(variant.type) + } return Annotated(obj, variant.ifcond) - def visit_builtin_type(self, name, info, json_type): + def visit_builtin_type(self, name: str, info: Optional[QAPISourceInfo], + json_type: str) -> None: self._gen_tree(name, 'builtin', {'json-type': json_type}, [], None) - def visit_enum_type(self, name, info, ifcond, features, members, prefix): + def visit_enum_type(self, name: str, info: Optional[QAPISourceInfo], + ifcond: Sequence[str], + features: List[QAPISchemaFeature], + members: List[QAPISchemaEnumMember], + prefix: Optional[str]) -> None: self._gen_tree( name, 'enum', {'values': [Annotated(m.name, m.ifcond) for m in members]}, ifcond, features ) - def visit_array_type(self, name, info, ifcond, element_type): + def visit_array_type(self, name: str, info: Optional[QAPISourceInfo], + ifcond: Sequence[str], + element_type: QAPISchemaType) -> None: element = self._use_type(element_type) self._gen_tree('[' + element + ']', 'array', {'element-type': element}, ifcond, None) - def visit_object_type_flat(self, name, info, ifcond, features, - members, variants): - obj = {'members': [self._gen_member(m) for m in members]} + def visit_object_type_flat(self, name: str, info: Optional[QAPISourceInfo], + ifcond: Sequence[str], + features: List[QAPISchemaFeature], + members: List[QAPISchemaObjectTypeMember], + variants: Optional[QAPISchemaVariants]) -> None: + obj: SchemaInfoObject = { + 'members': [self._gen_member(m) for m in members] + } if variants: obj['tag'] = variants.tag_member.name obj['variants'] = [self._gen_variant(v) for v in variants.variants] self._gen_tree(name, 'object', obj, ifcond, features) - def visit_alternate_type(self, name, info, ifcond, features, variants): + def visit_alternate_type(self, name: str, info: Optional[QAPISourceInfo], + ifcond: Sequence[str], + features: List[QAPISchemaFeature], + variants: QAPISchemaVariants) -> None: self._gen_tree( name, 'alternate', {'members': [Annotated({'type': self._use_type(m.type)}, @@ -278,27 +326,38 @@ const QLitObject %(c_name)s = %(c_string)s; ifcond, features ) - def visit_command(self, name, info, ifcond, features, - arg_type, ret_type, gen, success_response, boxed, - allow_oob, allow_preconfig, coroutine): + def visit_command(self, name: str, info: Optional[QAPISourceInfo], + ifcond: Sequence[str], + features: List[QAPISchemaFeature], + arg_type: Optional[QAPISchemaObjectType], + ret_type: Optional[QAPISchemaType], gen: bool, + success_response: bool, boxed: bool, allow_oob: bool, + allow_preconfig: bool, coroutine: bool) -> None: assert self._schema is not None arg_type = arg_type or self._schema.the_empty_object_type ret_type = ret_type or self._schema.the_empty_object_type - obj = {'arg-type': self._use_type(arg_type), - 'ret-type': self._use_type(ret_type)} + obj: SchemaInfoCommand = { + 'arg-type': self._use_type(arg_type), + 'ret-type': self._use_type(ret_type) + } if allow_oob: obj['allow-oob'] = allow_oob self._gen_tree(name, 'command', obj, ifcond, features) - def visit_event(self, name, info, ifcond, features, arg_type, boxed): + def visit_event(self, name: str, info: Optional[QAPISourceInfo], + ifcond: Sequence[str], features: List[QAPISchemaFeature], + arg_type: Optional[QAPISchemaObjectType], + boxed: bool) -> None: assert self._schema is not None + arg_type = arg_type or self._schema.the_empty_object_type self._gen_tree(name, 'event', {'arg-type': self._use_type(arg_type)}, ifcond, features) -def gen_introspect(schema, output_dir, prefix, opt_unmask): +def gen_introspect(schema: QAPISchema, output_dir: str, prefix: str, + opt_unmask: bool) -> None: vis = QAPISchemaGenIntrospectVisitor(prefix, opt_unmask) schema.visit(vis) vis.write(output_dir) diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini index 04bd5db527..0a000d58b3 100644 --- a/scripts/qapi/mypy.ini +++ b/scripts/qapi/mypy.ini @@ -13,11 +13,6 @@ disallow_untyped_defs = False disallow_incomplete_defs = False check_untyped_defs = False -[mypy-qapi.introspect] -disallow_untyped_defs = False -disallow_incomplete_defs = False -check_untyped_defs = False - [mypy-qapi.parser] disallow_untyped_defs = False disallow_incomplete_defs = False diff --git a/scripts/qapi/schema.py b/scripts/qapi/schema.py index 353e8020a2..ff16578f6d 100644 --- a/scripts/qapi/schema.py +++ b/scripts/qapi/schema.py @@ -28,7 +28,7 @@ from .parser import QAPISchemaParser class QAPISchemaEntity: meta: Optional[str] = None - def __init__(self, name, info, doc, ifcond=None, features=None): + def __init__(self, name: str, info, doc, ifcond=None, features=None): assert name is None or isinstance(name, str) for f in features or []: assert isinstance(f, QAPISchemaFeature) -- 2.26.2