* [PATCH v3 1/6] qapi: Add some pylint ignores
2025-04-24 21:08 [PATCH v3 0/6] python: add QAPI and qapidoc et al to python linter tests John Snow
@ 2025-04-24 21:08 ` John Snow
2025-04-24 21:09 ` [PATCH v3 2/6] docs/qapidoc: linting fixes John Snow
` (5 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: John Snow @ 2025-04-24 21:08 UTC (permalink / raw)
To: qemu-devel
Cc: John Snow, Markus Armbruster, Michael Roth, Cleber Rosa,
Peter Maydell
This restores the linting baseline in QAPI.
Signed-off-by: John Snow <jsnow@redhat.com>
---
scripts/qapi/backend.py | 2 ++
1 file changed, 2 insertions(+)
diff --git a/scripts/qapi/backend.py b/scripts/qapi/backend.py
index 14e60aa67af..49ae6ecdd33 100644
--- a/scripts/qapi/backend.py
+++ b/scripts/qapi/backend.py
@@ -13,6 +13,7 @@
class QAPIBackend(ABC):
+ # pylint: disable=too-few-public-methods
@abstractmethod
def generate(self,
@@ -36,6 +37,7 @@ def generate(self,
class QAPICBackend(QAPIBackend):
+ # pylint: disable=too-few-public-methods
def generate(self,
schema: QAPISchema,
--
2.48.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH v3 2/6] docs/qapidoc: linting fixes
2025-04-24 21:08 [PATCH v3 0/6] python: add QAPI and qapidoc et al to python linter tests John Snow
2025-04-24 21:08 ` [PATCH v3 1/6] qapi: Add some pylint ignores John Snow
@ 2025-04-24 21:09 ` John Snow
2025-04-24 21:09 ` [PATCH v3 3/6] python: update missing dependencies from minreqs John Snow
` (4 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: John Snow @ 2025-04-24 21:09 UTC (permalink / raw)
To: qemu-devel
Cc: John Snow, Markus Armbruster, Michael Roth, Cleber Rosa,
Peter Maydell
This restores the linting baseline in qapidoc. The order of some imports
change slightly here due to configuring isort a little better:
previously, isort was having difficulty understanding that "compat" and
"qapidoc_legacy" were local modules because docs/sphinx "isn't a python
package". Configuring this manually, isort chooses a different import
ordering, which _is_ intentional here.
Also: extra ignores are added for pylint. The most recent versions of
pylint don't require these ignores, but the oldest versions we support
do, so in the extra ignores go.
Signed-off-by: John Snow <jsnow@redhat.com>
---
docs/sphinx/qapi_domain.py | 25 ++++++++++++++-----------
docs/sphinx/qapidoc.py | 5 +++--
2 files changed, 17 insertions(+), 13 deletions(-)
diff --git a/docs/sphinx/qapi_domain.py b/docs/sphinx/qapi_domain.py
index c94af5719ca..ebc46a72c61 100644
--- a/docs/sphinx/qapi_domain.py
+++ b/docs/sphinx/qapi_domain.py
@@ -20,16 +20,6 @@
from docutils import nodes
from docutils.parsers.rst import directives
-
-from compat import (
- CompatField,
- CompatGroupedField,
- CompatTypedField,
- KeywordNode,
- ParserFix,
- Signature,
- SpaceNode,
-)
from sphinx import addnodes
from sphinx.directives import ObjectDescription
from sphinx.domains import (
@@ -44,6 +34,16 @@
from sphinx.util.docutils import SphinxDirective
from sphinx.util.nodes import make_id, make_refnode
+from compat import (
+ CompatField,
+ CompatGroupedField,
+ CompatTypedField,
+ KeywordNode,
+ ParserFix,
+ Signature,
+ SpaceNode,
+)
+
if TYPE_CHECKING:
from typing import (
@@ -56,7 +56,6 @@
)
from docutils.nodes import Element, Node
-
from sphinx.addnodes import desc_signature, pending_xref
from sphinx.application import Sphinx
from sphinx.builders import Builder
@@ -168,6 +167,8 @@ class QAPIDescription(ParserFix):
"""
def handle_signature(self, sig: str, signode: desc_signature) -> Signature:
+ # pylint: disable=unused-argument
+
# Do nothing. The return value here is the "name" of the entity
# being documented; for QAPI, this is the same as the
# "signature", which is just a name.
@@ -210,6 +211,8 @@ def _get_fqn(self, name: Signature) -> str:
def add_target_and_index(
self, name: Signature, sig: str, signode: desc_signature
) -> None:
+ # pylint: disable=unused-argument
+
# name is the return value of handle_signature.
# sig is the original, raw text argument to handle_signature.
# For QAPI, these are identical, currently.
diff --git a/docs/sphinx/qapidoc.py b/docs/sphinx/qapidoc.py
index 661b2c4ed0e..8011ac9efaf 100644
--- a/docs/sphinx/qapidoc.py
+++ b/docs/sphinx/qapidoc.py
@@ -27,6 +27,7 @@
from __future__ import annotations
+
__version__ = "2.0"
from contextlib import contextmanager
@@ -56,8 +57,6 @@
QAPISchemaVisitor,
)
from qapi.source import QAPISourceInfo
-
-from qapidoc_legacy import QAPISchemaGenRSTVisitor # type: ignore
from sphinx import addnodes
from sphinx.directives.code import CodeBlock
from sphinx.errors import ExtensionError
@@ -65,6 +64,8 @@
from sphinx.util.docutils import SphinxDirective, switch_source_input
from sphinx.util.nodes import nested_parse_with_titles
+from qapidoc_legacy import QAPISchemaGenRSTVisitor # type: ignore
+
if TYPE_CHECKING:
from typing import (
--
2.48.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH v3 3/6] python: update missing dependencies from minreqs
2025-04-24 21:08 [PATCH v3 0/6] python: add QAPI and qapidoc et al to python linter tests John Snow
2025-04-24 21:08 ` [PATCH v3 1/6] qapi: Add some pylint ignores John Snow
2025-04-24 21:09 ` [PATCH v3 2/6] docs/qapidoc: linting fixes John Snow
@ 2025-04-24 21:09 ` John Snow
2025-04-24 21:09 ` [PATCH v3 4/6] python: add qapi static analysis tests John Snow
` (3 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: John Snow @ 2025-04-24 21:09 UTC (permalink / raw)
To: qemu-devel
Cc: John Snow, Markus Armbruster, Michael Roth, Cleber Rosa,
Peter Maydell
We pin all dependencies for the "check-minreqs" test because pip lacks a
dependency resolver that installs "the oldest possible package that
meets dependency criteria". So, in order to test our stated minimum
requirements, we pin all of our dependencies (and their dependencies,
transitively) at the oldest possible versions that still work and pass
tests; proving that our minimum requirements are correct.
(It also ensures no new features accidentally sneak in from developers
on newer platforms.)
A few transitive dependencies were omitted from the pinned dependency
file by accident; as a result, pip's dependency solver can pull in newer
dependencies, which we don't want. This patch corrects the previous
oversight and pins the missing dependencies.
Signed-off-by: John Snow <jsnow@redhat.com>
---
python/tests/minreqs.txt | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/python/tests/minreqs.txt b/python/tests/minreqs.txt
index a3f423efd84..19c0f5e4c50 100644
--- a/python/tests/minreqs.txt
+++ b/python/tests/minreqs.txt
@@ -38,10 +38,14 @@ pyflakes==2.5.0
# Transitive mypy dependencies
mypy-extensions==1.0.0
+tomli==1.1.0
typing-extensions==4.7.1
# Transitive pylint dependencies
astroid==2.15.4
+dill==0.2
lazy-object-proxy==1.4.0
+platformdirs==2.2.0
toml==0.10.0
+tomlkit==0.10.1
wrapt==1.14.0
--
2.48.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH v3 4/6] python: add qapi static analysis tests
2025-04-24 21:08 [PATCH v3 0/6] python: add QAPI and qapidoc et al to python linter tests John Snow
` (2 preceding siblings ...)
2025-04-24 21:09 ` [PATCH v3 3/6] python: update missing dependencies from minreqs John Snow
@ 2025-04-24 21:09 ` John Snow
2025-04-24 21:09 ` [PATCH v3 5/6] python: Drop redundant warn_unused_configs = True John Snow
` (2 subsequent siblings)
6 siblings, 0 replies; 8+ messages in thread
From: John Snow @ 2025-04-24 21:09 UTC (permalink / raw)
To: qemu-devel
Cc: John Snow, Markus Armbruster, Michael Roth, Cleber Rosa,
Peter Maydell
Update the python tests to also check QAPI and the QAPI Sphinx
extensions. The docs/sphinx/qapidoc_legacy.py file is not included in
these checks, as it is destined for removal soon. mypy is also not
called on the QAPI Sphinx extensions, owing to difficulties supporting
Sphinx 3.x - 8.x while maintaining static type checking support. mypy
*is* called on all of the QAPI tools themselves, though.
flake8, isort and mypy use the tool configuration from the existing
python directory (in setup.cfg). pylint continues to use the special
configuration located in scripts/qapi/ - that configuration is more
permissive. If we wish to unify the two configurations, that's a
separate series and a discussion for a later date.
The list of pylint ignores is also updated, owing again to the wide
window of pylint version support: newer versions require pragmas to
occasionally silence the "too many positional arguments" warning, but
older versions do not have such a warning category and will instead yelp
about an unrecognized option. Silence that warning, too.
As a result of this patch, one would be able to run any of the following
tests locally from the qemu.git/python directory and have it cover the
QAPI tooling as well. All of the following options run the python tests,
static analysis tests, and linter checks; but with different
combinations of dependencies and interpreters.
- "make check-minreqs" Run tests specifically under our oldest supported
Python and our oldest supported dependencies. This is the test that
runs on GitLab as "check-python-minreqs". This helps ensure we do not
regress support on older platforms accidentally.
- "make check-tox" Runs the tests under the newest supported
dependencies, but under each supported version of Python in turn. At
time of writing, this is Python 3.8 to 3.13 inclusive. This test helps
catch bleeding-edge problems before they become problems for developer
workstations. This is the GitLab test "check-python-tox" and is an
optionally run, may-fail test due to the unpredictable nature of new
dependencies being released into the ecosystem that may cause
regressions.
- "make check-dev" Runs the tests under the newest supported
dependencies using whatever version of Python the user happens to have
installed. This is a quick convenience check that does not map to any
particular GitLab test.
(Note! check-dev may be busted on Fedora 41 and bleeding edge versions
of setuptools. That's unrelated to this patch and I'll address it
separately and soon. Thank you for your patience, --mgmt)
Finally, finally, finally: this means that QAPI tooling will be linted
and type-checked from the GitLab pipelines.
Signed-off-by: John Snow <jsnow@redhat.com>
---
python/setup.cfg | 1 +
python/tests/minreqs.txt | 21 +++++++++++++++++++++
python/tests/qapi-flake8.sh | 4 ++++
python/tests/qapi-isort.sh | 6 ++++++
python/tests/qapi-mypy.sh | 2 ++
python/tests/qapi-pylint.sh | 6 ++++++
scripts/qapi/pylintrc | 1 +
7 files changed, 41 insertions(+)
create mode 100755 python/tests/qapi-flake8.sh
create mode 100755 python/tests/qapi-isort.sh
create mode 100755 python/tests/qapi-mypy.sh
create mode 100755 python/tests/qapi-pylint.sh
diff --git a/python/setup.cfg b/python/setup.cfg
index cf5af7e6641..84d8a1fd30d 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -47,6 +47,7 @@ devel =
urwid >= 2.1.2
urwid-readline >= 0.13
Pygments >= 2.9.0
+ sphinx >= 3.4.3
# Provides qom-fuse functionality
fuse =
diff --git a/python/tests/minreqs.txt b/python/tests/minreqs.txt
index 19c0f5e4c50..94928936d44 100644
--- a/python/tests/minreqs.txt
+++ b/python/tests/minreqs.txt
@@ -11,6 +11,9 @@
# When adding new dependencies, pin the very oldest non-yanked version
# on PyPI that allows the test suite to pass.
+# Dependencies for qapidoc/qapi_domain et al
+sphinx==3.4.3
+
# Dependencies for the TUI addon (Required for successful linting)
urwid==2.1.2
urwid-readline==0.13
@@ -49,3 +52,21 @@ platformdirs==2.2.0
toml==0.10.0
tomlkit==0.10.1
wrapt==1.14.0
+
+# Transitive sphinx dependencies
+Jinja2==2.7
+MarkupSafe==1.1.0
+alabaster==0.7.1
+babel==1.3
+docutils==0.12
+imagesize==0.5.0
+packaging==14.0
+pytz==2011b0
+requests==2.5.0
+snowballstemmer==1.1
+sphinxcontrib-applehelp==1.0.0
+sphinxcontrib-devhelp==1.0.0
+sphinxcontrib-htmlhelp==1.0.0
+sphinxcontrib-jsmath==1.0.0
+sphinxcontrib-qthelp==1.0.0
+sphinxcontrib-serializinghtml==1.0.0
diff --git a/python/tests/qapi-flake8.sh b/python/tests/qapi-flake8.sh
new file mode 100755
index 00000000000..7b5983d64a9
--- /dev/null
+++ b/python/tests/qapi-flake8.sh
@@ -0,0 +1,4 @@
+#!/bin/sh -e
+python3 -m flake8 ../scripts/qapi/ \
+ ../docs/sphinx/qapidoc.py \
+ ../docs/sphinx/qapi_domain.py
diff --git a/python/tests/qapi-isort.sh b/python/tests/qapi-isort.sh
new file mode 100755
index 00000000000..f31f12d3425
--- /dev/null
+++ b/python/tests/qapi-isort.sh
@@ -0,0 +1,6 @@
+#!/bin/sh -e
+python3 -m isort --sp . -c ../scripts/qapi/
+# Force isort to recognize "compat" as a local module and not third-party
+python3 -m isort --sp . -c -p compat -p qapidoc_legacy \
+ ../docs/sphinx/qapi_domain.py \
+ ../docs/sphinx/qapidoc.py
diff --git a/python/tests/qapi-mypy.sh b/python/tests/qapi-mypy.sh
new file mode 100755
index 00000000000..377b29b873b
--- /dev/null
+++ b/python/tests/qapi-mypy.sh
@@ -0,0 +1,2 @@
+#!/bin/sh -e
+python3 -m mypy ../scripts/qapi
diff --git a/python/tests/qapi-pylint.sh b/python/tests/qapi-pylint.sh
new file mode 100755
index 00000000000..f4bb7a5a795
--- /dev/null
+++ b/python/tests/qapi-pylint.sh
@@ -0,0 +1,6 @@
+#!/bin/sh -e
+SETUPTOOLS_USE_DISTUTILS=stdlib python3 -m pylint \
+ --rcfile=../scripts/qapi/pylintrc \
+ ../scripts/qapi/ \
+ ../docs/sphinx/qapidoc.py \
+ ../docs/sphinx/qapi_domain.py
diff --git a/scripts/qapi/pylintrc b/scripts/qapi/pylintrc
index d24eece7411..e16283ada3d 100644
--- a/scripts/qapi/pylintrc
+++ b/scripts/qapi/pylintrc
@@ -19,6 +19,7 @@ disable=consider-using-f-string,
too-many-instance-attributes,
too-many-positional-arguments,
too-many-statements,
+ unknown-option-value,
useless-option-value,
[REPORTS]
--
2.48.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH v3 5/6] python: Drop redundant warn_unused_configs = True
2025-04-24 21:08 [PATCH v3 0/6] python: add QAPI and qapidoc et al to python linter tests John Snow
` (3 preceding siblings ...)
2025-04-24 21:09 ` [PATCH v3 4/6] python: add qapi static analysis tests John Snow
@ 2025-04-24 21:09 ` John Snow
2025-04-24 21:09 ` [PATCH v3 6/6] qapi: delete un-needed python static analysis configs John Snow
2025-04-25 6:11 ` [PATCH v3 0/6] python: add QAPI and qapidoc et al to python linter tests Markus Armbruster
6 siblings, 0 replies; 8+ messages in thread
From: John Snow @ 2025-04-24 21:09 UTC (permalink / raw)
To: qemu-devel
Cc: John Snow, Markus Armbruster, Michael Roth, Cleber Rosa,
Peter Maydell
From: Markus Armbruster <armbru@redhat.com>
strict = True implies warn_unused_configs = True.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: John Snow <jsnow@redhat.com>
---
python/setup.cfg | 1 -
1 file changed, 1 deletion(-)
diff --git a/python/setup.cfg b/python/setup.cfg
index 84d8a1fd30d..757f41d9491 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -80,7 +80,6 @@ exclude = __pycache__,
[mypy]
strict = True
python_version = 3.8
-warn_unused_configs = True
namespace_packages = True
warn_unused_ignores = False
--
2.48.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH v3 6/6] qapi: delete un-needed python static analysis configs
2025-04-24 21:08 [PATCH v3 0/6] python: add QAPI and qapidoc et al to python linter tests John Snow
` (4 preceding siblings ...)
2025-04-24 21:09 ` [PATCH v3 5/6] python: Drop redundant warn_unused_configs = True John Snow
@ 2025-04-24 21:09 ` John Snow
2025-04-25 6:11 ` [PATCH v3 0/6] python: add QAPI and qapidoc et al to python linter tests Markus Armbruster
6 siblings, 0 replies; 8+ messages in thread
From: John Snow @ 2025-04-24 21:09 UTC (permalink / raw)
To: qemu-devel
Cc: John Snow, Markus Armbruster, Michael Roth, Cleber Rosa,
Peter Maydell
Since the previous commit, python/setup.cfg applies to scripts/qapi/ as
well. Configuration files in scripts/qapi/ override python/setup.cfg.
scripts/qapi/.flake8 and scripts/qapi/.isort.cfg actually match
python/setup.cfg exactly, and can go.
The differences between scripts/qapi/mypy.ini and python/setup.cfg are
harmless: namespace_packages being set to True is a requirement for the
PEP420 nested package structure of QEMU but not for scripts/qapi, but
has no effect on type checking the QAPI code. warn_unused_ignores is
used in python/ to be able to target a wide variety of mypy versions;
some of which that have added new ignore categories that are not present
in older versions.
Ultimately, scripts/qapi/mypy.ini can be removed without any real change
in behavior to how mypy enforces type safety there.
The pylint config is being left in place because the settings differ
enough from the python/ directory settings that we need a chit-chat on
how to merge them O:-)
Signed-off-by: John Snow <jsnow@redhat.com>
---
scripts/qapi/.flake8 | 3 ---
scripts/qapi/.isort.cfg | 7 -------
scripts/qapi/mypy.ini | 4 ----
3 files changed, 14 deletions(-)
delete mode 100644 scripts/qapi/.flake8
delete mode 100644 scripts/qapi/.isort.cfg
delete mode 100644 scripts/qapi/mypy.ini
diff --git a/scripts/qapi/.flake8 b/scripts/qapi/.flake8
deleted file mode 100644
index a873ff67309..00000000000
--- a/scripts/qapi/.flake8
+++ /dev/null
@@ -1,3 +0,0 @@
-[flake8]
-# Prefer pylint's bare-except checks to flake8's
-extend-ignore = E722
diff --git a/scripts/qapi/.isort.cfg b/scripts/qapi/.isort.cfg
deleted file mode 100644
index 643caa1fbd6..00000000000
--- a/scripts/qapi/.isort.cfg
+++ /dev/null
@@ -1,7 +0,0 @@
-[settings]
-force_grid_wrap=4
-force_sort_within_sections=True
-include_trailing_comma=True
-line_length=72
-lines_after_imports=2
-multi_line_output=3
diff --git a/scripts/qapi/mypy.ini b/scripts/qapi/mypy.ini
deleted file mode 100644
index 8109470a031..00000000000
--- a/scripts/qapi/mypy.ini
+++ /dev/null
@@ -1,4 +0,0 @@
-[mypy]
-strict = True
-disallow_untyped_calls = False
-python_version = 3.8
--
2.48.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH v3 0/6] python: add QAPI and qapidoc et al to python linter tests
2025-04-24 21:08 [PATCH v3 0/6] python: add QAPI and qapidoc et al to python linter tests John Snow
` (5 preceding siblings ...)
2025-04-24 21:09 ` [PATCH v3 6/6] qapi: delete un-needed python static analysis configs John Snow
@ 2025-04-25 6:11 ` Markus Armbruster
6 siblings, 0 replies; 8+ messages in thread
From: Markus Armbruster @ 2025-04-25 6:11 UTC (permalink / raw)
To: John Snow; +Cc: qemu-devel, Michael Roth, Cleber Rosa, Peter Maydell
John Snow <jsnow@redhat.com> writes:
> Hiya, this series turns on automated linting for scripts/qapi,
> docs/sphinx/qapidoc.py and docs/sphinx/qapi_domain.py.
>
> This includes flake8/isort/pylint/mypy for scripts/qapi, but omits mypy
> from the Sphinx plugins owing to my inability to strictly type the
> extensions given the wide versions of Sphinx we actually support.
>
> Though I have been using black in my own development, I have not yet
> enabled it anywhere automatically. Maybe soon.
>
> NOTE: the latest Fedora image doesn't have Python38 anymore; this series
> does not address that. A more substantial fix for testing will be
> forthcoming.
Series
Reviewed-by: Markus Armbruster <armbru@redhat.com>
^ permalink raw reply [flat|nested] 8+ messages in thread