* [RFC 1/2] python: be more selective in hiding mypy subclassing warning
2025-02-24 19:11 [RFC 0/2] python: integrate linter tests natively with meson Daniel P. Berrangé
@ 2025-02-24 19:11 ` Daniel P. Berrangé
2025-04-15 10:48 ` Thomas Huth
2025-02-24 19:11 ` [RFC 2/2] python: integrate linter tests natively with meson Daniel P. Berrangé
` (2 subsequent siblings)
3 siblings, 1 reply; 8+ messages in thread
From: Daniel P. Berrangé @ 2025-02-24 19:11 UTC (permalink / raw)
To: qemu-devel
Cc: Daniel P. Berrangé, Marc-André Lureau,
Markus Armbruster, Philippe Mathieu-Daudé, Cleber Rosa,
Paolo Bonzini, John Snow
Use an annotation inline to the file instead of in setup.cfg
This has the added advantage that the mypy checks now pass when
using 'mypy /path/to/python/qemu' as well as 'mypy -p qemu'.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
python/qemu/qmp/qmp_tui.py | 12 ++++++------
python/qemu/utils/qom_fuse.py | 2 +-
python/setup.cfg | 8 --------
3 files changed, 7 insertions(+), 15 deletions(-)
diff --git a/python/qemu/qmp/qmp_tui.py b/python/qemu/qmp/qmp_tui.py
index 2d9ebbd20b..8720d70e60 100644
--- a/python/qemu/qmp/qmp_tui.py
+++ b/python/qemu/qmp/qmp_tui.py
@@ -401,7 +401,7 @@ def run(self, debug: bool = False) -> None:
raise err
-class StatusBar(urwid.Text):
+class StatusBar(urwid.Text): # type: ignore
"""
A simple statusbar modelled using the Text widget. The status can be
set using the set_text function. All text set is aligned to right.
@@ -412,7 +412,7 @@ def __init__(self, text: str = ''):
super().__init__(text, align='right')
-class Editor(urwid_readline.ReadlineEdit):
+class Editor(urwid_readline.ReadlineEdit): # type: ignore
"""
A simple editor modelled using the urwid_readline.ReadlineEdit widget.
Mimcs GNU readline shortcuts and provides history support.
@@ -485,7 +485,7 @@ def keypress(self, size: Tuple[int, int], key: str) -> Optional[str]:
return None
-class EditorWidget(urwid.Filler):
+class EditorWidget(urwid.Filler): # type: ignore
"""
Wrapper around the editor widget.
@@ -498,7 +498,7 @@ def __init__(self, parent: App) -> None:
super().__init__(Editor(parent), valign='top')
-class HistoryBox(urwid.ListBox):
+class HistoryBox(urwid.ListBox): # type: ignore
"""
This widget is modelled using the ListBox widget, contains the list of
all messages both QMP messages and log messages to be shown in the TUI.
@@ -535,7 +535,7 @@ def mouse_event(self, size: Tuple[int, int], _event: str, button: float,
super().keypress(size, 'down')
-class HistoryWindow(urwid.Frame):
+class HistoryWindow(urwid.Frame): # type: ignore
"""
This window composes the HistoryBox and EditorWidget in a horizontal split.
By default the first focus is given to the history box.
@@ -572,7 +572,7 @@ def cb_add_to_history(self, msg: str, level: Optional[str] = None) -> None:
self.history.add_to_history(formatted)
-class Window(urwid.Frame):
+class Window(urwid.Frame): # type: ignore
"""
This window is the top most widget of the TUI and will contain other
windows. Each child of this widget is responsible for displaying a specific
diff --git a/python/qemu/utils/qom_fuse.py b/python/qemu/utils/qom_fuse.py
index cf7e344bd5..0a1e02af87 100644
--- a/python/qemu/utils/qom_fuse.py
+++ b/python/qemu/utils/qom_fuse.py
@@ -56,7 +56,7 @@
fuse.fuse_python_api = (0, 2)
-class QOMFuse(QOMCommand, Operations):
+class QOMFuse(QOMCommand, Operations): # type: ignore
"""
QOMFuse implements both fuse.Operations and QOMCommand.
diff --git a/python/setup.cfg b/python/setup.cfg
index cf5af7e664..8dcd4c946e 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -83,14 +83,6 @@ warn_unused_configs = True
namespace_packages = True
warn_unused_ignores = False
-[mypy-qemu.utils.qom_fuse]
-# fusepy has no type stubs:
-allow_subclassing_any = True
-
-[mypy-qemu.qmp.qmp_tui]
-# urwid and urwid_readline have no type stubs:
-allow_subclassing_any = True
-
# The following missing import directives are because these libraries do not
# provide type stubs. Allow them on an as-needed basis for mypy.
[mypy-fuse]
--
2.47.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [RFC 2/2] python: integrate linter tests natively with meson
2025-02-24 19:11 [RFC 0/2] python: integrate linter tests natively with meson Daniel P. Berrangé
2025-02-24 19:11 ` [RFC 1/2] python: be more selective in hiding mypy subclassing warning Daniel P. Berrangé
@ 2025-02-24 19:11 ` Daniel P. Berrangé
2025-02-25 6:04 ` [RFC 0/2] " Markus Armbruster
2025-04-14 16:14 ` Daniel P. Berrangé
3 siblings, 0 replies; 8+ messages in thread
From: Daniel P. Berrangé @ 2025-02-24 19:11 UTC (permalink / raw)
To: qemu-devel
Cc: Daniel P. Berrangé, Marc-André Lureau,
Markus Armbruster, Philippe Mathieu-Daudé, Cleber Rosa,
Paolo Bonzini, John Snow
This creates meson test targets for the python linters which has a
number of benefits:
* They can be invoked in the same manner as all other tests
using meson or ninja, or the top level Makefile wrappers
* They avoid the extra indirection of running via avocado
* They get run by default when doing 'make check'
There is one general 'python' suite defined, and then a 'pythonVER'
suite for each tox environment listed in python/setup.cfg.
All python linters across all version get triggered by 'make check'.
To run just the python linters without other tests
$ make check-python
To run just the python linters for Python 3.12
$ make check-python312
Static lints can be enabled per source directory in meson.build by
adding
pythontests += {
'dir': meson.current_source_dir(),
'pylint': true,
'mypy': true,
'flake8': true,
'isort': true,
}
If any given lint is omitted in the config, it defaults to disabled.
When the '--disable-downloads' option is given to 'configure', running
of the python tests is disabled since tox would otherwise attempt to
download files to populate the per-version envs.
TBD: if you have meson < 1.7, then 'build_by_default: false' does
not work correctly, so the tox envs get created by 'make', rather
than delayed until 'make check'. We could bump min meson to fix this
or accept that older versions have sub-optimal behaviour
TBD: setup.cfg defines 6 python venvs 3.8 -> 3.13 currently, which
gives alot of duplication in what we run with 'make check' by default.
This adds about 1 minute of wallclock time to a non-parallel build on
my (high end) laptop, or 15 seconds to a parallel build. This is a
combination of populating the tox envs, and pylint being generally
slow. IIRC there's a way to split tests in 'quick' vs 'thorough' so
'make check' runs 'quick' only by default, but I've not investigated
this yet.
TBD: purge python/Makefile and python/tests/*.sh wrappers. It is
desirable to have only one way of running tests that everyone agrees
on, so that we have consistency of results. Assuming the native meson
integration is good enough, this should make the Makefile obsolete,
as well as the .sh wrappers.
TBD: I added isort & mypy for the 'scripts/qapi' code as a demo, but
flake8 & pylint don't pass so are disabled.
TBD: I've not figured out a nice way to integrate qemu-iotests without
the linters.py indirection which is painful to consume. Perhaps solvable
by just giving all the iotests real file extensions (.sh & .py) so the
linters can auto-detect python files. Would then need to have an exclude
list in qemu-iotests/meson.build to skip non-compliant files initially.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
meson.build | 1 +
python/meson.build | 43 ++++++++++++++++++++++++++++++++++++++
python/scripts/__init__.py | 0
scripts/meson.build | 9 ++++++++
tests/meson.build | 1 +
tests/python/meson.build | 17 +++++++++++++++
6 files changed, 71 insertions(+)
create mode 100644 python/meson.build
create mode 100644 python/scripts/__init__.py
create mode 100644 tests/python/meson.build
diff --git a/meson.build b/meson.build
index 0ee79c664d..f1cbfda149 100644
--- a/meson.build
+++ b/meson.build
@@ -4440,6 +4440,7 @@ if stap.found()
endforeach
endif
+subdir('python')
subdir('scripts')
subdir('tools')
subdir('pc-bios')
diff --git a/python/meson.build b/python/meson.build
new file mode 100644
index 0000000000..73360b6d43
--- /dev/null
+++ b/python/meson.build
@@ -0,0 +1,43 @@
+
+tox = find_program('tox', required: false)
+toxcfg = ['-c', meson.current_source_dir(),
+ '--workdir', meson.current_source_dir(),
+ '--root', meson.current_source_dir()]
+pythonlints = ['pylint', 'mypy', 'flake8', 'isort']
+
+pythontests = [
+ {
+ 'dir': meson.current_source_dir() / 'qemu',
+ 'pylint': true,
+ 'mypy': true,
+ 'flake8': true,
+ 'isort': true,
+ },
+ {
+ 'dir': meson.current_source_dir() / 'scripts',
+ 'pylint': true,
+ 'mypy': true,
+ 'flake8': true,
+ 'isort': true,
+ },
+]
+
+pyenvs = []
+if tox.found() and get_option('wrap_mode') != 'nodownload'
+ rc = run_command([tox, 'list'] + toxcfg,
+ capture: true,
+ check: true)
+
+ foreach envline: rc.stdout().strip().split('\n')
+ if envline.startswith('py')
+ pyenv = envline.split(' ')[0]
+ message('Adding python environment: ' + pyenv)
+
+ mkenv = custom_target('tox-env-' + pyenv,
+ command: [tox, 'run', '-e', pyenv, '--notest'] + toxcfg,
+ output: pyenv,
+ build_by_default: false)
+ pyenvs += { 'name': pyenv, 'version': pyenv.substring(2), 'target': mkenv }
+ endif
+ endforeach
+endif
diff --git a/python/scripts/__init__.py b/python/scripts/__init__.py
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/scripts/meson.build b/scripts/meson.build
index 532277f5a2..6a1fd9e5a9 100644
--- a/scripts/meson.build
+++ b/scripts/meson.build
@@ -3,3 +3,12 @@ if stap.found()
endif
test('xml-preprocess', files('xml-preprocess-test.py'), suite: ['unit'])
+
+
+pythontests += {
+ 'dir': meson.current_source_dir() / 'qapi',
+ 'pylint': false,
+ 'mypy': true,
+ 'flake8': false,
+ 'isort': true,
+}
diff --git a/tests/meson.build b/tests/meson.build
index c59619220f..89eed7d259 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -88,3 +88,4 @@ subdir('qapi-schema')
subdir('qtest')
subdir('migration-stress')
subdir('functional')
+subdir('python')
diff --git a/tests/python/meson.build b/tests/python/meson.build
new file mode 100644
index 0000000000..28be424d85
--- /dev/null
+++ b/tests/python/meson.build
@@ -0,0 +1,17 @@
+
+foreach pyenv: pyenvs
+ foreach scenario: pythontests
+ suites = ['python', 'python' + pyenv['version']]
+
+ foreach lint: pythonlints
+ if scenario.get(lint, false)
+ name = lint + scenario['dir'].replace(meson.project_source_root(), '').replace('/', '-')
+ toxargs = toxcfg + ['exec', '-e', pyenv['name'],
+ '--',
+ 'python3', '-m', lint,
+ scenario['dir']]
+ test(name, tox, args: toxargs, suite: suites, depends: pyenv['target'])
+ endif
+ endforeach
+ endforeach
+endforeach
--
2.47.1
^ permalink raw reply related [flat|nested] 8+ messages in thread