[RFC 2/2] python: integrate linter tests natively with meson

["Daniel P. Berrangé" <berrange@redhat.com>]

This creates meson test targets for the python linters which has a
number of benefits:

 * They can be invoked in the same manner as all other tests
   using meson or ninja, or the top level Makefile wrappers
 * They avoid the extra indirection of running via avocado
 * They get run by default when doing 'make check'

There is one general 'python' suite defined, and then a 'pythonVER'
suite for each tox environment listed in python/setup.cfg.

All python linters across all version get triggered by 'make check'.

To run just the python linters without other tests

  $ make check-python

To run just the python linters for Python 3.12

  $ make check-python312

Static lints can be enabled per source directory in meson.build by
adding

 pythontests += {
    'dir': meson.current_source_dir(),
    'pylint': true,
    'mypy': true,
    'flake8': true,
    'isort': true,
 }

If any given lint is omitted in the config, it defaults to disabled.

When the '--disable-downloads' option is given to 'configure', running
of the python tests is disabled since tox would otherwise attempt to
download files to populate the per-version envs.

TBD: if you have meson < 1.7, then 'build_by_default: false' does
not work correctly, so the tox envs get created by 'make', rather
than delayed until 'make check'. We could bump min meson to fix this
or accept that older versions have sub-optimal behaviour

TBD: setup.cfg defines 6 python venvs 3.8 -> 3.13 currently, which
gives alot of duplication in what we run with 'make check' by default.
This adds about 1 minute of wallclock time to a non-parallel build on
my (high end) laptop, or 15 seconds to a parallel build. This is a
combination of populating the tox envs, and pylint being generally
slow. IIRC there's a way to split tests in 'quick' vs 'thorough' so
'make check' runs 'quick' only by default, but I've not investigated
this yet.

TBD: purge python/Makefile and python/tests/*.sh wrappers. It is
desirable to have only one way of running tests that everyone agrees
on, so that we have consistency of results. Assuming the native meson
integration is good enough, this should make the Makefile obsolete,
as well as the .sh wrappers.

TBD: I added isort & mypy for the 'scripts/qapi' code as a demo, but
flake8 & pylint don't pass so are disabled.

TBD: I've not figured out a nice way to integrate qemu-iotests without
the linters.py indirection which is painful to consume. Perhaps solvable
by just giving all the iotests real file extensions (.sh & .py) so the
linters can auto-detect python files. Would then need to have an exclude
list in qemu-iotests/meson.build to skip non-compliant files initially.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 meson.build                |  1 +
 python/meson.build         | 43 ++++++++++++++++++++++++++++++++++++++
 python/scripts/__init__.py |  0
 scripts/meson.build        |  9 ++++++++
 tests/meson.build          |  1 +
 tests/python/meson.build   | 17 +++++++++++++++
 6 files changed, 71 insertions(+)
 create mode 100644 python/meson.build
 create mode 100644 python/scripts/__init__.py
 create mode 100644 tests/python/meson.build

diff --git a/meson.build b/meson.build
index 0ee79c664d..f1cbfda149 100644
--- a/meson.build
+++ b/meson.build
@@ -4440,6 +4440,7 @@ if stap.found()
   endforeach
 endif
 
+subdir('python')
 subdir('scripts')
 subdir('tools')
 subdir('pc-bios')
diff --git a/python/meson.build b/python/meson.build
new file mode 100644
index 0000000000..73360b6d43
--- /dev/null
+++ b/python/meson.build
@@ -0,0 +1,43 @@
+
+tox = find_program('tox', required: false)
+toxcfg = ['-c', meson.current_source_dir(),
+          '--workdir', meson.current_source_dir(),
+          '--root', meson.current_source_dir()]
+pythonlints = ['pylint', 'mypy', 'flake8', 'isort']
+
+pythontests = [
+    {
+        'dir': meson.current_source_dir() / 'qemu',
+        'pylint': true,
+        'mypy': true,
+        'flake8': true,
+        'isort': true,
+    },
+    {
+        'dir': meson.current_source_dir() / 'scripts',
+        'pylint': true,
+        'mypy': true,
+        'flake8': true,
+        'isort': true,
+    },
+]
+
+pyenvs = []
+if tox.found() and get_option('wrap_mode') != 'nodownload'
+    rc = run_command([tox, 'list'] + toxcfg,
+                     capture: true,
+                     check: true)
+
+    foreach envline: rc.stdout().strip().split('\n')
+        if envline.startswith('py')
+            pyenv = envline.split(' ')[0]
+            message('Adding python environment: ' + pyenv)
+
+            mkenv = custom_target('tox-env-' + pyenv,
+                                  command: [tox, 'run', '-e', pyenv, '--notest'] + toxcfg,
+                                  output: pyenv,
+                                  build_by_default: false)
+            pyenvs += { 'name': pyenv, 'version': pyenv.substring(2), 'target': mkenv }
+        endif
+    endforeach
+endif
diff --git a/python/scripts/__init__.py b/python/scripts/__init__.py
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/scripts/meson.build b/scripts/meson.build
index 532277f5a2..6a1fd9e5a9 100644
--- a/scripts/meson.build
+++ b/scripts/meson.build
@@ -3,3 +3,12 @@ if stap.found()
 endif
 
 test('xml-preprocess', files('xml-preprocess-test.py'), suite: ['unit'])
+
+
+pythontests += {
+    'dir': meson.current_source_dir() / 'qapi',
+    'pylint': false,
+    'mypy': true,
+    'flake8': false,
+    'isort': true,
+}
diff --git a/tests/meson.build b/tests/meson.build
index c59619220f..89eed7d259 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -88,3 +88,4 @@ subdir('qapi-schema')
 subdir('qtest')
 subdir('migration-stress')
 subdir('functional')
+subdir('python')
diff --git a/tests/python/meson.build b/tests/python/meson.build
new file mode 100644
index 0000000000..28be424d85
--- /dev/null
+++ b/tests/python/meson.build
@@ -0,0 +1,17 @@
+
+foreach pyenv: pyenvs
+    foreach scenario: pythontests
+        suites = ['python', 'python' + pyenv['version']]
+
+        foreach lint: pythonlints
+            if scenario.get(lint, false)
+                name = lint + scenario['dir'].replace(meson.project_source_root(), '').replace('/', '-')
+                toxargs = toxcfg + ['exec', '-e', pyenv['name'],
+                                    '--',
+                                    'python3', '-m', lint,
+                                    scenario['dir']]
+                test(name, tox, args: toxargs, suite: suites, depends: pyenv['target'])
+            endif
+        endforeach
+    endforeach
+endforeach
-- 
2.47.1