Re: [PATCH v3 00/15] python: drop qemu.qmp from qemu.git tree

[John Snow <jsnow@redhat.com>]

First of all: Ping!

Please-Review: Daniel P. Berrangé <berrange@redhat.com>
Please-Review: Paolo Bonzini <pbonzini@redhat.com>
Please-Review: Alex Bennée <alex.bennee@linaro.org>

It would be nice to get this series staged right as the tree opens so
that I can address any build issues it creates in a timely manner
during the next window.

Second, [below, in-line] ...

On Fri, Dec 5, 2025 at 1:01 AM John Snow <jsnow@redhat.com> wrote:
>
> Hello!
>
> This series drops the in-tree version of our python-qemu-qmp package
> ("qemu.qmp") in favor of the version hosted on PyPI, whose repository is
> located at https://gitlab.com/qemu-project/python-qemu-qmp.
>
> GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/2197613036
>        (FreeBSD isn't my fault...!)
>
> The major effects of this patch series are:
>
> 1. qemu.qmp will be installed from PyPI or vendored packages instead of
>    being utilized directly from the qemu.git tree.
> 2. There are no new python dependencies checked or installed during
>    configure.  All test dependencies are installed post-hoc on an
>    as-needed basis.
> 3. "make check-venv" is no longer required to be run manually before any
>    test that is integrated with meson; this includes "make check" and
>    "make check-functional".
> 4. "make check-venv" no longer installs functional test dependencies: it
>    installs only the core suite of python test dependencies.
> 5. "make check-venv" is now required as a pre-requisite for running
>    device-crash-test and manually executed iotests.
> 6. Unfortunately, python3-wheel and python3-setuptools are now required
>    on the host system if tests are to be executed and >= Python 3.13 is
>    used.
> 7. An awful lot of deleted lines of code, and a lot fewer headaches
>    managing two nearly-identical copies of this source code. O:-)
>
> Patches 1-5 are build system focused; they set up new pythondeps.toml,
> mkvenv, and meson systems in preparation for relying on an external
> qemu.qmp library, but does not yet make the switch.
>
> Patches 6-9 are testing and CI focused; they add necessary preparation
> steps to keep tests running happily once the in-tree qemu.qmp library is
> removed.
>
> Patches 10-15 are build system focused again; they implement the actual
> switchover to the external qemu.qmp library.
>
> v3:
>
>  - Fixed FreeBSD vm test
>  - Fixed Cirrus macOS test (Needed setuptools as a dep, "file:///" had
>    one too many forward slashes, which works fine for older
>    pip/setuptools, but was incompatibly fixed recently.)
>  - Fixed an issue that prevented installing the local python packages in
>    an offline, isolated build environment (need --no-isolated-build and
>    pip environment variable flags)
>  - Added the python3-wheel and python3-setuptools build dependencies to
>    lcitool, owing to the above issues
>  - Ensured that the $download flag (for PyPI access) from configure is
>    carried over into pytest/meson.build invocations.  This defaults to
>    "enabled", so test dependencies should not have any issue being
>    fetched on most developer workstations and build environments.
>  - Fixed several bugs in the mkvenv.py local package installation mechanism.
>  - Flags and workarounds added for installing local packages offline now
>    apply *only* to local package installations, and not "normal"
>    depspecs.
>  - Reordered patches into more logical "phases" for easier review (I
>    hope...?)
>
> Known issues: "make check-build" still installs all of the test
> dependencies to the pyvenv. I'm not sure if this is desired or not, but
> I ran into problems preventing it from doing this without breaking a ton
> of CI tests. I'll continue to investigate this, but wanted to send this
> v3 out.

I sussed it out. I *can* remove the pyvenv targets from "check-build",
and the reason CI tests fail when I do this is because the build
templates use NINJA=":", so running "make check" suppresses the meson
dependencies. It can be rectified by setting eight of the twelve total
occurrences of "make check-build" in our CI files to also run "make
check-venv".

It's not clear if we want this or not. It depends on what
"check-build" semantically means. On the one hand, it doesn't do any
prefetch for the functional tests, so you could make the argument that
environment setup is similar. On the other hand, I think it's used
almost exclusively by and for the CI test suite, in which case it
running the environment setup is likely a pretty helpful/useful thing.
It's something Thomas pointed out, but I don't feel like I have a
strong opinion on it currently.

--js

>
> v2:
>
>  - move "make check-venv" earlier in GitLab CI/CD pipeline, to avoid
>    re-running configure
>  - Fix functional tests not using PyPI to fetch pygdbmi package
>  - Remove pre-requisites which are now merged
>
> --js
>
> John Snow (15):
>   python/mkvenv: create timestamp file for each group "ensured"
>   python/mkvenv: bump 'qemu.qmp' dependency for testdeps
>   python/mkvenv: add 'checktests' and 'functests' dependency groups
>   python/mkvenv: add mechanism to install local package(s)
>   meson, mkvenv: add checktests and functests custom targets
>   tests: Use configured python to run GitLab iotests
>   tests: run "make check-venv" before running iotests
>   tests: ensure "make check-venv" is run for crash tests
>   tests/lcitool: add python3 wheel and setuptools deps for qemu
>   python: add vendored qemu.qmp package
>   meson, mkvenv: make iotests depend on checktests group
>   meson, mkvenv: make functional tests depend on functests group
>   meson, mkvenv: add qemu.git/python/qemu package to pythondeps.toml
>   tests: replace old "check-venv" target with meson target
>   python: delete qemu.qmp
>
>  meson.build                                   |    1 +
>  .gitlab-ci.d/buildtest.yml                    |   11 +-
>  .gitlab-ci.d/cirrus/freebsd-14.vars           |    2 +-
>  .gitlab-ci.d/cirrus/macos-14.vars             |    2 +-
>  python/qemu/qmp/__init__.py                   |   60 -
>  python/qemu/qmp/error.py                      |   53 -
>  python/qemu/qmp/events.py                     |  751 -----------
>  python/qemu/qmp/legacy.py                     |  339 -----
>  python/qemu/qmp/message.py                    |  217 ----
>  python/qemu/qmp/models.py                     |  146 ---
>  python/qemu/qmp/protocol.py                   | 1101 -----------------
>  python/qemu/qmp/py.typed                      |    0
>  python/qemu/qmp/qmp_client.py                 |  732 -----------
>  python/qemu/qmp/qmp_shell.py                  |  689 -----------
>  python/qemu/qmp/qmp_tui.py                    |  665 ----------
>  python/qemu/qmp/util.py                       |  150 ---
>  python/qemu/utils/qom_fuse.py                 |    1 -
>  python/scripts/mkvenv.py                      |   44 +-
>  python/scripts/vendor.py                      |    2 +
>  python/setup.cfg                              |   31 +-
>  python/tests/minreqs.txt                      |    8 +-
>  python/tests/protocol.py                      |  596 ---------
>  python/wheels/qemu_qmp-0.0.5-py3-none-any.whl |  Bin 0 -> 72263 bytes
>  pythondeps.toml                               |   13 +-
>  pyvenv/meson.build                            |   40 +
>  tests/Makefile.include                        |   25 +-
>  tests/docker/dockerfiles/alpine.docker        |    2 +
>  tests/docker/dockerfiles/centos9.docker       |    2 +
>  tests/functional/meson.build                  |    7 +-
>  tests/lcitool/projects/qemu.yml               |    2 +
>  tests/qemu-iotests/meson.build                |    2 +-
>  tests/vm/generated/freebsd.json               |    2 +
>  32 files changed, 126 insertions(+), 5570 deletions(-)
>  delete mode 100644 python/qemu/qmp/__init__.py
>  delete mode 100644 python/qemu/qmp/error.py
>  delete mode 100644 python/qemu/qmp/events.py
>  delete mode 100644 python/qemu/qmp/legacy.py
>  delete mode 100644 python/qemu/qmp/message.py
>  delete mode 100644 python/qemu/qmp/models.py
>  delete mode 100644 python/qemu/qmp/protocol.py
>  delete mode 100644 python/qemu/qmp/py.typed
>  delete mode 100644 python/qemu/qmp/qmp_client.py
>  delete mode 100644 python/qemu/qmp/qmp_shell.py
>  delete mode 100644 python/qemu/qmp/qmp_tui.py
>  delete mode 100644 python/qemu/qmp/util.py
>  delete mode 100644 python/tests/protocol.py
>  create mode 100644 python/wheels/qemu_qmp-0.0.5-py3-none-any.whl
>  create mode 100644 pyvenv/meson.build
>
> --
> 2.51.1
>
>