Re: [PATCH v3 00/15] python: drop qemu.qmp from qemu.git tree

["Daniel P. Berrangé" <berrange@redhat.com>]

On Tue, Dec 09, 2025 at 03:05:43PM -0500, John Snow wrote:
> On Tue, Dec 9, 2025 at 2:46 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Tue, Dec 09, 2025 at 12:23:21PM -0500, John Snow wrote:
> > > On Tue, Dec 9, 2025, 12:00 PM Daniel P. Berrangé <berrange@redhat.com>
> > > wrote:
> > >
> > > > On Fri, Dec 05, 2025 at 01:00:42AM -0500, John Snow wrote:
> > > > > Hello!
> > > > >
> > > > > This series drops the in-tree version of our python-qemu-qmp package
> > > > > ("qemu.qmp") in favor of the version hosted on PyPI, whose repository is
> > > > > located at https://gitlab.com/qemu-project/python-qemu-qmp.
> > > > >
> > > > > GitLab CI: https://gitlab.com/jsnow/qemu/-/pipelines/2197613036
> > > > >        (FreeBSD isn't my fault...!)
> > > > >
> > > > > The major effects of this patch series are:
> > > > >
> > > > > 1. qemu.qmp will be installed from PyPI or vendored packages instead of
> > > > >    being utilized directly from the qemu.git tree.
> > > >
> > > > This is not getting installed in enough scenarios IMHO.
> > > >
> > >
> > > It's hard to trigger an install when you don't use the build system,
> >
> > Well I am using the build system in the same way that I've always
> > used it with QEMU. IOW, the benchmark for this is the current way
> > things work with the python stuff in tree. The ideal would be to
> > match that behaviour with no workflow changes needed, if it is
> > practical to achieve that without major downsides.
> 
> I know, but if we add out-of-tree things, there's some fundamental
> things that change - we need to load that dependency somewhere,
> somewhen.

Yep, there's a tradeoff, so we have to figure out what's practical
to achieve.

> > > though... If you bypass make/meson/ninja entirely I'm not really sure what
> > > I can do to bootstrap the test deps.
> >
> > Can we just do it unconditionally in configure / meson ? These aren't
> > big files to download, and we're already dealing other python stuff
> > for the build, and git submodules, and rust crates. Pulling in qemu.qmp
> > doesn't feel like it is a burden to do by default ?
> 
> I definitely could, and I think it would be rather convenient; I only
> have some minor concerns about it:
> 
> - We don't promise tests on all platforms that we promise builds on.
> We have definitely broken this in the past. It might work currently,
> but it's a line I want to be aware we are crossing. It may necessitate
> python3-wheel and python3-setuptools just to build in this case.
> That's probably not an issue on workstations, but it's more bricks on
> the wall that are actually not truly needed until you run tests (or
> prepare to run tests).

We should focus on doing what's optimal for platforms where
tests do work, since they're inherantly our most important
platforms. If we proactively fetch python deps for tests that
happen to be broken on a platform already, I'm not too fussed
about that, as long as we don't break things more than they
are already broken.


> - Configure will get slower by default. I can install the core test
> deps by default if people don't mind the additional delay time. It
> might be something like 2-4 seconds, depending. If you don't care, I
> don't.

IIUC,  'pip' will have a local cache of downloads somewhere under
$HOME, so presumably when we install deps, we're only going to have
a download hit the very first time on a machine, or when the needed
versions change ? If we're mostly just copying files out of the
local cache, the time overhead should remain fairly minimal most of
the time.

We have precedent for downloading stuff at configure stage from
the Rust deps and git submodules. Those Rust deps are used for
built code rather than tests, but if we did need some rust deps
only for tests, presumably they'd still be done in configure ?

> - Things like functional tests are still going to require some kind of
> environment prep for all the extra stuff they require, I don't think
> it's reasonable to preload all of that stuff at configure time, and we
> never have. "make check-functional" is sufficient to pull those deps
> in, but if there are ways to engage the functional tests manually that
> people are using, I think another preparation step is unavoidable
> there.

I'd we interested to know how much overhead loading the functional
test deps actually is ?

As a conceptual point beyond this series, I've always tended to expect
that a "configure" script (or equivalent) is responsible for validating
the host OS pre-requisites and/or fetching neede pre-requisites.

Anything after that (whether builds or tests) would then (mostly) work
from stuff that was already acquired by configure.

This lets devs see upfront if there are any problems with their host,
instead of the host problems being hit later where the errors are
mixed in with build logs.

With functional tests we of course don't want to download the assets
upfront as that's GB's of data. I would tend towards suggesting that
any python deps should be fetched upfront at the configure stage
though, unless we see an unacceptable time hit from that.

> So, in addition to the "pyrun" wrapper I mailed in a separate reply
> (which I think is a good idea anyway, regardless of what direction we
> go here), I see two main paths that address your issues in differing
> amounts:

I sent that as a formal proposal here:

  https://lists.nongnu.org/archive/html/qemu-devel/2025-12/msg01351.html

For the functional test example illustrated there though, we would
preferrably want the functional test deps fetched upfront.

> (1) make configure prepare the test deps *by default*, adding a flag
> like --without-test-deps to disable it, or

Note, this is not just about test deps. The qemu python code is used by
other misc QEMU developer tools, such as the qmp-shell command. I'm
considering the python deps as more like "QEMU build env pre-requisites",
which is what pushed me towards believeing they should be fetched by
default

> (2) Continue not prepping the test deps, but allow --with-test-deps to
> pre-load them.
> 
> More or less the same solution, just with different defaults. I'm
> fairly ambivalent, my only personal "habit" here is "I am really
> reluctant to touch configure unless there is a strong consensus around
> it because I dislike having to make arguments at that level."


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|