Re: [RFC PATCH 0/9] tests: run python tests under the build/tests/venv environment

["Daniel P. Berrangé" <berrange@redhat.com>]

On Fri, May 13, 2022 at 11:55:18AM -0400, John Snow wrote:
> On Fri, May 13, 2022, 6:29 AM Daniel P. Berrangé <berrange@redhat.com>
> wrote:
> 
> > On Fri, May 13, 2022 at 09:35:23AM +0100, Daniel P. Berrangé wrote:
> > > On Thu, May 12, 2022 at 08:06:00PM -0400, John Snow wrote:
> > > > RFC: This is a very early, crude attempt at switching over to an
> > > > external Python package dependency for QMP. This series does not
> > > > actually make the switch in and of itself, but instead just switches to
> > > > the paradigm of using a venv in general to install the QEMU python
> > > > packages instead of using PYTHONPATH to load them from the source tree.
> > > >
> > > > (By installing the package, we can process dependencies.)
> > > >
> > > > I'm sending it to the list so I can show you some of what's ugly so far
> > > > and my notes on how I might make it less ugly.
> > > >
> > > > (1) This doesn't trigger venv creation *from* iotests, it merely prints
> > > > a friendly error message if "make check-venv" has not been run
> > > > first. Not the greatest.
> > >
> > > So if we run the sequence
> > >
> > >   mkdir build
> > >   cd build
> > >   ../configure
> > >   make
> > >   ./tests/qemu-iotests/check 001
> > >
> > > It won't work anymore, until we 'make check-venv' (or simply
> > > 'make check') ?
> > >
> > > I'm somewhat inclined to say that venv should be created
> > > unconditionally by default. ie a plain 'make' should always
> > > everything needed to be able to invoke the tests directly.
> > >
> > > > (2) This isn't acceptable for SRPM builds, because it uses PyPI to
> > fetch
> > > > packages just-in-time. My thought is to use an environment variable
> > like
> > > > QEMU_CHECK_NO_INTERNET that changes the behavior of the venv setup
> > > > process. We can use "--system-site-packages" as an argument to venv
> > > > creation and "--no-index" as an argument to pip installation to achieve
> > > > good behavior in SRPM building scenarios. It'd be up to the spec-writer
> > > > to opt into that behavior.
> > >
> > > I think I'd expect --system-site-packages to be the default behaviour.
> > > We expect QEMU to be compatible with the packages available in the
> > > distros that we're targetting. So if the dev has the python packages
> > > installed from their distro, we should be using them preferentially.
> > >
> > > This is similar to how we bundle slirp/capstone/etc, but will
> > > preferentially use the distro version if it is available.
> >
> > AFAICT from testing it, when '--system-site-packages' is set
> > for the venv, then 'pip install' appears to end up being a
> > no-op if the package is already present in the host, but
> > installs it if missing.
> >
> > IOW, if we default to --system-site-packages, but still
> > also run 'pip install', we should "do the right thing".
> > It'll use any  distro packages that are available, and
> > augment with stuff from pip. In the no-op case, pip will
> > still try to consult the pipy servers to fetch version
> > info IIUC, so we need to be able to stop that. So I'd
> > suggest a  --python-env arg taking three values
> >
> >  * "auto" (the default) - add --system-site-packages, but
> >    also run 'pip install'. Good for developers day-to-day
> >
> 
> Sounds like a decent balance...
> 
> ...My only concern is that the system packages might be very old and it's
> possible that the qemu packages will be "too new" or have conflicts with
> the system deps.
> 
> I'll just have to test this.
> 
> e.g. what if I want to require mypy >= 0.900 for testing, but you have a
> system package that requires mypy < 0.700?

I would expect us to not require packages that are not present in
the distros implied by 

  https://www.qemu.org/docs/master/about/build-platforms.html

if that was absolutely a must have, then gracefully skip tests
if the system version wasn't new enough. The user could always
pass --python-env=pip if they want to force new enough

> The python dep compatibility matrix I try to enforce and support for
> testing is already feeling overly wide. This might force me to support an
> even wider matrix, which I think is the precisely wrong direction for venvs
> where we want tighter control as a rule.

As above, from a QEMU POV we have clearly defined our targetted
platform matrix. Splitting off python packages isn't a reason
to change QEMU's platform matrix IMHO.

> >  * "system" - add --system-site-packages but never run
> >    'pip install'.  Good for formal package builds.
> >
> 
> We still have to install the in-tree qemu ns package, but we can use
> --no-index to do it. It'll fail if the deps aren't met.

> >  * "pip"  - don't add --system-site-packages, always run
> >    'pip install'. Good for testing compatibility with
> >    bleeding edge python versions, or if explicit full
> >    independance from host python install is desired.
> >
> 
> as arguments to configure, this spread of options makes sense to me than
> paolo's version, but I've still got some doubt on mixing system and venv
> packages.
> 
> I am also as of yet not sold on building the venv *from* configure, see my
> response to Paolo on that topic.
> 
> I'll keep plugging away for now, but the big picture is still a tad murky
> in my head.
> 
> --js

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|