Re: iotests and python dependencies

[John Snow <jsnow@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 5289 bytes --]

On Thu, May 5, 2022, 4:51 AM Kevin Wolf <kwolf@redhat.com> wrote:

> Am 04.05.2022 um 21:38 hat John Snow geschrieben:
> > Howdy!
> >
> > So, I want to finally delete python/qemu/qmp from qemu.git, and this
> > creates a small problem -- namely, iotests needs access to it in order
> > to run the python-based tests.
> >
> > What I think needs to happen is that we create a virtual environment
> > that installs python/qemu/. The reason this cannot be done with
> > PYTHONPATH alone anymore is because the qmp package itself won't be
> > there anymore, we need an installer like `pip` to actually fetch it
> > for us and put it somewhere. (i.e., we need to process the
> > dependencies of python/qemu now and can't treat it as a pre-installed
> > location.)
> >
> > Avocado tests are already creating a venv for the purposes of
> > installing and running Avocado. We can amend e.g. "../../python" to
> > tests/requirements.txt and the Avocado environment is A-OK good-to-go.
> > The Makefile magic for avocado tests creates a venv-per-build.
>
> "per build" sounded pretty bad, because I thought it meant building a
> new venv every time I run either 'make' or the tests. This would
> obviously be very noticable for short-running tests like some iotests.
> But fortunately this is not what it does, it keeps the venv around in
> the build directory. So it's only per build directory really, which I
> guess is fine.
>

Whoops, yeah. I meant per build directory. In contrast to the Python unit
tests themselves, which create some test venvs tied directly to the source
directory and are build-agnostic.


> > It seems to work well enough. One thing to note here is that the
> > supported invocation for avocado tests is only through the Makefile,
> > which handles creating and entering the venv to make the command
> > seamless.
> >
> > iotests, however, manages its own execution environment with
> > testenv.py, and we support running iotests from outside of the
> > Makefile, for example by going to $build/tests/qemu-iotests and
> > running ./check.
>
> Yes, and I agree that this is important.
>

Figured as much. Not plotting to take this away, I promise. Just getting my
requirements straight before I spend time concocting something.


> > Now ... I could update testenv.py to be smart enough to create and
> > enter a python venv, and have even prototyped this. It seems to work
> > pretty well! This approach seemed like the least invasive to how
> > iotests are expected to be run and used. But a downside with this
> > approach is that now avocado tests and iotests are each managing their
> > own python venv. Worse, vm-tests and device-crash-test are still
> > unhandled entirely.
>
> Is there a reason why testenv.py couldn't enter just the shared venv in
> build/tests/venv?
>

It can, but it'd have to be made first so it can enter it. I figured this
would only happen "on-demand", like how check-avocado works, so I'd need
some way for iotests and the Makefile to share the venv creation code,
which is certainly an option.


> If not, I guess it would be enough if iotests just checks that the venv
> exists and all of the dependencies are there in the right version and
> error out if not, telling the user to run 'make check-venv'.
>

I will spend unreasonable engineering hours avoiding this, if only for
pride. I want everything to be as seamless and easy as it's always been.


> Or actually, it could just unconditionally run 'make check-venv' by
> itself, which is probably easier to implement than checking the
> dependencies and more convenient for the user, too.
>

Oh, that's one way to get them to share the venv-creation pathway. Hadn't
occurred to me, but this might be easy to do.


> (One more detail: 'make check-venv GIT_SUBMODULES_ACTION=ignore' seems
> to make it pretty much instantaneous if the venv is current, but leaving
> the submodule mechanism enabled adds a noticable delay.)
>

Noted.


> > I'd like to find a solution where I create a unified python testing
> > venv tied to the build shared by avocado, iotests, vm-tests and
> > device-crash-test. I'm not completely sure how exactly I'll manage
> > that right now, but I wanted to throw this out there in case there are
> > some requirements I might be overlooking.
> >
> > I think vm-tests and avocado-tests can both have a venv created for
> > them and activated before the test runs. device-crash-test I believe
> > will need a script change in the gitlab ci yaml. iotests is somewhat
> > unique in that it needs to run both by manual invocation and from
> > makefile invocations. If I want a shared VM between all of these, I'll
> > need to isolate the create-and-enter-venv logic somewhere where it can
> > be shared both inside and outside of a Makefile.
>
> If just calling 'make' isn't an option, then moving that part out into a
> separate script probably wouldn't be too hard either. But 'make' has the
> advantage that it already contains the check if the venv is already
> there and requirements.txt hasn't changed, which you would have to
> replicate otherwise.
>

Yeah, exactly.


> Kevin
>

Thanks for the ideas!

As of now, I've got a branch where I've moved vm-tests over onto the same
venv with avocado-tests. If I can bridge the gap to iotests, I'll be pretty
happy.

[-- Attachment #2: Type: text/html, Size: 8102 bytes --]