[PATCH 0/2] Fix venv issues with Avocado by reverting to an older version

[PATCH 0/2] Fix venv issues with Avocado by reverting to an older version

[Paolo Bonzini]

Bumping avocado to version 101 has two issues.  First, there are problems
where Avocado is not logging of command lines or terminal output, and not
collecting Python logs outside the avocado namespace.

Second, the recent changes to Python handling mean that there is a single
virtual environment for all the build, instead of a separate one for testing.
Requiring a too-new version of avocado causes conflicts with any avocado
plugins installed on the host:

   $ make check-venv
   make[1]: Entering directory '/home/berrange/src/virt/qemu/build'
     GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
     VENVPIP install -e /home/berrange/src/virt/qemu/python/
     VENVPIP install -r /home/berrange/src/virt/qemu/tests/requirements.txt
   ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
   avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
   avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
   make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'

To avoid this issue, tests/requirements.txt should use a ">=" constraint
and the version of Avocado should be limited to what distros provide
in the system packages.  Only Fedora has Avocado, and more specifically
version 92.0.  For now, this series reverts to the older requirement
(version >=88.1) while leaving further version bumps to future changes.

Paolo

Paolo Bonzini (2):
  Revert "tests/requirements.txt: bump up avocado-framework version to
    101.0"
  tests: update avocado installation to use mkvenv

 python/tests/minreqs.txt |  3 ++-
 tests/Makefile.include   | 26 +++++++++++++-------------
 tests/requirements.txt   |  9 ---------
 3 files changed, 15 insertions(+), 23 deletions(-)
 delete mode 100644 tests/requirements.txt

-- 
2.40.1

[PATCH 1/2] Revert "tests/requirements.txt: bump up avocado-framework version to 101.0"

[Paolo Bonzini]

This reverts commit ec5ffa0056389c3c10ea2de1e78366f66f4e5abc.

Bumping avocado to version 101 has two issues.  First, there are problems
where Avocado is not logging of command lines or terminal output, and not
collecting Python logs outside the avocado namespace.

Second, the recent changes to Python handling mean that there is a single
virtual environment for all the build, instead of a separate one for testing.
Requiring a too-new version of avocado causes conflicts with any avocado
plugins installed on the host:

   $ make check-venv
   make[1]: Entering directory '/home/berrange/src/virt/qemu/build'
     GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
     VENVPIP install -e /home/berrange/src/virt/qemu/python/
     VENVPIP install -r /home/berrange/src/virt/qemu/tests/requirements.txt
   ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
   avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
   avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
   make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'

To avoid this issue, tests/requirements.txt should use a ">=" constraint
and the version of Avocado should be limited to what distros provide
in the system packages.  Only Fedora has Avocado, and more specifically
version 92.0 (though 98.0 is also available as a module).  As a first
step, this patch reverts the introduction of a too-new Avocado.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 tests/Makefile.include | 18 +++++++-----------
 tests/requirements.txt |  2 +-
 2 files changed, 8 insertions(+), 12 deletions(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 0184ef22373..8294a44816c 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -136,18 +136,14 @@ get-vm-image-fedora-31-%: check-venv
 # download all vm images, according to defined targets
 get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
 
-JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
-
 check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
-	$(call quiet-command, 							\
-            $(PYTHON) -m avocado 						\
-            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) 	\
-            $(if $(AVOCADO_TAGS),, 						\
-			--filter-by-tags-include-empty 				\
-			--filter-by-tags-include-empty-key) 			\
-		--max-parallel-tasks $(JOBS_OPTION:-j%=%) 			\
-            $(AVOCADO_CMDLINE_TAGS) 						\
-            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), 			\
+	$(call quiet-command, \
+            $(PYTHON) -m avocado \
+            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
+            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
+			--filter-by-tags-include-empty-key) \
+            $(AVOCADO_CMDLINE_TAGS) \
+            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
             "AVOCADO", "tests/avocado")
 
 check-acceptance-deprecated-warning:
diff --git a/tests/requirements.txt b/tests/requirements.txt
index 0e008b9aec3..07e713ef5ac 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -5,5 +5,5 @@
 # Note that qemu.git/python/ is implicitly installed to this venv when
 # 'make check-venv' is run, and will persist until configure is run
 # again.
-avocado-framework==101.0
+avocado-framework==88.1
 pycdlib==1.11.0
-- 
2.40.1

[PATCH 2/2] tests: update avocado installation to use mkvenv

[Paolo Bonzini]

The recent changes to Python handling mean that there is a single
virtual environment for all the build, instead of a separate one for testing.
Because this virtual environment will often have system site packages
available, it makes sense to use mkvenv.py to install avocado, which will
avoid using PyPI if a new-enough version is available in the system.

However, requiring a specific version of avocado will cause conflicts with
any avocado plugins installed on the host:

   $ make check-venv
   make[1]: Entering directory '/home/berrange/src/virt/qemu/build'
     GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
     VENVPIP install -e /home/berrange/src/virt/qemu/python/
     VENVPIP install -r /home/berrange/src/virt/qemu/tests/requirements.txt
   ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
   avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
   avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
   make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'

so the requirements should use a ">=" constraint and the versions of
Avocado and pycdlib should be limited to what distros provide
in the system packages.  Only Fedora has Avocado, and more specifically
version 92.0, while the following distros have pycdlib:

   CentOS Stream 8             1.11.0
   CentOS Stream 9             1.11.0
   Fedora 37                   1.13.0
   Fedora 38                   1.14.0
   Ubuntu 22.04                1.11.0
   Debian bookworm             1.12.0

So the current minimal versions specified by tests/requirements.txt are
okay.  Move them to the check-venv target and add the corresponding
constraints to python/tests/minreqs.txt as well.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1663
Reported-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 python/tests/minreqs.txt | 3 ++-
 tests/Makefile.include   | 8 ++++++--
 tests/requirements.txt   | 9 ---------
 3 files changed, 8 insertions(+), 12 deletions(-)
 delete mode 100644 tests/requirements.txt

diff --git a/python/tests/minreqs.txt b/python/tests/minreqs.txt
index 1ce72cef6d8..6c9336ad119 100644
--- a/python/tests/minreqs.txt
+++ b/python/tests/minreqs.txt
@@ -23,7 +23,8 @@ distlib==0.3.6
 fusepy==2.0.4
 
 # Test-runners, utilities, etc.
-avocado-framework==90.0
+avocado-framework==88.1
+pycdlib==1.11.0
 
 # Linters
 flake8==5.0.4
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 8294a44816c..82240d631fe 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -1,5 +1,8 @@
 # -*- Mode: makefile -*-
 
+AVOCADO_MIN_VERSION=88.1
+PYCDLIB_MIN_VERSION=1.11.0
+
 .PHONY: check-help
 check-help:
 	@echo "Regression testing targets:"
@@ -90,7 +93,7 @@ distclean-tcg: $(DISTCLEAN_TCG_TARGET_RULES)
 TARGETS=$(patsubst libqemu-%.fa, %, $(filter libqemu-%.fa, $(ninja-targets)))
 
 TESTS_VENV_TOKEN=$(BUILD_DIR)/pyvenv/tests.group
-TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
+TESTS_VENV_REQ=$(SRC_PATH)/python/tests/minreqs.txt
 TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 ifndef AVOCADO_TESTS
 	AVOCADO_TESTS=tests/avocado
@@ -112,7 +115,8 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
 
 $(TESTS_VENV_TOKEN): $(TESTS_VENV_REQ)
 	$(call quiet-venv-pip,install -e "$(SRC_PATH)/python/")
-	$(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
+	$(quiet-@)$(PYTHON) $(SRC_PATH)/python/scripts/mkvenv.py ensure --diagnose avocado --online \
+	      'avocado-framework>=$(AVOCADO_MIN_VERSION)' 'pycdlib>=$(PYCDLIB_MIN_VERSION)'
 	$(call quiet-command, touch $@)
 
 $(TESTS_RESULTS_DIR):
diff --git a/tests/requirements.txt b/tests/requirements.txt
deleted file mode 100644
index 07e713ef5ac..00000000000
--- a/tests/requirements.txt
+++ /dev/null
@@ -1,9 +0,0 @@
-# Add Python module requirements, one per line, to be installed
-# in the qemu build_dir/pyvenv Python virtual environment. For more info,
-# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-#
-# Note that qemu.git/python/ is implicitly installed to this venv when
-# 'make check-venv' is run, and will persist until configure is run
-# again.
-avocado-framework==88.1
-pycdlib==1.11.0
-- 
2.40.1

Re: [PATCH 0/2] Fix venv issues with Avocado by reverting to an older version

[Peter Maydell]

On Mon, 5 Jun 2023 at 08:58, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> Bumping avocado to version 101 has two issues.  First, there are problems
> where Avocado is not logging of command lines or terminal output, and not
> collecting Python logs outside the avocado namespace.
>
> Second, the recent changes to Python handling mean that there is a single
> virtual environment for all the build, instead of a separate one for testing.
> Requiring a too-new version of avocado causes conflicts with any avocado
> plugins installed on the host:
>
>    $ make check-venv
>    make[1]: Entering directory '/home/berrange/src/virt/qemu/build'
>      GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
>      VENVPIP install -e /home/berrange/src/virt/qemu/python/
>      VENVPIP install -r /home/berrange/src/virt/qemu/tests/requirements.txt
>    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
>    avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
>    avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
>    make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'
>
> To avoid this issue, tests/requirements.txt should use a ">=" constraint
> and the version of Avocado should be limited to what distros provide
> in the system packages.  Only Fedora has Avocado, and more specifically
> version 92.0.  For now, this series reverts to the older requirement
> (version >=88.1) while leaving further version bumps to future changes.

If the new Avocado version is broken, don't we also need a < constraint
so we don't get it by mistake ?

In particular, for a local build tree that currently has 101 installed,
if the tree is updated to include these two patches together, will that
correctly downgrade it to 88.1?

thanks
-- PMM

Re: [PATCH 0/2] Fix venv issues with Avocado by reverting to an older version

[Paolo Bonzini]

On 6/5/23 11:46, Peter Maydell wrote:
>> To avoid this issue, tests/requirements.txt should use a ">=" constraint
>> and the version of Avocado should be limited to what distros provide
>> in the system packages.  Only Fedora has Avocado, and more specifically
>> version 92.0.  For now, this series reverts to the older requirement
>> (version >=88.1) while leaving further version bumps to future changes.
> If the new Avocado version is broken, don't we also need a < constraint
> so we don't get it by mistake ?

I expected those to be bugs that get fixed in 102 or 101.1, so not a 
reason to impose a strict constraint.  But you're right, the version 
that would be installed from PyPI is the latest; I didn't notice because 
I do have avocado installed outside pyvenv/.

Is the logging issue limited to the one fixed by 
https://www.mail-archive.com/qemu-devel@nongnu.org/msg962758.html?  Or 
is there something more?

> In particular, for a local build tree that currently has 101 installed,
> if the tree is updated to include these two patches together, will that
> correctly downgrade it to 88.1?

No, it won't.  What you can do is "pyvenv/bin/pip uninstall 
avocado-framework".

Paolo

Re: [PATCH 0/2] Fix venv issues with Avocado by reverting to an older version

[Peter Maydell]

On Mon, 5 Jun 2023 at 11:51, Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 6/5/23 11:46, Peter Maydell wrote:
> >> To avoid this issue, tests/requirements.txt should use a ">=" constraint
> >> and the version of Avocado should be limited to what distros provide
> >> in the system packages.  Only Fedora has Avocado, and more specifically
> >> version 92.0.  For now, this series reverts to the older requirement
> >> (version >=88.1) while leaving further version bumps to future changes.
> > If the new Avocado version is broken, don't we also need a < constraint
> > so we don't get it by mistake ?
>
> I expected those to be bugs that get fixed in 102 or 101.1, so not a
> reason to impose a strict constraint.  But you're right, the version
> that would be installed from PyPI is the latest; I didn't notice because
> I do have avocado installed outside pyvenv/.
>
> Is the logging issue limited to the one fixed by
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg962758.html?  Or
> is there something more?

I don't know, as I haven't tested with that patch.

> > In particular, for a local build tree that currently has 101 installed,
> > if the tree is updated to include these two patches together, will that
> > correctly downgrade it to 88.1?
>
> No, it won't.  What you can do is "pyvenv/bin/pip uninstall
> avocado-framework".

I think if we're going to revert back from Avocado 101 we should
actively do that, not merely allow older versions.

-- PMM

Re: [PATCH 0/2] Fix venv issues with Avocado by reverting to an older version

[Paolo Bonzini]

On 6/5/23 12:58, Peter Maydell wrote:
> On Mon, 5 Jun 2023 at 11:51, Paolo Bonzini <pbonzini@redhat.com> wrote:
>>
>> On 6/5/23 11:46, Peter Maydell wrote:
>>>> To avoid this issue, tests/requirements.txt should use a ">=" constraint
>>>> and the version of Avocado should be limited to what distros provide
>>>> in the system packages.  Only Fedora has Avocado, and more specifically
>>>> version 92.0.  For now, this series reverts to the older requirement
>>>> (version >=88.1) while leaving further version bumps to future changes.
>>> If the new Avocado version is broken, don't we also need a < constraint
>>> so we don't get it by mistake ?
>>
>> I expected those to be bugs that get fixed in 102 or 101.1, so not a
>> reason to impose a strict constraint.  But you're right, the version
>> that would be installed from PyPI is the latest; I didn't notice because
>> I do have avocado installed outside pyvenv/.
>>
>> Is the logging issue limited to the one fixed by
>> https://www.mail-archive.com/qemu-devel@nongnu.org/msg962758.html?  Or
>> is there something more?
> 
> I don't know, as I haven't tested with that patch.
> 
>>> In particular, for a local build tree that currently has 101 installed,
>>> if the tree is updated to include these two patches together, will that
>>> correctly downgrade it to 88.1?
>>
>> No, it won't.  What you can do is "pyvenv/bin/pip uninstall
>> avocado-framework".
> 
> I think if we're going to revert back from Avocado 101 we should
> actively do that, not merely allow older versions.

I think for now I can revert 9c6692db550f739a84fe4b677428df09d9fafdc3 
too (which will have the side effect of reverting to the older version) 
together with the first patch.

Paolo