[PATCH v2 0/2] Revert recent Avocado changes

[PATCH v2 0/2] Revert recent Avocado changes

[Paolo Bonzini]

Bumping avocado to version 101 has two issues.  First, there are problems
where Avocado is not logging of command lines or terminal output, and not
collecting Python logs outside the avocado namespace.

Second, the recent changes to Python handling mean that there is a single
virtual environment for all the build, instead of a separate one for testing.
Requiring a too-new version of avocado causes conflicts with any avocado
plugins installed on the host:

   $ make check-venv
   make[1]: Entering directory '/home/berrange/src/virt/qemu/build'
     GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
     VENVPIP install -e /home/berrange/src/virt/qemu/python/
     VENVPIP install -r /home/berrange/src/virt/qemu/tests/requirements.txt
   ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
   avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
   avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
   make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'

To avoid this issue, tests/requirements.txt should use a ">=" constraint
and the version of Avocado should be limited to what distros provide
in the system packages.  However, this requires more work, so for now
reintroduce the avocado-specific virtual environment instead of using
pyvenv/.

Paolo

Supersedes: <20230605075823.48871-1-pbonzini@redhat.com>

Paolo Bonzini (2):
  Revert "tests/requirements.txt: bump up avocado-framework version to
    101.0"
  tests: Use separate virtual environment for avocado

 .gitlab-ci.d/buildtest.yml                    |  6 ++--
 docs/devel/acpi-bits.rst                      |  6 ++--
 docs/devel/testing.rst                        | 14 ++++----
 .../org.centos/stream/8/x86_64/test-avocado   |  4 +--
 scripts/device-crash-test                     |  2 +-
 tests/Makefile.include                        | 32 +++++++++----------
 tests/requirements.txt                        |  9 ++----
 tests/vm/Makefile.include                     |  2 +-
 8 files changed, 35 insertions(+), 40 deletions(-)

-- 
2.40.1

[PATCH v2 1/2] Revert "tests/requirements.txt: bump up avocado-framework version to 101.0"

[Paolo Bonzini]

This reverts commit ec5ffa0056389c3c10ea2de1e78366f66f4e5abc.

Bumping avocado to version 101 has two issues.  First, there are problems
where Avocado is not logging of command lines or terminal output, and not
collecting Python logs outside the avocado namespace.

Second, the recent changes to Python handling mean that there is a single
virtual environment for all the build, instead of a separate one for testing.
Requiring a too-new version of avocado causes conflicts with any avocado
plugins installed on the host:

   $ make check-venv
   make[1]: Entering directory '/home/berrange/src/virt/qemu/build'
     GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
     VENVPIP install -e /home/berrange/src/virt/qemu/python/
     VENVPIP install -r /home/berrange/src/virt/qemu/tests/requirements.txt
   ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
   avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
   avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
   make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'

To avoid this issue, tests/requirements.txt should use a ">=" constraint
and the version of Avocado should be limited to what distros provide
in the system packages.  Only Fedora has Avocado, and more specifically
version 92.0 (though 98.0 is also available as a module).  As a first
step, this patch reverts the introduction of a too-new Avocado.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 tests/Makefile.include | 18 +++++++-----------
 tests/requirements.txt |  2 +-
 2 files changed, 8 insertions(+), 12 deletions(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index 0184ef22373..8294a44816c 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -136,18 +136,14 @@ get-vm-image-fedora-31-%: check-venv
 # download all vm images, according to defined targets
 get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOWNLOAD))
 
-JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
-
 check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
-	$(call quiet-command, 							\
-            $(PYTHON) -m avocado 						\
-            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) 	\
-            $(if $(AVOCADO_TAGS),, 						\
-			--filter-by-tags-include-empty 				\
-			--filter-by-tags-include-empty-key) 			\
-		--max-parallel-tasks $(JOBS_OPTION:-j%=%) 			\
-            $(AVOCADO_CMDLINE_TAGS) 						\
-            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), 			\
+	$(call quiet-command, \
+            $(PYTHON) -m avocado \
+            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
+            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
+			--filter-by-tags-include-empty-key) \
+            $(AVOCADO_CMDLINE_TAGS) \
+            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
             "AVOCADO", "tests/avocado")
 
 check-acceptance-deprecated-warning:
diff --git a/tests/requirements.txt b/tests/requirements.txt
index 0e008b9aec3..07e713ef5ac 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -5,5 +5,5 @@
 # Note that qemu.git/python/ is implicitly installed to this venv when
 # 'make check-venv' is run, and will persist until configure is run
 # again.
-avocado-framework==101.0
+avocado-framework==88.1
 pycdlib==1.11.0
-- 
2.40.1

[PATCH v2 2/2] tests: Use separate virtual environment for avocado

[Paolo Bonzini]

This reverts commits eea2d141179 ("Makefile: remove $(TESTS_PYTHON)",
2023-05-26) and 9c6692db550 ("tests: Use configure-provided pyvenv for
tests", 2023-05-18).

Right now, there is a conflict between wanting a ">=" constraint when
using a distro-provided package and wanting a "==" constraint when
installing Avocado from PyPI; this would provide the best of both worlds
in terms of resiliency for both distros that have required packages and
distros that don't.

The conflict is visible also for meson, where we would like to install
the latest 0.63.x version but also accept a distro 1.1.x version.
But it is worse for avocado, for two reasons:

1) we cannot use an "==" constraint to install avocado if the venv
includes a system avocado.  The distro will package plugins that have
"==" constraints on the version that is included in the distro, and, using
"pip install avocado==88.1" on a venv that includes system packages will
result in this error:

   ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
   avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
   avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
   make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'

2) we cannot use ">=" either if the venv does _not_ include a system
avocado, because that would result in the installation of v101.0 which
is the one we've just reverted.

So the idea is to encode the dependencies as an (acceptable, locked)
tuple, like this hypothetical TOML that would be committed inside
python/ and used by mkvenv.py:

  [meson]
  meson = { minimum = "0.63.0", install = "0.63.3", canary = "meson" }

  [docs]
  # 6.0 drops support for Python 3.7, we still officially support it
  sphinx = { minimum = "1.6", install = "<6.0", canary = "sphinx-build" }
  sphinx_rtd_theme = { minimum = "0.5" }

  [avocado]
  avocado-framework = { minimum = "88.1", install = "88.1", canary = "avocado" }

Once this is implemented, it would also be possible to install avocado in
pyvenv/ using "mkvenv.py ensure", thus using the distro package on Fedora
and CentOS Stream (the only distros where it's available).  But until
this is implemented, keep avocado in a separate venv.  There is still the
benefit of using a single python for meson custom_targets and for sphinx.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
---
 .gitlab-ci.d/buildtest.yml                       |  6 +++---
 docs/devel/acpi-bits.rst                         |  6 +++---
 docs/devel/testing.rst                           | 14 +++++++-------
 .../ci/org.centos/stream/8/x86_64/test-avocado   |  4 ++--
 scripts/device-crash-test                        |  2 +-
 tests/Makefile.include                           | 16 +++++++++-------
 tests/requirements.txt                           |  7 ++-----
 tests/vm/Makefile.include                        |  2 +-
 8 files changed, 28 insertions(+), 29 deletions(-)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 0f1be14cb62..1922caf5363 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -103,7 +103,7 @@ crash-test-debian:
   script:
     - cd build
     - make NINJA=":" check-venv
-    - pyvenv/bin/python3 scripts/device-crash-test -q --tcg-only ./qemu-system-i386
+    - tests/venv/bin/python3 scripts/device-crash-test -q --tcg-only ./qemu-system-i386
 
 build-system-fedora:
   extends:
@@ -146,8 +146,8 @@ crash-test-fedora:
   script:
     - cd build
     - make NINJA=":" check-venv
-    - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc
-    - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32
+    - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc
+    - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32
 
 build-system-centos:
   extends:
diff --git a/docs/devel/acpi-bits.rst b/docs/devel/acpi-bits.rst
index 9677b0098f4..22e2580200c 100644
--- a/docs/devel/acpi-bits.rst
+++ b/docs/devel/acpi-bits.rst
@@ -61,19 +61,19 @@ Under ``tests/avocado/`` as the root we have:
    ::
 
      $ make check-venv (needed only the first time to create the venv)
-     $ ./pyvenv/bin/avocado run -t acpi tests/avocado
+     $ ./tests/venv/bin/avocado run -t acpi tests/avocado
 
    The above will run all acpi avocado tests including this one.
    In order to run the individual tests, perform the following:
    ::
 
-     $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py --tap -
+     $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py --tap -
 
    The above will produce output in tap format. You can omit "--tap -" in the
    end and it will produce output like the following:
    ::
 
-      $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py
+      $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py
       Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
       JOB ID     : eab225724da7b64c012c65705dc2fa14ab1defef
       JOB LOG    : /home/anisinha/avocado/job-results/job-2022-10-10T17.58-eab2257/job.log
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 2cafec41782..203facb417b 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -888,9 +888,9 @@ You can run the avocado tests simply by executing:
 
   make check-avocado
 
-This involves the automatic installation, from PyPI, of all the
-necessary avocado-framework dependencies into the QEMU venv within the
-build tree (at ``./pyvenv``). Test results are also saved within the
+This involves the automatic creation of Python virtual environment
+within the build tree (at ``tests/venv``) which will have all the
+right dependencies, and will save tests results also within the
 build tree (at ``tests/results``).
 
 Note: the build environment must be using a Python 3 stack, and have
@@ -947,7 +947,7 @@ may be invoked by running:
 
  .. code::
 
-  pyvenv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
+  tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
 
 Note that if ``make check-avocado`` was not executed before, it is
 possible to create the Python virtual environment with the dependencies
@@ -962,20 +962,20 @@ a test file. To run tests from a single file within the build tree, use:
 
  .. code::
 
-  pyvenv/bin/avocado run tests/avocado/$TESTFILE
+  tests/venv/bin/avocado run tests/avocado/$TESTFILE
 
 To run a single test within a test file, use:
 
  .. code::
 
-  pyvenv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
+  tests/venv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
 
 Valid test names are visible in the output from any previous execution
 of Avocado or ``make check-avocado``, and can also be queried using:
 
  .. code::
 
-  pyvenv/bin/avocado list tests/avocado
+  tests/venv/bin/avocado list tests/avocado
 
 Manual Installation
 ~~~~~~~~~~~~~~~~~~~
diff --git a/scripts/ci/org.centos/stream/8/x86_64/test-avocado b/scripts/ci/org.centos/stream/8/x86_64/test-avocado
index 73e7a1a3126..e0443fc8ae8 100755
--- a/scripts/ci/org.centos/stream/8/x86_64/test-avocado
+++ b/scripts/ci/org.centos/stream/8/x86_64/test-avocado
@@ -4,7 +4,7 @@
 # KVM and x86_64, or tests that are generic enough to be valid for all
 # targets. Such a test list can be generated with:
 #
-# ./pyvenv/bin/avocado list --filter-by-tags-include-empty \
+# ./tests/venv/bin/avocado list --filter-by-tags-include-empty \
 #   --filter-by-tags-include-empty-key -t accel:kvm,arch:x86_64 \
 #   tests/avocado/
 #
@@ -22,7 +22,7 @@
 #   - tests/avocado/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types
 #
 make get-vm-images
-./pyvenv/bin/avocado run \
+./tests/venv/bin/avocado run \
     --job-results-dir=tests/results/ \
     tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_i440fx_kvm \
     tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_kvm \
diff --git a/scripts/device-crash-test b/scripts/device-crash-test
index 353aa575d7b..b74d887331d 100755
--- a/scripts/device-crash-test
+++ b/scripts/device-crash-test
@@ -43,7 +43,7 @@ except ModuleNotFoundError as exc:
     print(f"Module '{exc.name}' not found.")
     print("  Try 'make check-venv' from your build directory,")
     print("  and then one way to run this script is like so:")
-    print(f'  > $builddir/pyvenv/bin/python3 "{path}"')
+    print(f'  > $builddir/tests/venv/bin/python3 "{path}"')
     sys.exit(1)
 
 logger = logging.getLogger('device-crash-test')
diff --git a/tests/Makefile.include b/tests/Makefile.include
index 8294a44816c..9422ddaece5 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -89,9 +89,10 @@ distclean-tcg: $(DISTCLEAN_TCG_TARGET_RULES)
 # Build up our target list from the filtered list of ninja targets
 TARGETS=$(patsubst libqemu-%.fa, %, $(filter libqemu-%.fa, $(ninja-targets)))
 
-TESTS_VENV_TOKEN=$(BUILD_DIR)/pyvenv/tests.group
+TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
 TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
 TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
+TESTS_PYTHON=$(TESTS_VENV_DIR)/bin/python3
 ifndef AVOCADO_TESTS
 	AVOCADO_TESTS=tests/avocado
 endif
@@ -107,10 +108,11 @@ else
 endif
 
 quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
-    $(PYTHON) -m pip -q --disable-pip-version-check $1, \
+    $(TESTS_PYTHON) -m pip -q --disable-pip-version-check $1, \
     "VENVPIP","$1")
 
-$(TESTS_VENV_TOKEN): $(TESTS_VENV_REQ)
+$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
+	$(call quiet-command, $(PYTHON) -m venv $@, VENV, $@)
 	$(call quiet-venv-pip,install -e "$(SRC_PATH)/python/")
 	$(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
 	$(call quiet-command, touch $@)
@@ -119,7 +121,7 @@ $(TESTS_RESULTS_DIR):
 	$(call quiet-command, mkdir -p $@, \
             MKDIR, $@)
 
-check-venv: $(TESTS_VENV_TOKEN)
+check-venv: $(TESTS_VENV_DIR)
 
 FEDORA_31_ARCHES_TARGETS=$(patsubst %-softmmu,%, $(filter %-softmmu,$(TARGETS)))
 FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(FEDORA_31_ARCHES_TARGETS))
@@ -129,7 +131,7 @@ FEDORA_31_DOWNLOAD=$(filter $(FEDORA_31_ARCHES),$(FEDORA_31_ARCHES_CANDIDATES))
 # download one specific Fedora 31 image
 get-vm-image-fedora-31-%: check-venv
 	$(call quiet-command, \
-             $(PYTHON) -m avocado vmimage get \
+             $(TESTS_PYTHON) -m avocado vmimage get \
              --distro=fedora --distro-version=31 --arch=$*, \
 	"AVOCADO", "Downloading avocado tests VM image for $*")
 
@@ -138,7 +140,7 @@ get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOW
 
 check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
 	$(call quiet-command, \
-            $(PYTHON) -m avocado \
+            $(TESTS_PYTHON) -m avocado \
             --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
             $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
 			--filter-by-tags-include-empty-key) \
@@ -161,7 +163,7 @@ check:
 check-build: run-ninja
 
 check-clean:
-	rm -rf $(TESTS_RESULTS_DIR)
+	rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
 
 clean: check-clean clean-tcg
 distclean: distclean-tcg
diff --git a/tests/requirements.txt b/tests/requirements.txt
index 07e713ef5ac..0ba561b6bdf 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,9 +1,6 @@
 # Add Python module requirements, one per line, to be installed
-# in the qemu build_dir/pyvenv Python virtual environment. For more info,
+# in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
-#
-# Note that qemu.git/python/ is implicitly installed to this venv when
-# 'make check-venv' is run, and will persist until configure is run
-# again.
+# Note that qemu.git/python/ is always implicitly installed.
 avocado-framework==88.1
 pycdlib==1.11.0
diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
index f0f5d32fb0f..c2a8ca1c175 100644
--- a/tests/vm/Makefile.include
+++ b/tests/vm/Makefile.include
@@ -5,7 +5,7 @@ ifeq ($(realpath $(SRC_PATH)),$(realpath .))
 VM_PYTHON = PYTHONPATH=$(SRC_PATH)/python /usr/bin/env python3
 VM_VENV =
 else
-VM_PYTHON = $(PYTHON)
+VM_PYTHON = $(TESTS_PYTHON)
 VM_VENV = check-venv
 endif
 
-- 
2.40.1

Re: [PATCH v2 2/2] tests: Use separate virtual environment for avocado

[John Snow]

On Mon, Jun 5, 2023 at 12:00 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> This reverts commits eea2d141179 ("Makefile: remove $(TESTS_PYTHON)",
> 2023-05-26) and 9c6692db550 ("tests: Use configure-provided pyvenv for
> tests", 2023-05-18).
>
> Right now, there is a conflict between wanting a ">=" constraint when
> using a distro-provided package and wanting a "==" constraint when
> installing Avocado from PyPI; this would provide the best of both worlds
> in terms of resiliency for both distros that have required packages and
> distros that don't.
>
> The conflict is visible also for meson, where we would like to install
> the latest 0.63.x version but also accept a distro 1.1.x version.
> But it is worse for avocado, for two reasons:
>
> 1) we cannot use an "==" constraint to install avocado if the venv
> includes a system avocado.  The distro will package plugins that have
> "==" constraints on the version that is included in the distro, and, using
> "pip install avocado==88.1" on a venv that includes system packages will
> result in this error:
>
>    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
>    avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
>    avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
>    make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'
>
> 2) we cannot use ">=" either if the venv does _not_ include a system
> avocado, because that would result in the installation of v101.0 which
> is the one we've just reverted.
>
> So the idea is to encode the dependencies as an (acceptable, locked)
> tuple, like this hypothetical TOML that would be committed inside
> python/ and used by mkvenv.py:
>
>   [meson]
>   meson = { minimum = "0.63.0", install = "0.63.3", canary = "meson" }
>
>   [docs]
>   # 6.0 drops support for Python 3.7, we still officially support it
>   sphinx = { minimum = "1.6", install = "<6.0", canary = "sphinx-build" }
>   sphinx_rtd_theme = { minimum = "0.5" }
>
>   [avocado]
>   avocado-framework = { minimum = "88.1", install = "88.1", canary = "avocado" }
>
> Once this is implemented, it would also be possible to install avocado in
> pyvenv/ using "mkvenv.py ensure", thus using the distro package on Fedora
> and CentOS Stream (the only distros where it's available).  But until
> this is implemented, keep avocado in a separate venv.  There is still the
> benefit of using a single python for meson custom_targets and for sphinx.
>
> Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  .gitlab-ci.d/buildtest.yml                       |  6 +++---
>  docs/devel/acpi-bits.rst                         |  6 +++---
>  docs/devel/testing.rst                           | 14 +++++++-------
>  .../ci/org.centos/stream/8/x86_64/test-avocado   |  4 ++--
>  scripts/device-crash-test                        |  2 +-
>  tests/Makefile.include                           | 16 +++++++++-------
>  tests/requirements.txt                           |  7 ++-----
>  tests/vm/Makefile.include                        |  2 +-
>  8 files changed, 28 insertions(+), 29 deletions(-)
>
> diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
> index 0f1be14cb62..1922caf5363 100644
> --- a/.gitlab-ci.d/buildtest.yml
> +++ b/.gitlab-ci.d/buildtest.yml
> @@ -103,7 +103,7 @@ crash-test-debian:
>    script:
>      - cd build
>      - make NINJA=":" check-venv
> -    - pyvenv/bin/python3 scripts/device-crash-test -q --tcg-only ./qemu-system-i386
> +    - tests/venv/bin/python3 scripts/device-crash-test -q --tcg-only ./qemu-system-i386
>
>  build-system-fedora:
>    extends:
> @@ -146,8 +146,8 @@ crash-test-fedora:
>    script:
>      - cd build
>      - make NINJA=":" check-venv
> -    - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc
> -    - pyvenv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32
> +    - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-ppc
> +    - tests/venv/bin/python3 scripts/device-crash-test -q ./qemu-system-riscv32
>
>  build-system-centos:
>    extends:
> diff --git a/docs/devel/acpi-bits.rst b/docs/devel/acpi-bits.rst
> index 9677b0098f4..22e2580200c 100644
> --- a/docs/devel/acpi-bits.rst
> +++ b/docs/devel/acpi-bits.rst
> @@ -61,19 +61,19 @@ Under ``tests/avocado/`` as the root we have:
>     ::
>
>       $ make check-venv (needed only the first time to create the venv)
> -     $ ./pyvenv/bin/avocado run -t acpi tests/avocado
> +     $ ./tests/venv/bin/avocado run -t acpi tests/avocado
>
>     The above will run all acpi avocado tests including this one.
>     In order to run the individual tests, perform the following:
>     ::
>
> -     $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py --tap -
> +     $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py --tap -
>
>     The above will produce output in tap format. You can omit "--tap -" in the
>     end and it will produce output like the following:
>     ::
>
> -      $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py
> +      $ ./tests/venv/bin/avocado run tests/avocado/acpi-bits.py
>        Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits
>        JOB ID     : eab225724da7b64c012c65705dc2fa14ab1defef
>        JOB LOG    : /home/anisinha/avocado/job-results/job-2022-10-10T17.58-eab2257/job.log
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index 2cafec41782..203facb417b 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -888,9 +888,9 @@ You can run the avocado tests simply by executing:
>
>    make check-avocado
>
> -This involves the automatic installation, from PyPI, of all the
> -necessary avocado-framework dependencies into the QEMU venv within the
> -build tree (at ``./pyvenv``). Test results are also saved within the
> +This involves the automatic creation of Python virtual environment
> +within the build tree (at ``tests/venv``) which will have all the
> +right dependencies, and will save tests results also within the
>  build tree (at ``tests/results``).
>
>  Note: the build environment must be using a Python 3 stack, and have
> @@ -947,7 +947,7 @@ may be invoked by running:
>
>   .. code::
>
> -  pyvenv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
> +  tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
>
>  Note that if ``make check-avocado`` was not executed before, it is
>  possible to create the Python virtual environment with the dependencies
> @@ -962,20 +962,20 @@ a test file. To run tests from a single file within the build tree, use:
>
>   .. code::
>
> -  pyvenv/bin/avocado run tests/avocado/$TESTFILE
> +  tests/venv/bin/avocado run tests/avocado/$TESTFILE
>
>  To run a single test within a test file, use:
>
>   .. code::
>
> -  pyvenv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
> +  tests/venv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
>
>  Valid test names are visible in the output from any previous execution
>  of Avocado or ``make check-avocado``, and can also be queried using:
>
>   .. code::
>
> -  pyvenv/bin/avocado list tests/avocado
> +  tests/venv/bin/avocado list tests/avocado
>
>  Manual Installation
>  ~~~~~~~~~~~~~~~~~~~
> diff --git a/scripts/ci/org.centos/stream/8/x86_64/test-avocado b/scripts/ci/org.centos/stream/8/x86_64/test-avocado
> index 73e7a1a3126..e0443fc8ae8 100755
> --- a/scripts/ci/org.centos/stream/8/x86_64/test-avocado
> +++ b/scripts/ci/org.centos/stream/8/x86_64/test-avocado
> @@ -4,7 +4,7 @@
>  # KVM and x86_64, or tests that are generic enough to be valid for all
>  # targets. Such a test list can be generated with:
>  #
> -# ./pyvenv/bin/avocado list --filter-by-tags-include-empty \
> +# ./tests/venv/bin/avocado list --filter-by-tags-include-empty \
>  #   --filter-by-tags-include-empty-key -t accel:kvm,arch:x86_64 \
>  #   tests/avocado/
>  #
> @@ -22,7 +22,7 @@
>  #   - tests/avocado/virtio_check_params.py:VirtioMaxSegSettingsCheck.test_machine_types
>  #
>  make get-vm-images
> -./pyvenv/bin/avocado run \
> +./tests/venv/bin/avocado run \
>      --job-results-dir=tests/results/ \
>      tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_i440fx_kvm \
>      tests/avocado/boot_linux.py:BootLinuxX8664.test_pc_q35_kvm \
> diff --git a/scripts/device-crash-test b/scripts/device-crash-test
> index 353aa575d7b..b74d887331d 100755
> --- a/scripts/device-crash-test
> +++ b/scripts/device-crash-test
> @@ -43,7 +43,7 @@ except ModuleNotFoundError as exc:
>      print(f"Module '{exc.name}' not found.")
>      print("  Try 'make check-venv' from your build directory,")
>      print("  and then one way to run this script is like so:")
> -    print(f'  > $builddir/pyvenv/bin/python3 "{path}"')
> +    print(f'  > $builddir/tests/venv/bin/python3 "{path}"')
>      sys.exit(1)
>
>  logger = logging.getLogger('device-crash-test')
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index 8294a44816c..9422ddaece5 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -89,9 +89,10 @@ distclean-tcg: $(DISTCLEAN_TCG_TARGET_RULES)
>  # Build up our target list from the filtered list of ninja targets
>  TARGETS=$(patsubst libqemu-%.fa, %, $(filter libqemu-%.fa, $(ninja-targets)))
>
> -TESTS_VENV_TOKEN=$(BUILD_DIR)/pyvenv/tests.group
> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>  TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
>  TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> +TESTS_PYTHON=$(TESTS_VENV_DIR)/bin/python3
>  ifndef AVOCADO_TESTS
>         AVOCADO_TESTS=tests/avocado
>  endif
> @@ -107,10 +108,11 @@ else
>  endif
>
>  quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
> -    $(PYTHON) -m pip -q --disable-pip-version-check $1, \
> +    $(TESTS_PYTHON) -m pip -q --disable-pip-version-check $1, \
>      "VENVPIP","$1")
>
> -$(TESTS_VENV_TOKEN): $(TESTS_VENV_REQ)
> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> +       $(call quiet-command, $(PYTHON) -m venv $@, VENV, $@)
>         $(call quiet-venv-pip,install -e "$(SRC_PATH)/python/")
>         $(call quiet-venv-pip,install -r $(TESTS_VENV_REQ))
>         $(call quiet-command, touch $@)
> @@ -119,7 +121,7 @@ $(TESTS_RESULTS_DIR):
>         $(call quiet-command, mkdir -p $@, \
>              MKDIR, $@)
>
> -check-venv: $(TESTS_VENV_TOKEN)
> +check-venv: $(TESTS_VENV_DIR)
>
>  FEDORA_31_ARCHES_TARGETS=$(patsubst %-softmmu,%, $(filter %-softmmu,$(TARGETS)))
>  FEDORA_31_ARCHES_CANDIDATES=$(patsubst ppc64,ppc64le,$(FEDORA_31_ARCHES_TARGETS))
> @@ -129,7 +131,7 @@ FEDORA_31_DOWNLOAD=$(filter $(FEDORA_31_ARCHES),$(FEDORA_31_ARCHES_CANDIDATES))
>  # download one specific Fedora 31 image
>  get-vm-image-fedora-31-%: check-venv
>         $(call quiet-command, \
> -             $(PYTHON) -m avocado vmimage get \
> +             $(TESTS_PYTHON) -m avocado vmimage get \
>               --distro=fedora --distro-version=31 --arch=$*, \
>         "AVOCADO", "Downloading avocado tests VM image for $*")
>
> @@ -138,7 +140,7 @@ get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, $(FEDORA_31_DOW
>
>  check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>         $(call quiet-command, \
> -            $(PYTHON) -m avocado \
> +            $(TESTS_PYTHON) -m avocado \
>              --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
>              $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>                         --filter-by-tags-include-empty-key) \
> @@ -161,7 +163,7 @@ check:
>  check-build: run-ninja
>
>  check-clean:
> -       rm -rf $(TESTS_RESULTS_DIR)
> +       rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
>
>  clean: check-clean clean-tcg
>  distclean: distclean-tcg
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index 07e713ef5ac..0ba561b6bdf 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -1,9 +1,6 @@
>  # Add Python module requirements, one per line, to be installed
> -# in the qemu build_dir/pyvenv Python virtual environment. For more info,
> +# in the tests/venv Python virtual environment. For more info,
>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> -#
> -# Note that qemu.git/python/ is implicitly installed to this venv when
> -# 'make check-venv' is run, and will persist until configure is run
> -# again.
> +# Note that qemu.git/python/ is always implicitly installed.
>  avocado-framework==88.1
>  pycdlib==1.11.0
> diff --git a/tests/vm/Makefile.include b/tests/vm/Makefile.include
> index f0f5d32fb0f..c2a8ca1c175 100644
> --- a/tests/vm/Makefile.include
> +++ b/tests/vm/Makefile.include
> @@ -5,7 +5,7 @@ ifeq ($(realpath $(SRC_PATH)),$(realpath .))
>  VM_PYTHON = PYTHONPATH=$(SRC_PATH)/python /usr/bin/env python3
>  VM_VENV =
>  else
> -VM_PYTHON = $(PYTHON)
> +VM_PYTHON = $(TESTS_PYTHON)
>  VM_VENV = check-venv
>  endif
>
> --
> 2.40.1

Boo.

ACK

--js

Re: [PATCH v2 0/2] Revert recent Avocado changes

[John Snow]

On Mon, Jun 5, 2023 at 12:00 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> Bumping avocado to version 101 has two issues.  First, there are problems
> where Avocado is not logging of command lines or terminal output, and not
> collecting Python logs outside the avocado namespace.
>
> Second, the recent changes to Python handling mean that there is a single
> virtual environment for all the build, instead of a separate one for testing.
> Requiring a too-new version of avocado causes conflicts with any avocado
> plugins installed on the host:
>
>    $ make check-venv
>    make[1]: Entering directory '/home/berrange/src/virt/qemu/build'
>      GIT     ui/keycodemapdb tests/fp/berkeley-testfloat-3 tests/fp/berkeley-softfloat-3 dtc
>      VENVPIP install -e /home/berrange/src/virt/qemu/python/
>      VENVPIP install -r /home/berrange/src/virt/qemu/tests/requirements.txt
>    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
>    avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
>    avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 101.0 which is incompatible.
>    make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'
>
> To avoid this issue, tests/requirements.txt should use a ">=" constraint
> and the version of Avocado should be limited to what distros provide
> in the system packages.  However, this requires more work, so for now
> reintroduce the avocado-specific virtual environment instead of using
> pyvenv/.

That still want you want to do as of right now? (I'm catching up.)

OK if so. We don't lose too much.

>
> Paolo
>
> Supersedes: <20230605075823.48871-1-pbonzini@redhat.com>
>
> Paolo Bonzini (2):
>   Revert "tests/requirements.txt: bump up avocado-framework version to
>     101.0"
>   tests: Use separate virtual environment for avocado
>
>  .gitlab-ci.d/buildtest.yml                    |  6 ++--
>  docs/devel/acpi-bits.rst                      |  6 ++--
>  docs/devel/testing.rst                        | 14 ++++----
>  .../org.centos/stream/8/x86_64/test-avocado   |  4 +--
>  scripts/device-crash-test                     |  2 +-
>  tests/Makefile.include                        | 32 +++++++++----------
>  tests/requirements.txt                        |  9 ++----
>  tests/vm/Makefile.include                     |  2 +-
>  8 files changed, 35 insertions(+), 40 deletions(-)
>
> --
> 2.40.1
>