* [PATCH v1 00/11] Convert avocado tests to normal Python unittests
@ 2024-07-16 11:26 Thomas Huth
2024-07-16 11:26 ` [PATCH 01/11] tests/functional: Add base classes for the upcoming pytest-based tests Thomas Huth
` (13 more replies)
0 siblings, 14 replies; 36+ messages in thread
From: Thomas Huth @ 2024-07-16 11:26 UTC (permalink / raw)
To: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé
Cc: Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
The Avocado v88 that we use in QEMU is already on a life support
system: It is not supported by upstream anymore, and with the latest
versions of Python, it won't work anymore since it depends on the
"imp" module that has been removed in Python 3.12.
There have been several attempts to update the test suite in QEMU
to a newer version of Avocado, but so far no attempt has successfully
been merged yet.
Additionally, the whole "make check" test suite in QEMU is using the
meson test runner nowadays, so running the python-based tests via the
Avocodo test runner looks and feels quite like an oddball, requiring
the users to deal with the knowledge of multiple test runners in
parallel (e.g. the timeout settings work completely differently).
So instead of trying to update the python-based test suite in QEMU
to a newer version of Avocado, we should maybe try to better integrate
it with the meson test runner instead. Indeed most tests work quite
nicely without the Avocado framework already, as you can see with
this patch series - it does not convert all tests, just a subset so
far, but this already proves that many tests only need small modifi-
cations to work without Avocado.
Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
classes (e.g. based on cloud-init images or using SSH) really depend
on the Avocado framework, so we'd need a solution for those if we
want to continue using them. One solution might be to simply use the
required functions from avocado.utils for these tests, and still run
them via the meson test runner instead, but that needs some further
investigation that will be done later.
Now if you want to try out these patches: Apply the patches, then
recompile and then run:
make check-functional
You can also run single targets e.g. with:
make check-functional-ppc
You can also run the tests without any test runner now by
setting the PYTHONPATH environment variable to the "python" folder
of your source tree, and by specifying the build directory via
QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
export PYTHONPATH=$HOME/qemu/python
export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
export PYTHONPATH=$HOME/qemu/build
~/qemu/tests/functional/test_virtio_version.py
The logs of the tests can be found in the build directory under
tests/functional/<testname> - console log and general logs will
be put in separate files there.
Still to be done: Update the documentation for this new test framework.
RFC -> v1:
- Now using pycotap for running the tests instead of "pytest"
- Change the name from "tests/pytest" to "tests/functional" accordingly
- Make it possible to run the tests directly
- Use Python's urllib instead of wget for downloading
- Lots of makefile / meson integration improvements
- Converted more tests
- Update MAINTAINERS file accordingly
- Added a patch to run check-functional in the gitlab-CI
- ... lots of other changes I forgot about ... in fact, I changed so
many things that I also did not dare to pick up the Reviewed-bys
from the RFC
Thomas Huth (11):
tests/functional: Add base classes for the upcoming pytest-based tests
tests/functional: Convert simple avocado tests into standalone python
tests
tests/functional: Convert avocado tests that just need a small
adjustment
tests/functional: Add python-based tests to the meson build system
tests/functional: Implement fetch_asset() method for downloading
assets
tests/functional: Convert some tests that download files via
fetch_asset()
tests/functional: Add a function for extracting files from an archive
tests/functional: Convert some avocado tests that needed
avocado.utils.archive
tests/functional: Set up logging
tests/functional: Convert the s390x avocado tests into standalone
tests
gitlab-ci: Add "check-functional" to the build tests
MAINTAINERS | 22 +-
.gitlab-ci.d/buildtest-template.yml | 3 +-
.gitlab-ci.d/buildtest.yml | 60 +--
pythondeps.toml | 3 +-
tests/Makefile.include | 18 +-
tests/functional/meson.build | 112 +++++
tests/functional/qemu_test/__init__.py | 384 ++++++++++++++++++
tests/functional/qemu_test/utils.py | 28 ++
.../test_arm_canona1100.py} | 21 +-
.../test_arm_n8x0.py} | 25 +-
.../test_avr_mega2560.py} | 11 +-
.../test_cpu_queries.py} | 7 +-
.../test_empty_cpu_model.py} | 7 +-
.../test_info_usernet.py} | 11 +-
.../test_loongarch64_virt.py} | 16 +-
.../test_mem_addr_space.py} | 53 +--
.../test_mips64el_loongson3v.py} | 26 +-
.../test_netdev_ethtool.py} | 32 +-
.../test_pc_cpu_hotplug_props.py} | 11 +-
.../ppc_405.py => functional/test_ppc_405.py} | 19 +-
.../test_ppc_74xx.py} | 74 ++--
.../test_ppc_bamboo.py} | 23 +-
.../test_s390x_ccw_virtio.py} | 32 +-
.../test_s390x_topology.py} | 70 ++--
.../test_sparc64_sun4u.py} | 25 +-
.../version.py => functional/test_version.py} | 13 +-
.../test_virtio_version.py} | 8 +-
tests/meson.build | 1 +
28 files changed, 795 insertions(+), 320 deletions(-)
create mode 100644 tests/functional/meson.build
create mode 100644 tests/functional/qemu_test/__init__.py
create mode 100644 tests/functional/qemu_test/utils.py
rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (71%)
mode change 100644 => 100755
rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (71%)
mode change 100644 => 100755
rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (90%)
mode change 100644 => 100755
rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%)
mode change 100644 => 100755
rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
mode change 100644 => 100755
rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
mode change 100644 => 100755
rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (89%)
mode change 100644 => 100755
rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (92%)
mode change 100644 => 100755
rename tests/{avocado/machine_mips_loongson3v.py => functional/test_mips64el_loongson3v.py} (55%)
mode change 100644 => 100755
rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (81%)
mode change 100644 => 100755
rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
mode change 100644 => 100755
rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
mode change 100644 => 100755
rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
mode change 100644 => 100755
rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (75%)
mode change 100644 => 100755
rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (95%)
mode change 100644 => 100755
rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (90%)
mode change 100644 => 100755
rename tests/{avocado/machine_sparc64_sun4u.py => functional/test_sparc64_sun4u.py} (60%)
mode change 100644 => 100755
rename tests/{avocado/version.py => functional/test_version.py} (78%)
mode change 100644 => 100755
rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
mode change 100644 => 100755
--
2.45.2
^ permalink raw reply [flat|nested] 36+ messages in thread
* [PATCH 01/11] tests/functional: Add base classes for the upcoming pytest-based tests
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
@ 2024-07-16 11:26 ` Thomas Huth
2024-07-16 11:26 ` [PATCH 02/11] tests/functional: Convert simple avocado tests into standalone python tests Thomas Huth
` (12 subsequent siblings)
13 siblings, 0 replies; 36+ messages in thread
From: Thomas Huth @ 2024-07-16 11:26 UTC (permalink / raw)
To: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé
Cc: Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
The file is mostly a copy of the tests/avocado/avocado_qemu/__init__.py
file with some adjustments to get rid of the Avocado dependencies (i.e.
we also have to drop the LinuxSSHMixIn and LinuxTest for now).
The emulator binary and build directory are now passed via
environment variables that will be set via meson.build later.
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
tests/functional/qemu_test/__init__.py | 326 +++++++++++++++++++++++++
1 file changed, 326 insertions(+)
create mode 100644 tests/functional/qemu_test/__init__.py
diff --git a/tests/functional/qemu_test/__init__.py b/tests/functional/qemu_test/__init__.py
new file mode 100644
index 0000000000..e73705d40a
--- /dev/null
+++ b/tests/functional/qemu_test/__init__.py
@@ -0,0 +1,326 @@
+# Test class and utilities for functional tests
+#
+# Copyright 2018, 2024 Red Hat, Inc.
+#
+# Original Author (Avocado-based tests):
+# Cleber Rosa <crosa@redhat.com>
+#
+# Adaption for standalone version:
+# Thomas Huth <thuth@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import logging
+import os
+import pycotap
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+import uuid
+import unittest
+
+from pathlib import Path
+from qemu.machine import QEMUMachine
+from qemu.utils import (get_info_usernet_hostfwd_port, kvm_available,
+ tcg_available)
+
+def _source_dir():
+ # Determine top-level directory of the QEMU sources
+ return Path(__file__).parent.parent.parent.parent
+
+def _build_dir():
+ root = os.getenv('QEMU_BUILD_ROOT')
+ if root is not None:
+ return Path(root)
+ # Makefile.mtest only exists in build dir, so if it is available, use CWD
+ if os.path.exists('Makefile.mtest'):
+ return Path(os.getcwd())
+
+ root = os.path.join(_source_dir(), 'build')
+ if os.path.exists(root):
+ return Path(root)
+
+ raise Exception("Cannot identify build dir, set QEMU_BUILD_ROOT")
+
+def has_cmd(name, args=None):
+ """
+ This function is for use in a @skipUnless decorator, e.g.:
+
+ @skipUnless(*has_cmd('sudo -n', ('sudo', '-n', 'true')))
+ def test_something_that_needs_sudo(self):
+ ...
+ """
+
+ if args is None:
+ args = ('which', name)
+
+ try:
+ _, stderr, exitcode = run_cmd(args)
+ except Exception as e:
+ exitcode = -1
+ stderr = str(e)
+
+ if exitcode != 0:
+ cmd_line = ' '.join(args)
+ err = f'{name} required, but "{cmd_line}" failed: {stderr.strip()}'
+ return (False, err)
+ else:
+ return (True, '')
+
+def has_cmds(*cmds):
+ """
+ This function is for use in a @skipUnless decorator and
+ allows checking for the availability of multiple commands, e.g.:
+
+ @skipUnless(*has_cmds(('cmd1', ('cmd1', '--some-parameter')),
+ 'cmd2', 'cmd3'))
+ def test_something_that_needs_cmd1_and_cmd2(self):
+ ...
+ """
+
+ for cmd in cmds:
+ if isinstance(cmd, str):
+ cmd = (cmd,)
+
+ ok, errstr = has_cmd(*cmd)
+ if not ok:
+ return (False, errstr)
+
+ return (True, '')
+
+def run_cmd(args):
+ subp = subprocess.Popen(args,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ universal_newlines=True)
+ stdout, stderr = subp.communicate()
+ ret = subp.returncode
+
+ return (stdout, stderr, ret)
+
+def is_readable_executable_file(path):
+ return os.path.isfile(path) and os.access(path, os.R_OK | os.X_OK)
+
+def _console_interaction(test, success_message, failure_message,
+ send_string, keep_sending=False, vm=None):
+ assert not keep_sending or send_string
+ if vm is None:
+ vm = test.vm
+ console = vm.console_file
+ console_logger = logging.getLogger('console')
+ while True:
+ if send_string:
+ vm.console_socket.sendall(send_string.encode())
+ if not keep_sending:
+ send_string = None # send only once
+ try:
+ msg = console.readline().decode().strip()
+ except UnicodeDecodeError:
+ msg = None
+ if not msg:
+ continue
+ console_logger.debug(msg)
+ if success_message is None or success_message in msg:
+ break
+ if failure_message and failure_message in msg:
+ console.close()
+ fail = 'Failure message found in console: "%s". Expected: "%s"' % \
+ (failure_message, success_message)
+ test.fail(fail)
+
+def interrupt_interactive_console_until_pattern(test, success_message,
+ failure_message=None,
+ interrupt_string='\r'):
+ """
+ Keep sending a string to interrupt a console prompt, while logging the
+ console output. Typical use case is to break a boot loader prompt, such:
+
+ Press a key within 5 seconds to interrupt boot process.
+ 5
+ 4
+ 3
+ 2
+ 1
+ Booting default image...
+
+ :param test: a test containing a VM that will have its console
+ read and probed for a success or failure message
+ :type test: :class:`qemu_test.QemuSystemTest`
+ :param success_message: if this message appears, test succeeds
+ :param failure_message: if this message appears, test fails
+ :param interrupt_string: a string to send to the console before trying
+ to read a new line
+ """
+ _console_interaction(test, success_message, failure_message,
+ interrupt_string, True)
+
+def wait_for_console_pattern(test, success_message, failure_message=None,
+ vm=None):
+ """
+ Waits for messages to appear on the console, while logging the content
+
+ :param test: a test containing a VM that will have its console
+ read and probed for a success or failure message
+ :type test: :class:`qemu_test.QemuSystemTest`
+ :param success_message: if this message appears, test succeeds
+ :param failure_message: if this message appears, test fails
+ """
+ _console_interaction(test, success_message, failure_message, None, vm=vm)
+
+def exec_command(test, command):
+ """
+ Send a command to a console (appending CRLF characters), while logging
+ the content.
+
+ :param test: a test containing a VM.
+ :type test: :class:`qemu_test.QemuSystemTest`
+ :param command: the command to send
+ :type command: str
+ """
+ _console_interaction(test, None, None, command + '\r')
+
+def exec_command_and_wait_for_pattern(test, command,
+ success_message, failure_message=None):
+ """
+ Send a command to a console (appending CRLF characters), then wait
+ for success_message to appear on the console, while logging the.
+ content. Mark the test as failed if failure_message is found instead.
+
+ :param test: a test containing a VM that will have its console
+ read and probed for a success or failure message
+ :type test: :class:`qemu_test.QemuSystemTest`
+ :param command: the command to send
+ :param success_message: if this message appears, test succeeds
+ :param failure_message: if this message appears, test fails
+ """
+ _console_interaction(test, success_message, failure_message, command + '\r')
+
+class QemuBaseTest(unittest.TestCase):
+
+ qemu_bin = os.getenv('QEMU_TEST_QEMU_BINARY')
+
+ BUILD_DIR = _build_dir()
+
+ workdir = None
+ log = logging.getLogger('qemu-test')
+
+ def setUp(self, bin_prefix):
+ self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
+
+ self.workdir = os.path.join(self.BUILD_DIR, 'tests/functional')
+ self.workdir = os.path.join(self.workdir, self.id())
+ if not os.path.exists(self.workdir):
+ os.makedirs(self.workdir)
+
+ def main():
+ tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
+ test_output_log = pycotap.LogMode.LogToError)
+ unittest.main(testRunner = tr)
+
+
+class QemuSystemTest(QemuBaseTest):
+ """Facilitates system emulation tests."""
+
+ cpu = None
+ machine = None
+ _machinehelp = None
+
+ def setUp(self):
+ self._vms = {}
+
+ super().setUp('qemu-system-')
+
+ def set_machine(self, machinename):
+ # TODO: We should use QMP to get the list of available machines
+ if not self._machinehelp:
+ self._machinehelp = run_cmd([self.qemu_bin, '-M', 'help'])[0];
+ if self._machinehelp.find(machinename) < 0:
+ self.skipTest('no support for machine ' + machinename)
+ self.machine = machinename
+
+ def require_accelerator(self, accelerator):
+ """
+ Requires an accelerator to be available for the test to continue
+
+ It takes into account the currently set qemu binary.
+
+ If the check fails, the test is canceled. If the check itself
+ for the given accelerator is not available, the test is also
+ canceled.
+
+ :param accelerator: name of the accelerator, such as "kvm" or "tcg"
+ :type accelerator: str
+ """
+ checker = {'tcg': tcg_available,
+ 'kvm': kvm_available}.get(accelerator)
+ if checker is None:
+ self.skipTest("Don't know how to check for the presence "
+ "of accelerator %s" % accelerator)
+ if not checker(qemu_bin=self.qemu_bin):
+ self.skipTest("%s accelerator does not seem to be "
+ "available" % accelerator)
+
+ def require_netdev(self, netdevname):
+ netdevhelp = run_cmd([self.qemu_bin,
+ '-M', 'none', '-netdev', 'help'])[0];
+ if netdevhelp.find('\n' + netdevname + '\n') < 0:
+ self.skipTest('no support for " + netdevname + " networking')
+
+ def require_device(self, devicename):
+ devhelp = run_cmd([self.qemu_bin,
+ '-M', 'none', '-device', 'help'])[0];
+ if devhelp.find(devicename) < 0:
+ self.skipTest('no support for device ' + devicename)
+
+ def _new_vm(self, name, *args):
+ vm = QEMUMachine(self.qemu_bin, base_temp_dir=self.workdir)
+ self.log.debug('QEMUMachine "%s" created', name)
+ self.log.debug('QEMUMachine "%s" temp_dir: %s', name, vm.temp_dir)
+ self.log.debug('QEMUMachine "%s" log_dir: %s', name, vm.log_dir)
+ if args:
+ vm.add_args(*args)
+ return vm
+
+ @property
+ def vm(self):
+ return self.get_vm(name='default')
+
+ def get_vm(self, *args, name=None):
+ if not name:
+ name = str(uuid.uuid4())
+ if self._vms.get(name) is None:
+ self._vms[name] = self._new_vm(name, *args)
+ if self.cpu is not None:
+ self._vms[name].add_args('-cpu', self.cpu)
+ if self.machine is not None:
+ self._vms[name].set_machine(self.machine)
+ return self._vms[name]
+
+ def set_vm_arg(self, arg, value):
+ """
+ Set an argument to list of extra arguments to be given to the QEMU
+ binary. If the argument already exists then its value is replaced.
+
+ :param arg: the QEMU argument, such as "-cpu" in "-cpu host"
+ :type arg: str
+ :param value: the argument value, such as "host" in "-cpu host"
+ :type value: str
+ """
+ if not arg or not value:
+ return
+ if arg not in self.vm.args:
+ self.vm.args.extend([arg, value])
+ else:
+ idx = self.vm.args.index(arg) + 1
+ if idx < len(self.vm.args):
+ self.vm.args[idx] = value
+ else:
+ self.vm.args.append(value)
+
+ def tearDown(self):
+ for vm in self._vms.values():
+ vm.shutdown()
+ super().tearDown()
--
2.45.2
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 02/11] tests/functional: Convert simple avocado tests into standalone python tests
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
2024-07-16 11:26 ` [PATCH 01/11] tests/functional: Add base classes for the upcoming pytest-based tests Thomas Huth
@ 2024-07-16 11:26 ` Thomas Huth
2024-07-16 18:14 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 03/11] tests/functional: Convert avocado tests that just need a small adjustment Thomas Huth
` (11 subsequent siblings)
13 siblings, 1 reply; 36+ messages in thread
From: Thomas Huth @ 2024-07-16 11:26 UTC (permalink / raw)
To: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé
Cc: Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
These test are rather simple and don't need any modifications apart
from adjusting the "from avocado_qemu" line. To ease debugging, make
the files executable and add a shebang line and Python '__main__'
handling, too, so that these tests can now be run by executing them
directly.
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
.../test_cpu_queries.py} | 7 ++-
.../test_empty_cpu_model.py} | 7 ++-
.../test_mem_addr_space.py} | 53 +++----------------
.../test_pc_cpu_hotplug_props.py} | 11 ++--
.../test_virtio_version.py} | 8 +--
5 files changed, 29 insertions(+), 57 deletions(-)
rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%)
mode change 100644 => 100755
rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
mode change 100644 => 100755
rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (92%)
mode change 100644 => 100755
rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
mode change 100644 => 100755
rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
mode change 100644 => 100755
diff --git a/tests/avocado/cpu_queries.py b/tests/functional/test_cpu_queries.py
old mode 100644
new mode 100755
similarity index 89%
rename from tests/avocado/cpu_queries.py
rename to tests/functional/test_cpu_queries.py
index d3faa14720..97dd600c01
--- a/tests/avocado/cpu_queries.py
+++ b/tests/functional/test_cpu_queries.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# Sanity check of query-cpu-* results
#
# Copyright (c) 2019 Red Hat, Inc.
@@ -8,7 +10,7 @@
# This work is licensed under the terms of the GNU GPL, version 2 or
# later. See the COPYING file in the top-level directory.
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
class QueryCPUModelExpansion(QemuSystemTest):
"""
@@ -33,3 +35,6 @@ def test(self):
e = self.vm.cmd('query-cpu-model-expansion', model=model,
type='full')
self.assertEqual(e['model']['name'], c['name'])
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
diff --git a/tests/avocado/empty_cpu_model.py b/tests/functional/test_empty_cpu_model.py
old mode 100644
new mode 100755
similarity index 84%
rename from tests/avocado/empty_cpu_model.py
rename to tests/functional/test_empty_cpu_model.py
index d906ef3d3c..0081b06d85
--- a/tests/avocado/empty_cpu_model.py
+++ b/tests/functional/test_empty_cpu_model.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# Check for crash when using empty -cpu option
#
# Copyright (c) 2019 Red Hat, Inc.
@@ -7,7 +9,7 @@
#
# This work is licensed under the terms of the GNU GPL, version 2 or
# later. See the COPYING file in the top-level directory.
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
class EmptyCPUModel(QemuSystemTest):
def test(self):
@@ -17,3 +19,6 @@ def test(self):
self.vm.wait()
self.assertEqual(self.vm.exitcode(), 1, "QEMU exit code should be 1")
self.assertRegex(self.vm.get_log(), r'-cpu option cannot be empty')
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
diff --git a/tests/avocado/mem-addr-space-check.py b/tests/functional/test_mem_addr_space.py
old mode 100644
new mode 100755
similarity index 92%
rename from tests/avocado/mem-addr-space-check.py
rename to tests/functional/test_mem_addr_space.py
index 85541ea051..bb0cf062ca
--- a/tests/avocado/mem-addr-space-check.py
+++ b/tests/functional/test_mem_addr_space.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# Check for crash when using memory beyond the available guest processor
# address space.
#
@@ -8,8 +10,7 @@
#
# SPDX-License-Identifier: GPL-2.0-or-later
-from avocado_qemu import QemuSystemTest
-import signal
+from qemu_test import QemuSystemTest
import time
class MemAddrCheck(QemuSystemTest):
@@ -23,9 +24,6 @@ class MemAddrCheck(QemuSystemTest):
# for all 32-bit cases, pci64_hole_size is 0.
def test_phybits_low_pse36(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
With pse36 feature ON, a processor has 36 bits of addressing. So it can
access up to a maximum of 64GiB of memory. Memory hotplug region begins
at 4 GiB boundary when "above_4g_mem_size" is 0 (this would be true when
@@ -53,9 +51,6 @@ def test_phybits_low_pse36(self):
def test_phybits_low_pae(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
With pae feature ON, a processor has 36 bits of addressing. So it can
access up to a maximum of 64GiB of memory. Rest is the same as the case
with pse36 above.
@@ -73,9 +68,6 @@ def test_phybits_low_pae(self):
def test_phybits_ok_pentium_pse36(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
Setting maxmem to 59.5G and making sure that QEMU can start with the
same options as the failing case above with pse36 cpu feature.
"""
@@ -92,9 +84,6 @@ def test_phybits_ok_pentium_pse36(self):
def test_phybits_ok_pentium_pae(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
Test is same as above but now with pae cpu feature turned on.
Setting maxmem to 59.5G and making sure that QEMU can start fine
with the same options as the case above.
@@ -112,9 +101,6 @@ def test_phybits_ok_pentium_pae(self):
def test_phybits_ok_pentium2(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
Pentium2 has 36 bits of addressing, so its same as pentium
with pse36 ON.
"""
@@ -131,9 +117,6 @@ def test_phybits_ok_pentium2(self):
def test_phybits_low_nonpse36(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
Pentium processor has 32 bits of addressing without pse36 or pae
so it can access physical address up to 4 GiB. Setting maxmem to
4 GiB should make QEMU fail to start with "phys-bits too low"
@@ -154,9 +137,6 @@ def test_phybits_low_nonpse36(self):
# now lets test some 64-bit CPU cases.
def test_phybits_low_tcg_q35_70_amd(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
For q35 7.1 machines and above, there is a HT window that starts at
1024 GiB and ends at 1 TiB - 1. If the max GPA falls in this range,
"above_4G" memory is adjusted to start at 1 TiB boundary for AMD cpus
@@ -183,9 +163,6 @@ def test_phybits_low_tcg_q35_70_amd(self):
def test_phybits_low_tcg_q35_71_amd(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
AMD_HT_START is defined to be at 1012 GiB. So for q35 machines
version > 7.0 and AMD cpus, instead of 1024 GiB limit for 40 bit
processor address space, it has to be 1012 GiB , that is 12 GiB
@@ -206,9 +183,6 @@ def test_phybits_low_tcg_q35_71_amd(self):
def test_phybits_ok_tcg_q35_70_amd(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
Same as q35-7.0 AMD case except that here we check that QEMU can
successfully start when maxmem is < 988G.
"""
@@ -225,9 +199,6 @@ def test_phybits_ok_tcg_q35_70_amd(self):
def test_phybits_ok_tcg_q35_71_amd(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
Same as q35-7.1 AMD case except that here we check that QEMU can
successfully start when maxmem is < 976G.
"""
@@ -244,9 +215,6 @@ def test_phybits_ok_tcg_q35_71_amd(self):
def test_phybits_ok_tcg_q35_71_intel(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
Same parameters as test_phybits_low_tcg_q35_71_amd() but use
Intel cpu instead. QEMU should start fine in this case as
"above_4G" memory starts at 4G.
@@ -265,9 +233,6 @@ def test_phybits_ok_tcg_q35_71_intel(self):
def test_phybits_low_tcg_q35_71_amd_41bits(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
AMD processor with 41 bits. Max cpu hw address = 2 TiB.
By setting maxram above 1012 GiB - 32 GiB - 4 GiB = 976 GiB, we can
force "above_4G" memory to start at 1 TiB for q35-7.1 machines
@@ -292,9 +257,6 @@ def test_phybits_low_tcg_q35_71_amd_41bits(self):
def test_phybits_ok_tcg_q35_71_amd_41bits(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
AMD processor with 41 bits. Max cpu hw address = 2 TiB.
Same as above but by setting maxram between 976 GiB and 992 Gib,
QEMU should start fine.
@@ -313,9 +275,6 @@ def test_phybits_ok_tcg_q35_71_amd_41bits(self):
def test_phybits_low_tcg_q35_intel_cxl(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
cxl memory window starts after memory device range. Here, we use 1 GiB
of cxl window memory. 4G_mem end aligns at 4G. pci64_hole is 32 GiB and
starts after the cxl memory window.
@@ -336,9 +295,6 @@ def test_phybits_low_tcg_q35_intel_cxl(self):
def test_phybits_ok_tcg_q35_intel_cxl(self):
"""
- :avocado: tags=machine:q35
- :avocado: tags=arch:x86_64
-
Same as above but here we do not reserve any cxl memory window. Hence,
with the exact same parameters as above, QEMU should start fine even
with cxl enabled.
@@ -353,3 +309,6 @@ def test_phybits_ok_tcg_q35_intel_cxl(self):
time.sleep(self.DELAY_Q35_BOOT_SEQUENCE)
self.vm.shutdown()
self.assertNotRegex(self.vm.get_log(), r'phys-bits too low')
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
diff --git a/tests/avocado/pc_cpu_hotplug_props.py b/tests/functional/test_pc_cpu_hotplug_props.py
old mode 100644
new mode 100755
similarity index 90%
rename from tests/avocado/pc_cpu_hotplug_props.py
rename to tests/functional/test_pc_cpu_hotplug_props.py
index 4bd3e02665..9d5a37cb17
--- a/tests/avocado/pc_cpu_hotplug_props.py
+++ b/tests/functional/test_pc_cpu_hotplug_props.py
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
#
# Ensure CPU die-id can be omitted on -device
#
@@ -20,16 +21,16 @@
# License along with this library; if not, see <http://www.gnu.org/licenses/>.
#
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
class OmittedCPUProps(QemuSystemTest):
- """
- :avocado: tags=arch:x86_64
- :avocado: tags=cpu:qemu64
- """
+
def test_no_die_id(self):
self.vm.add_args('-nodefaults', '-S')
self.vm.add_args('-smp', '1,sockets=2,cores=2,threads=2,maxcpus=8')
self.vm.add_args('-device', 'qemu64-x86_64-cpu,socket-id=1,core-id=0,thread-id=0')
self.vm.launch()
self.assertEqual(len(self.vm.cmd('query-cpus-fast')), 2)
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
diff --git a/tests/avocado/virtio_version.py b/tests/functional/test_virtio_version.py
old mode 100644
new mode 100755
similarity index 98%
rename from tests/avocado/virtio_version.py
rename to tests/functional/test_virtio_version.py
index afe5e828b5..eb23060564
--- a/tests/avocado/virtio_version.py
+++ b/tests/functional/test_virtio_version.py
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
"""
Check compatibility of virtio device types
"""
@@ -12,7 +13,7 @@
import os
from qemu.machine import QEMUMachine
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
# Virtio Device IDs:
VIRTIO_NET = 1
@@ -60,8 +61,6 @@ class VirtioVersionCheck(QemuSystemTest):
Check if virtio-version-specific device types result in the
same device tree created by `disable-modern` and
`disable-legacy`.
-
- :avocado: tags=arch:x86_64
"""
# just in case there are failures, show larger diff:
@@ -173,3 +172,6 @@ def test_modern_only_devs(self):
self.check_modern_only('virtio-mouse-pci', VIRTIO_INPUT)
self.check_modern_only('virtio-tablet-pci', VIRTIO_INPUT)
self.check_modern_only('virtio-keyboard-pci', VIRTIO_INPUT)
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
--
2.45.2
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 03/11] tests/functional: Convert avocado tests that just need a small adjustment
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
2024-07-16 11:26 ` [PATCH 01/11] tests/functional: Add base classes for the upcoming pytest-based tests Thomas Huth
2024-07-16 11:26 ` [PATCH 02/11] tests/functional: Convert simple avocado tests into standalone python tests Thomas Huth
@ 2024-07-16 11:26 ` Thomas Huth
2024-07-16 18:15 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 04/11] tests/functional: Add python-based tests to the meson build system Thomas Huth
` (10 subsequent siblings)
13 siblings, 1 reply; 36+ messages in thread
From: Thomas Huth @ 2024-07-16 11:26 UTC (permalink / raw)
To: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé
Cc: Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
These simple tests can be converted to stand-alone tests quite easily,
e.g. by just setting the machine to 'none' now manually or by adding
"-cpu" command line parameters, since we don't support the corresponding
avocado tags in the new python test framework.
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
.../test_info_usernet.py} | 11 ++-
.../test_ppc_74xx.py} | 74 ++++++++-----------
.../version.py => functional/test_version.py} | 13 ++--
3 files changed, 47 insertions(+), 51 deletions(-)
rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
mode change 100644 => 100755
rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
mode change 100644 => 100755
rename tests/{avocado/version.py => functional/test_version.py} (78%)
mode change 100644 => 100755
diff --git a/tests/avocado/info_usernet.py b/tests/functional/test_info_usernet.py
old mode 100644
new mode 100755
similarity index 87%
rename from tests/avocado/info_usernet.py
rename to tests/functional/test_info_usernet.py
index e1aa7a6e0a..ded973253a
--- a/tests/avocado/info_usernet.py
+++ b/tests/functional/test_info_usernet.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# Test for the hmp command "info usernet"
#
# Copyright (c) 2021 Red Hat, Inc.
@@ -8,18 +10,16 @@
# This work is licensed under the terms of the GNU GPL, version 2 or
# later. See the COPYING file in the top-level directory.
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
from qemu.utils import get_info_usernet_hostfwd_port
class InfoUsernet(QemuSystemTest):
- """
- :avocado: tags=machine:none
- """
def test_hostfwd(self):
self.require_netdev('user')
+ self.machine = 'none'
self.vm.add_args('-netdev', 'user,id=vnet,hostfwd=:127.0.0.1:0-:22')
self.vm.launch()
res = self.vm.cmd('human-monitor-command',
@@ -31,3 +31,6 @@ def test_hostfwd(self):
self.assertGreater(port, 0,
('Found a redirected port that is not greater than'
' zero'))
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
diff --git a/tests/avocado/ppc_74xx.py b/tests/functional/test_ppc_74xx.py
old mode 100644
new mode 100755
similarity index 74%
rename from tests/avocado/ppc_74xx.py
rename to tests/functional/test_ppc_74xx.py
index f54757c243..5386016f26
--- a/tests/avocado/ppc_74xx.py
+++ b/tests/functional/test_ppc_74xx.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# Smoke tests for 74xx cpus (aka G4).
#
# Copyright (c) 2021, IBM Corp.
@@ -5,132 +7,120 @@
# This work is licensed under the terms of the GNU GPL, version 2 or
# later. See the COPYING file in the top-level directory.
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
class ppc74xxCpu(QemuSystemTest):
- """
- :avocado: tags=arch:ppc
- :avocado: tags=accel:tcg
- """
+
timeout = 5
def test_ppc_7400(self):
- """
- :avocado: tags=cpu:7400
- """
self.require_accelerator("tcg")
+ self.set_machine('g3beige')
self.vm.set_console()
+ self.vm.add_args('-cpu', '7400')
self.vm.launch()
wait_for_console_pattern(self, '>> OpenBIOS')
wait_for_console_pattern(self, '>> CPU type PowerPC,G4')
def test_ppc_7410(self):
- """
- :avocado: tags=cpu:7410
- """
self.require_accelerator("tcg")
+ self.set_machine('g3beige')
self.vm.set_console()
+ self.vm.add_args('-cpu', '7410')
self.vm.launch()
wait_for_console_pattern(self, '>> OpenBIOS')
wait_for_console_pattern(self, '>> CPU type PowerPC,74xx')
def test_ppc_7441(self):
- """
- :avocado: tags=cpu:7441
- """
self.require_accelerator("tcg")
+ self.set_machine('g3beige')
self.vm.set_console()
+ self.vm.add_args('-cpu', '7441')
self.vm.launch()
wait_for_console_pattern(self, '>> OpenBIOS')
wait_for_console_pattern(self, '>> CPU type PowerPC,G4')
def test_ppc_7445(self):
- """
- :avocado: tags=cpu:7445
- """
self.require_accelerator("tcg")
+ self.set_machine('g3beige')
self.vm.set_console()
+ self.vm.add_args('-cpu', '7445')
self.vm.launch()
wait_for_console_pattern(self, '>> OpenBIOS')
wait_for_console_pattern(self, '>> CPU type PowerPC,G4')
def test_ppc_7447(self):
- """
- :avocado: tags=cpu:7447
- """
self.require_accelerator("tcg")
+ self.set_machine('g3beige')
self.vm.set_console()
+ self.vm.add_args('-cpu', '7447')
self.vm.launch()
wait_for_console_pattern(self, '>> OpenBIOS')
wait_for_console_pattern(self, '>> CPU type PowerPC,G4')
def test_ppc_7447a(self):
- """
- :avocado: tags=cpu:7447a
- """
self.require_accelerator("tcg")
+ self.set_machine('g3beige')
self.vm.set_console()
+ self.vm.add_args('-cpu', '7447a')
self.vm.launch()
wait_for_console_pattern(self, '>> OpenBIOS')
wait_for_console_pattern(self, '>> CPU type PowerPC,G4')
def test_ppc_7448(self):
- """
- :avocado: tags=cpu:7448
- """
self.require_accelerator("tcg")
+ self.set_machine('g3beige')
self.vm.set_console()
+ self.vm.add_args('-cpu', '7448')
self.vm.launch()
wait_for_console_pattern(self, '>> OpenBIOS')
wait_for_console_pattern(self, '>> CPU type PowerPC,MPC86xx')
def test_ppc_7450(self):
- """
- :avocado: tags=cpu:7450
- """
self.require_accelerator("tcg")
+ self.set_machine('g3beige')
self.vm.set_console()
+ self.vm.add_args('-cpu', '7450')
self.vm.launch()
wait_for_console_pattern(self, '>> OpenBIOS')
wait_for_console_pattern(self, '>> CPU type PowerPC,G4')
def test_ppc_7451(self):
- """
- :avocado: tags=cpu:7451
- """
self.require_accelerator("tcg")
+ self.set_machine('g3beige')
self.vm.set_console()
+ self.vm.add_args('-cpu', '7451')
self.vm.launch()
wait_for_console_pattern(self, '>> OpenBIOS')
wait_for_console_pattern(self, '>> CPU type PowerPC,G4')
def test_ppc_7455(self):
- """
- :avocado: tags=cpu:7455
- """
self.require_accelerator("tcg")
+ self.set_machine('g3beige')
self.vm.set_console()
+ self.vm.add_args('-cpu', '7455')
self.vm.launch()
wait_for_console_pattern(self, '>> OpenBIOS')
wait_for_console_pattern(self, '>> CPU type PowerPC,G4')
def test_ppc_7457(self):
- """
- :avocado: tags=cpu:7457
- """
self.require_accelerator("tcg")
+ self.set_machine('g3beige')
self.vm.set_console()
+ self.vm.add_args('-cpu', '7457')
self.vm.launch()
wait_for_console_pattern(self, '>> OpenBIOS')
wait_for_console_pattern(self, '>> CPU type PowerPC,G4')
def test_ppc_7457a(self):
- """
- :avocado: tags=cpu:7457a
- """
self.require_accelerator("tcg")
+ self.set_machine('g3beige')
self.vm.set_console()
+ self.vm.add_args('-cpu', '7457a')
self.vm.launch()
wait_for_console_pattern(self, '>> OpenBIOS')
wait_for_console_pattern(self, '>> CPU type PowerPC,G4')
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
diff --git a/tests/avocado/version.py b/tests/functional/test_version.py
old mode 100644
new mode 100755
similarity index 78%
rename from tests/avocado/version.py
rename to tests/functional/test_version.py
index c6139568a1..5e566d76b1
--- a/tests/avocado/version.py
+++ b/tests/functional/test_version.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# Version check example test
#
# Copyright (c) 2018 Red Hat, Inc.
@@ -9,17 +11,18 @@
# later. See the COPYING file in the top-level directory.
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
class Version(QemuSystemTest):
- """
- :avocado: tags=quick
- :avocado: tags=machine:none
- """
+
def test_qmp_human_info_version(self):
+ self.machine = 'none'
self.vm.add_args('-nodefaults')
self.vm.launch()
res = self.vm.cmd('human-monitor-command',
command_line='info version')
self.assertRegex(res, r'^(\d+\.\d+\.\d)')
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
--
2.45.2
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 04/11] tests/functional: Add python-based tests to the meson build system
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
` (2 preceding siblings ...)
2024-07-16 11:26 ` [PATCH 03/11] tests/functional: Convert avocado tests that just need a small adjustment Thomas Huth
@ 2024-07-16 11:26 ` Thomas Huth
2024-07-16 15:15 ` Fabiano Rosas
2024-07-16 16:55 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 05/11] tests/functional: Implement fetch_asset() method for downloading assets Thomas Huth
` (9 subsequent siblings)
13 siblings, 2 replies; 36+ messages in thread
From: Thomas Huth @ 2024-07-16 11:26 UTC (permalink / raw)
To: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé
Cc: Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
Integrate the new python-based test framework with the meson build
system. Since these tests now require the pycotap module, make
sure that it gets installed in the venv.
The changes to the meson.build files are partly based on an earlier
patch by Ani Sinha (but heavily modified by Thomas Huth e.g. to use
pycotap for running the tests instead).
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
pythondeps.toml | 3 +-
tests/Makefile.include | 18 ++++++++-
tests/functional/meson.build | 75 ++++++++++++++++++++++++++++++++++++
tests/meson.build | 1 +
4 files changed, 95 insertions(+), 2 deletions(-)
create mode 100644 tests/functional/meson.build
diff --git a/pythondeps.toml b/pythondeps.toml
index f6e590fdd8..c018b4d74a 100644
--- a/pythondeps.toml
+++ b/pythondeps.toml
@@ -26,9 +26,10 @@ meson = { accepted = ">=1.1.0", installed = "1.2.3", canary = "meson" }
sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
-[avocado]
+[tests]
# Note that qemu.git/python/ is always implicitly installed.
# Prefer an LTS version when updating the accepted versions of
# avocado-framework, for example right now the limit is 92.x.
avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
pycdlib = { accepted = ">=1.11.0" }
+pycotap = { accepted = ">=1.1.0" }
diff --git a/tests/Makefile.include b/tests/Makefile.include
index d39d5dd6a4..2bdf607977 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -9,6 +9,8 @@ check-help:
@echo "Individual test suites:"
@echo " $(MAKE) check-qtest-TARGET Run qtest tests for given target"
@echo " $(MAKE) check-qtest Run qtest tests"
+ @echo " $(MAKE) check-functional Run python-based functional tests"
+ @echo " $(MAKE) check-functional-TARG Run functional tests for a given target"
@echo " $(MAKE) check-unit Run qobject tests"
@echo " $(MAKE) check-qapi-schema Run QAPI schema tests"
@echo " $(MAKE) check-block Run block tests"
@@ -111,7 +113,7 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
$(TESTS_VENV_TOKEN): $(SRC_PATH)/pythondeps.toml
$(call quiet-venv-pip,install -e "$(SRC_PATH)/python/")
- $(MKVENV_ENSUREGROUP) $< avocado
+ $(MKVENV_ENSUREGROUP) $< tests
$(call quiet-command, touch $@)
$(TESTS_RESULTS_DIR):
@@ -152,6 +154,20 @@ check-acceptance-deprecated-warning:
check-acceptance: check-acceptance-deprecated-warning | check-avocado
+# Make sure that pycotap is installed before running any functional tests:
+ifneq ($(filter check-func%,$(MAKECMDGOALS))$(filter check,$(MAKECMDGOALS)),)
+do-meson-check: check-venv
+endif
+
+FUNCTIONAL_TARGETS=$(patsubst %-softmmu,check-functional-%, $(filter %-softmmu,$(TARGETS)))
+.PHONY: $(FUNCTIONAL_TARGETS)
+$(FUNCTIONAL_TARGETS):
+ @make SPEED=thorough $(subst -functional,-func,$@)
+
+.PHONY: check-functional
+check-functional:
+ @make SPEED=thorough check-func check-func-quick
+
# Consolidated targets
.PHONY: check check-clean get-vm-images
diff --git a/tests/functional/meson.build b/tests/functional/meson.build
new file mode 100644
index 0000000000..11352b5bb5
--- /dev/null
+++ b/tests/functional/meson.build
@@ -0,0 +1,75 @@
+# QEMU functional tests:
+# Tests that are put in the 'quick' category are run by default during
+# 'make check'. Everything that should not be run during 'make check'
+# (e.g. tests that fetch assets from the internet) should be put into
+# the 'thorough' category instead.
+
+# Most tests run too slow with TCI enabled, so skip the functional tests there
+if get_option('tcg_interpreter')
+ subdir_done()
+endif
+
+# Timeouts for individual tests that can be slow e.g. with debugging enabled
+test_timeouts = {
+ 'ppc_74xx' : 90,
+}
+
+tests_generic = [
+ 'empty_cpu_model',
+ 'info_usernet',
+ 'version',
+]
+
+tests_ppc_quick = [
+ 'ppc_74xx',
+]
+
+tests_x86_64_quick = [
+ 'cpu_queries',
+ 'mem_addr_space',
+ 'pc_cpu_hotplug_props',
+ 'virtio_version',
+]
+
+foreach speed : ['quick', 'thorough']
+ foreach dir : target_dirs
+ if not dir.endswith('-softmmu')
+ continue
+ endif
+
+ target_base = dir.split('-')[0]
+ test_emulator = emulators['qemu-system-' + target_base]
+
+ if speed == 'quick'
+ suites = ['func-quick', 'func-' + target_base]
+ target_tests = get_variable('tests_' + target_base + '_quick', []) + tests_generic
+ else
+ suites = ['func-' + speed, 'func-' + target_base + '-' + speed, speed]
+ target_tests = get_variable('tests_' + target_base + '_' + speed, [])
+ endif
+
+ test_deps = roms
+ test_env = environment()
+ if have_tools
+ test_env.set('QEMU_TEST_QEMU_IMG', meson.global_build_root() / 'qemu-img')
+ test_deps += [qemu_img]
+ endif
+ test_env.set('QEMU_TEST_QEMU_BINARY',
+ meson.global_build_root() / 'qemu-system-' + target_base)
+ test_env.set('QEMU_BUILD_ROOT', meson.project_build_root())
+ test_env.set('PYTHONPATH', meson.project_source_root() / 'python:' +
+ meson.current_source_dir())
+
+ foreach test : target_tests
+ test('func-@0@/@1@'.format(target_base, test),
+ python,
+ depends: [test_deps, test_emulator, emulator_modules],
+ env: test_env,
+ args: [meson.current_source_dir() / 'test_' + test + '.py'],
+ protocol: 'tap',
+ timeout: test_timeouts.get(test, 60),
+ priority: test_timeouts.get(test, 60),
+ suite: suites)
+ endforeach
+ endforeach
+endforeach
diff --git a/tests/meson.build b/tests/meson.build
index acb6807094..3345ad2098 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -85,3 +85,4 @@ subdir('unit')
subdir('qapi-schema')
subdir('qtest')
subdir('migration')
+subdir('functional')
--
2.45.2
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 05/11] tests/functional: Implement fetch_asset() method for downloading assets
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
` (3 preceding siblings ...)
2024-07-16 11:26 ` [PATCH 04/11] tests/functional: Add python-based tests to the meson build system Thomas Huth
@ 2024-07-16 11:26 ` Thomas Huth
2024-07-16 18:17 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 06/11] tests/functional: Convert some tests that download files via fetch_asset() Thomas Huth
` (8 subsequent siblings)
13 siblings, 1 reply; 36+ messages in thread
From: Thomas Huth @ 2024-07-16 11:26 UTC (permalink / raw)
To: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé
Cc: Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
In the new python test framework, we cannot use the fetch_asset()
function from Avocado anymore, so we have to provide our own
implementation now instead. Thus add such a function based on the
urllib python module for this purpose.
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
tests/functional/qemu_test/__init__.py | 36 ++++++++++++++++++++++++++
1 file changed, 36 insertions(+)
diff --git a/tests/functional/qemu_test/__init__.py b/tests/functional/qemu_test/__init__.py
index e73705d40a..77c3b14d34 100644
--- a/tests/functional/qemu_test/__init__.py
+++ b/tests/functional/qemu_test/__init__.py
@@ -11,6 +11,8 @@
# This work is licensed under the terms of the GNU GPL, version 2 or
# later. See the COPYING file in the top-level directory.
+import hashlib
+import urllib.request
import logging
import os
import pycotap
@@ -23,6 +25,7 @@
import unittest
from pathlib import Path
+from shutil import copyfileobj
from qemu.machine import QEMUMachine
from qemu.utils import (get_info_usernet_hostfwd_port, kvm_available,
tcg_available)
@@ -215,6 +218,39 @@ def setUp(self, bin_prefix):
if not os.path.exists(self.workdir):
os.makedirs(self.workdir)
+ def check_hash(self, file_name, expected_hash):
+ if not expected_hash:
+ return True
+ if len(expected_hash) == 40:
+ sum_prog = 'sha1sum'
+ elif len(expected_hash) == 64:
+ sum_prog = 'sha256sum'
+ elif len(expected_hash) == 128:
+ sum_prog = 'sha512sum'
+ else:
+ raise Exception("unknown hash type")
+ checksum = subprocess.check_output([sum_prog, file_name]).split()[0]
+ return expected_hash == checksum.decode("utf-8")
+
+ def fetch_asset(self, url, asset_hash):
+ cache_dir = os.path.expanduser("~/.cache/qemu/download")
+ if not os.path.exists(cache_dir):
+ os.makedirs(cache_dir)
+ fname = os.path.join(cache_dir,
+ hashlib.sha256(url.encode("utf-8")).hexdigest())
+ if os.path.exists(fname) and self.check_hash(fname, asset_hash):
+ return fname
+ self.log.info("Downloading %s to %s...", url, fname)
+ dl_fname = fname + ".download"
+ with urllib.request.urlopen(url) as src:
+ with open(dl_fname, "wb+") as dst:
+ copyfileobj(src, dst)
+ if not self.check_hash(dl_fname, asset_hash):
+ os.remove(dl_fname)
+ raise Exception("Hash of " + url + " does not match")
+ os.rename(dl_fname, fname)
+ return fname
+
def main():
tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
test_output_log = pycotap.LogMode.LogToError)
--
2.45.2
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 06/11] tests/functional: Convert some tests that download files via fetch_asset()
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
` (4 preceding siblings ...)
2024-07-16 11:26 ` [PATCH 05/11] tests/functional: Implement fetch_asset() method for downloading assets Thomas Huth
@ 2024-07-16 11:26 ` Thomas Huth
2024-07-16 18:19 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 07/11] tests/functional: Add a function for extracting files from an archive Thomas Huth
` (7 subsequent siblings)
13 siblings, 1 reply; 36+ messages in thread
From: Thomas Huth @ 2024-07-16 11:26 UTC (permalink / raw)
To: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé
Cc: Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
Now that we've got a working fetch_asset() function, we can convert
some Avocado tests that use this function for downloading their
required files.
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
MAINTAINERS | 12 +++----
tests/functional/meson.build | 25 +++++++++++++++
.../test_arm_n8x0.py} | 25 +++++++--------
.../test_avr_mega2560.py} | 11 ++++---
.../test_loongarch64_virt.py} | 16 ++++++----
.../test_mips64el_loongson3v.py} | 26 +++++++--------
.../test_netdev_ethtool.py} | 32 ++++++-------------
.../ppc_405.py => functional/test_ppc_405.py} | 19 ++++++-----
8 files changed, 89 insertions(+), 77 deletions(-)
rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (71%)
mode change 100644 => 100755
rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (90%)
mode change 100644 => 100755
rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (89%)
mode change 100644 => 100755
rename tests/{avocado/machine_mips_loongson3v.py => functional/test_mips64el_loongson3v.py} (55%)
mode change 100644 => 100755
rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (81%)
mode change 100644 => 100755
rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
mode change 100644 => 100755
diff --git a/MAINTAINERS b/MAINTAINERS
index 7d9811458c..533e22b548 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -221,7 +221,7 @@ S: Maintained
F: docs/system/target-avr.rst
F: gdb-xml/avr-cpu.xml
F: target/avr/
-F: tests/avocado/machine_avr6.py
+F: tests/functional/test_avr_mega2560.py
CRIS TCG CPUs
M: Edgar E. Iglesias <edgar.iglesias@gmail.com>
@@ -264,7 +264,7 @@ M: Song Gao <gaosong@loongson.cn>
S: Maintained
F: target/loongarch/
F: tests/tcg/loongarch64/
-F: tests/avocado/machine_loongarch.py
+F: tests/functional/test_loongarch64_virt.py
M68K TCG CPUs
M: Laurent Vivier <laurent@vivier.eu>
@@ -883,7 +883,7 @@ F: include/hw/display/blizzard.h
F: include/hw/input/lm832x.h
F: include/hw/input/tsc2xxx.h
F: include/hw/misc/cbus.h
-F: tests/avocado/machine_arm_n8x0.py
+F: tests/functional/test_arm_n8x0.py
F: docs/system/arm/nseries.rst
Palm
@@ -1392,7 +1392,7 @@ F: hw/mips/loongson3_bootp.h
F: hw/mips/loongson3_virt.c
F: include/hw/intc/loongson_ipi.h
F: include/hw/intc/loongson_liointc.h
-F: tests/avocado/machine_mips_loongson3v.py
+F: tests/functional/test_mips64el_loongson3v.py
Boston
M: Paul Burton <paulburton@kernel.org>
@@ -1418,7 +1418,7 @@ PowerPC Machines
L: qemu-ppc@nongnu.org
S: Orphan
F: hw/ppc/ppc405*
-F: tests/avocado/ppc_405.py
+F: tests/functional/test_ppc_405.py
Bamboo
L: qemu-ppc@nongnu.org
@@ -2483,7 +2483,7 @@ R: Sriram Yagnaraman <sriram.yagnaraman@ericsson.com>
S: Maintained
F: docs/system/devices/igb.rst
F: hw/net/igb*
-F: tests/avocado/netdev-ethtool.py
+F: tests/functional/test_netdev_ethtool.py
F: tests/qtest/igb-test.c
F: tests/qtest/libqos/igb.c
diff --git a/tests/functional/meson.build b/tests/functional/meson.build
index 11352b5bb5..947f9ac74f 100644
--- a/tests/functional/meson.build
+++ b/tests/functional/meson.build
@@ -11,6 +11,7 @@ endif
# Timeouts for individual tests that can be slow e.g. with debugging enabled
test_timeouts = {
+ 'netdev_ethtool' : 180,
'ppc_74xx' : 90,
}
@@ -20,10 +21,30 @@ tests_generic = [
'version',
]
+tests_arm_thorough = [
+ 'arm_n8x0',
+]
+
+tests_avr_thorough = [
+ 'avr_mega2560',
+]
+
+tests_loongarch64_thorough = [
+ 'loongarch64_virt',
+]
+
+tests_mips64el_thorough = [
+ 'mips64el_loongson3v',
+]
+
tests_ppc_quick = [
'ppc_74xx',
]
+tests_ppc_thorough = [
+ 'ppc_405',
+]
+
tests_x86_64_quick = [
'cpu_queries',
'mem_addr_space',
@@ -31,6 +52,10 @@ tests_x86_64_quick = [
'virtio_version',
]
+tests_x86_64_thorough = [
+ 'netdev_ethtool',
+]
+
foreach speed : ['quick', 'thorough']
foreach dir : target_dirs
if not dir.endswith('-softmmu')
diff --git a/tests/avocado/machine_arm_n8x0.py b/tests/functional/test_arm_n8x0.py
old mode 100644
new mode 100755
similarity index 71%
rename from tests/avocado/machine_arm_n8x0.py
rename to tests/functional/test_arm_n8x0.py
index 12e9a6803b..d451c80a73
--- a/tests/avocado/machine_arm_n8x0.py
+++ b/tests/functional/test_arm_n8x0.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# Functional test that boots a Linux kernel and checks the console
#
# Copyright (c) 2020 Red Hat, Inc.
@@ -10,9 +12,9 @@
import os
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from unittest import skipUnless
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
class N8x0Machine(QemuSystemTest):
"""Boots the Linux kernel and checks that the console is operational"""
@@ -32,18 +34,15 @@ def __do_test_n8x0(self):
self.vm.launch()
wait_for_console_pattern(self, 'TSC2005 driver initializing')
- @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+ @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
def test_n800(self):
- """
- :avocado: tags=arch:arm
- :avocado: tags=machine:n800
- """
+ self.set_machine('n800')
self.__do_test_n8x0()
- @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+ @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
def test_n810(self):
- """
- :avocado: tags=arch:arm
- :avocado: tags=machine:n810
- """
+ self.set_machine('n810')
self.__do_test_n8x0()
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
diff --git a/tests/avocado/machine_avr6.py b/tests/functional/test_avr_mega2560.py
old mode 100644
new mode 100755
similarity index 90%
rename from tests/avocado/machine_avr6.py
rename to tests/functional/test_avr_mega2560.py
index 5485db79c6..b9b89367c6
--- a/tests/avocado/machine_avr6.py
+++ b/tests/functional/test_avr_mega2560.py
@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
#
# QEMU AVR integration tests
#
@@ -19,16 +20,12 @@
import time
-from avocado_qemu import QemuSystemTest
+from qemu_test import QemuSystemTest
class AVR6Machine(QemuSystemTest):
timeout = 5
def test_freertos(self):
- """
- :avocado: tags=arch:avr
- :avocado: tags=machine:arduino-mega-2560-v3
- """
"""
https://github.com/seharris/qemu-avr-tests/raw/master/free-rtos/Demo/AVR_ATMega2560_GCC/demo.elf
constantly prints out 'ABCDEFGHIJKLMNOPQRSTUVWXABCDEFGHIJKLMNOPQRSTUVWX'
@@ -39,6 +36,7 @@ def test_freertos(self):
rom_hash = '7eb521f511ca8f2622e0a3c5e8dd686efbb911d4'
rom_path = self.fetch_asset(rom_url, asset_hash=rom_hash)
+ self.set_machine('arduino-mega-2560-v3')
self.vm.add_args('-bios', rom_path)
self.vm.add_args('-nographic')
self.vm.launch()
@@ -48,3 +46,6 @@ def test_freertos(self):
self.assertIn('ABCDEFGHIJKLMNOPQRSTUVWXABCDEFGHIJKLMNOPQRSTUVWX',
self.vm.get_log())
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
diff --git a/tests/avocado/machine_loongarch.py b/tests/functional/test_loongarch64_virt.py
old mode 100644
new mode 100755
similarity index 89%
rename from tests/avocado/machine_loongarch.py
rename to tests/functional/test_loongarch64_virt.py
index 8de308f2d6..3dc8bb3516
--- a/tests/avocado/machine_loongarch.py
+++ b/tests/functional/test_loongarch64_virt.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# SPDX-License-Identifier: GPL-2.0-or-later
#
# LoongArch virt test.
@@ -5,9 +7,9 @@
# Copyright (c) 2023 Loongson Technology Corporation Limited
#
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import exec_command_and_wait_for_pattern
-from avocado_qemu import wait_for_console_pattern
+from qemu_test import QemuSystemTest
+from qemu_test import exec_command_and_wait_for_pattern
+from qemu_test import wait_for_console_pattern
class LoongArchMachine(QemuSystemTest):
KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
@@ -21,10 +23,7 @@ def wait_for_console_pattern(self, success_message, vm=None):
def test_loongarch64_devices(self):
- """
- :avocado: tags=arch:loongarch64
- :avocado: tags=machine:virt
- """
+ self.set_machine('virt')
kernel_url = ('https://github.com/yangxiaojuan-loongson/qemu-binary/'
'releases/download/2024-05-30/vmlinuz.efi')
@@ -56,3 +55,6 @@ def test_loongarch64_devices(self):
self.wait_for_console_pattern('Run /sbin/init as init process')
exec_command_and_wait_for_pattern(self, 'cat /proc/cpuinfo',
'processor : 3')
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
diff --git a/tests/avocado/machine_mips_loongson3v.py b/tests/functional/test_mips64el_loongson3v.py
old mode 100644
new mode 100755
similarity index 55%
rename from tests/avocado/machine_mips_loongson3v.py
rename to tests/functional/test_mips64el_loongson3v.py
index 5194cf18c9..f8d40b48a9
--- a/tests/avocado/machine_mips_loongson3v.py
+++ b/tests/functional/test_mips64el_loongson3v.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# Functional tests for the Generic Loongson-3 Platform.
#
# Copyright (c) 2021 Jiaxun Yang <jiaxun.yang@flygoat.com>
@@ -10,30 +12,26 @@
import os
import time
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from unittest import skipUnless
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
class MipsLoongson3v(QemuSystemTest):
timeout = 60
- @skipUnless(os.getenv('AVOCADO_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
+ @skipUnless(os.getenv('QEMU_TEST_ALLOW_UNTRUSTED_CODE'), 'untrusted code')
def test_pmon_serial_console(self):
- """
- :avocado: tags=arch:mips64el
- :avocado: tags=endian:little
- :avocado: tags=machine:loongson3-virt
- :avocado: tags=cpu:Loongson-3A1000
- :avocado: tags=device:liointc
- :avocado: tags=device:goldfish_rtc
- """
+ self.set_machine('loongson3-virt')
- pmon_hash = '7c8b45dd81ccfc55ff28f5aa267a41c3'
+ pmon_hash = 'fcdf6bb2cb7885a4a62f31fcb0d5e368bac7b6cea28f40c6dfa678af22fea20a'
pmon_path = self.fetch_asset('https://github.com/loongson-community/pmon/'
'releases/download/20210112/pmon-3avirt.bin',
- asset_hash=pmon_hash, algorithm='md5')
+ asset_hash=pmon_hash)
self.vm.set_console()
self.vm.add_args('-bios', pmon_path)
self.vm.launch()
wait_for_console_pattern(self, 'CPU GODSON3 BogoMIPS:')
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
diff --git a/tests/avocado/netdev-ethtool.py b/tests/functional/test_netdev_ethtool.py
old mode 100644
new mode 100755
similarity index 81%
rename from tests/avocado/netdev-ethtool.py
rename to tests/functional/test_netdev_ethtool.py
index 5f33288f81..341c087e36
--- a/tests/avocado/netdev-ethtool.py
+++ b/tests/functional/test_netdev_ethtool.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# ethtool tests for emulated network devices
#
# This test leverages ethtool's --test sequence to validate network
@@ -5,15 +7,11 @@
#
# SPDX-License-Identifier: GPL-2.0-or-late
-from avocado import skip
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
+from unittest import skip
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
class NetDevEthtool(QemuSystemTest):
- """
- :avocado: tags=arch:x86_64
- :avocado: tags=machine:q35
- """
# Runs in about 17s under KVM, 19s under TCG, 25s under GCOV
timeout = 45
@@ -25,11 +23,10 @@ def get_asset(self, name, sha1):
'kE4nCFLdQcoBF9t/download?'
'path=%2Fnetdev-ethtool&files=' )
url = base_url + name
- # use explicit name rather than failing to neatly parse the
- # URL into a unique one
- return self.fetch_asset(name=name, locations=(url), asset_hash=sha1)
+ return self.fetch_asset(url, asset_hash=sha1)
def common_test_code(self, netdev, extra_args=None):
+ self.set_machine('q35')
# This custom kernel has drivers for all the supported network
# devices we can emulate in QEMU
@@ -68,15 +65,9 @@ def common_test_code(self, netdev, extra_args=None):
self.vm.kill()
def test_igb(self):
- """
- :avocado: tags=device:igb
- """
self.common_test_code("igb")
def test_igb_nomsi(self):
- """
- :avocado: tags=device:igb
- """
self.common_test_code("igb", "pci=nomsi")
# It seems the other popular cards we model in QEMU currently fail
@@ -88,14 +79,11 @@ def test_igb_nomsi(self):
@skip("Incomplete reg 0x00178 support")
def test_e1000(self):
- """
- :avocado: tags=device:e1000
- """
self.common_test_code("e1000")
@skip("Incomplete reg 0x00178 support")
def test_i82550(self):
- """
- :avocado: tags=device:i82550
- """
self.common_test_code("i82550")
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
diff --git a/tests/avocado/ppc_405.py b/tests/functional/test_ppc_405.py
old mode 100644
new mode 100755
similarity index 73%
rename from tests/avocado/ppc_405.py
rename to tests/functional/test_ppc_405.py
index 4e7e01aa76..f2368ada80
--- a/tests/avocado/ppc_405.py
+++ b/tests/functional/test_ppc_405.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# Test that the U-Boot firmware boots on ppc 405 machines and check the console
#
# Copyright (c) 2021 Red Hat, Inc.
@@ -5,10 +7,9 @@
# This work is licensed under the terms of the GNU GPL, version 2 or
# later. See the COPYING file in the top-level directory.
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado_qemu import exec_command_and_wait_for_pattern
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from qemu_test import exec_command_and_wait_for_pattern
class Ppc405Machine(QemuSystemTest):
@@ -26,11 +27,9 @@ def do_test_ppc405(self):
exec_command_and_wait_for_pattern(self, 'reset', 'AMCC PowerPC 405EP')
def test_ppc_ref405ep(self):
- """
- :avocado: tags=arch:ppc
- :avocado: tags=machine:ref405ep
- :avocado: tags=cpu:405ep
- :avocado: tags=accel:tcg
- """
self.require_accelerator("tcg")
+ self.set_machine('ref405ep')
self.do_test_ppc405()
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
--
2.45.2
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 07/11] tests/functional: Add a function for extracting files from an archive
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
` (5 preceding siblings ...)
2024-07-16 11:26 ` [PATCH 06/11] tests/functional: Convert some tests that download files via fetch_asset() Thomas Huth
@ 2024-07-16 11:26 ` Thomas Huth
2024-07-16 18:20 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 08/11] tests/functional: Convert some avocado tests that needed avocado.utils.archive Thomas Huth
` (6 subsequent siblings)
13 siblings, 1 reply; 36+ messages in thread
From: Thomas Huth @ 2024-07-16 11:26 UTC (permalink / raw)
To: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé
Cc: Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
Some Avocado-based tests use the "archive" module from avocado.utils
to extract files from an archive. To be able to use these tests
without Avocado, we have to provide our own function for extracting
files. Fortunately, there is already the tarfile module that will
provide us with this functionality, so let's just add a nice wrapper
function around that.
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
tests/functional/qemu_test/utils.py | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)
create mode 100644 tests/functional/qemu_test/utils.py
diff --git a/tests/functional/qemu_test/utils.py b/tests/functional/qemu_test/utils.py
new file mode 100644
index 0000000000..4eb5e5d5e5
--- /dev/null
+++ b/tests/functional/qemu_test/utils.py
@@ -0,0 +1,21 @@
+# Utilities for python-based QEMU tests
+#
+# Copyright 2024 Red Hat, Inc.
+#
+# Authors:
+# Thomas Huth <thuth@redhat.com>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+import tarfile
+
+def archive_extract(archive, dest_dir, member=None):
+ with tarfile.open(archive) as tf:
+ if hasattr(tarfile, 'data_filter'):
+ tf.extraction_filter = getattr(tarfile, 'data_filter',
+ (lambda member, path: member))
+ if member:
+ tf.extract(member=member, path=dest_dir)
+ else:
+ tf.extractall(path=dest_dir)
--
2.45.2
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 08/11] tests/functional: Convert some avocado tests that needed avocado.utils.archive
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
` (6 preceding siblings ...)
2024-07-16 11:26 ` [PATCH 07/11] tests/functional: Add a function for extracting files from an archive Thomas Huth
@ 2024-07-16 11:26 ` Thomas Huth
2024-07-16 18:20 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 09/11] tests/functional: Set up logging Thomas Huth
` (5 subsequent siblings)
13 siblings, 1 reply; 36+ messages in thread
From: Thomas Huth @ 2024-07-16 11:26 UTC (permalink / raw)
To: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé
Cc: Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
Instead of using the "archive" module from avocado.utils, switch
these tests to use the new wrapper function that is based on the
"tarfile" module instead.
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
MAINTAINERS | 6 ++---
tests/functional/meson.build | 6 +++++
.../test_arm_canona1100.py} | 21 +++++++++-------
.../test_ppc_bamboo.py} | 23 ++++++++---------
.../test_sparc64_sun4u.py} | 25 +++++++++++--------
5 files changed, 46 insertions(+), 35 deletions(-)
rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (71%)
mode change 100644 => 100755
rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (75%)
mode change 100644 => 100755
rename tests/{avocado/machine_sparc64_sun4u.py => functional/test_sparc64_sun4u.py} (60%)
mode change 100644 => 100755
diff --git a/MAINTAINERS b/MAINTAINERS
index 533e22b548..025227954c 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -730,7 +730,7 @@ S: Odd Fixes
F: include/hw/arm/digic.h
F: hw/*/digic*
F: include/hw/*/digic*
-F: tests/avocado/machine_arm_canona1100.py
+F: tests/functional/test_arm_canona1100.py
F: docs/system/arm/digic.rst
Goldfish RTC
@@ -1425,7 +1425,7 @@ L: qemu-ppc@nongnu.org
S: Orphan
F: hw/ppc/ppc440_bamboo.c
F: hw/pci-host/ppc4xx_pci.c
-F: tests/avocado/ppc_bamboo.py
+F: tests/functional/test_ppc_bamboo.py
e500
L: qemu-ppc@nongnu.org
@@ -1711,7 +1711,7 @@ F: include/hw/pci-host/sabre.h
F: hw/pci-bridge/simba.c
F: include/hw/pci-bridge/simba.h
F: pc-bios/openbios-sparc64
-F: tests/avocado/machine_sparc64_sun4u.py
+F: tests/functional/test_sparc64_sun4u.py
Sun4v
M: Artyom Tarasenko <atar4qemu@gmail.com>
diff --git a/tests/functional/meson.build b/tests/functional/meson.build
index 947f9ac74f..c8fc9f6c07 100644
--- a/tests/functional/meson.build
+++ b/tests/functional/meson.build
@@ -22,6 +22,7 @@ tests_generic = [
]
tests_arm_thorough = [
+ 'arm_canona1100',
'arm_n8x0',
]
@@ -43,6 +44,11 @@ tests_ppc_quick = [
tests_ppc_thorough = [
'ppc_405',
+ 'ppc_bamboo',
+]
+
+tests_sparc64_thorough = [
+ 'sparc64_sun4u',
]
tests_x86_64_quick = [
diff --git a/tests/avocado/machine_arm_canona1100.py b/tests/functional/test_arm_canona1100.py
old mode 100644
new mode 100755
similarity index 71%
rename from tests/avocado/machine_arm_canona1100.py
rename to tests/functional/test_arm_canona1100.py
index a42d8b0f2b..de6169f2af
--- a/tests/avocado/machine_arm_canona1100.py
+++ b/tests/functional/test_arm_canona1100.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# Functional test that boots the canon-a1100 machine with firmware
#
# Copyright (c) 2020 Red Hat, Inc.
@@ -8,9 +10,9 @@
# This work is licensed under the terms of the GNU GPL, version 2 or
# later. See the COPYING file in the top-level directory.
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import archive
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import archive_extract
class CanonA1100Machine(QemuSystemTest):
"""Boots the barebox firmware and checks that the console is operational"""
@@ -18,18 +20,19 @@ class CanonA1100Machine(QemuSystemTest):
timeout = 90
def test_arm_canona1100(self):
- """
- :avocado: tags=arch:arm
- :avocado: tags=machine:canon-a1100
- :avocado: tags=device:pflash_cfi02
- """
+ self.set_machine('canon-a1100')
+
tar_url = ('https://qemu-advcal.gitlab.io'
'/qac-best-of-multiarch/download/day18.tar.xz')
tar_hash = '068b5fc4242b29381acee94713509f8a876e9db6'
file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
- archive.extract(file_path, self.workdir)
+ archive_extract(file_path, dest_dir=self.workdir,
+ member="day18/barebox.canon-a1100.bin")
self.vm.set_console()
self.vm.add_args('-bios',
self.workdir + '/day18/barebox.canon-a1100.bin')
self.vm.launch()
wait_for_console_pattern(self, 'running /env/bin/init')
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
diff --git a/tests/avocado/ppc_bamboo.py b/tests/functional/test_ppc_bamboo.py
old mode 100644
new mode 100755
similarity index 75%
rename from tests/avocado/ppc_bamboo.py
rename to tests/functional/test_ppc_bamboo.py
index a81be3d608..652b31e95a
--- a/tests/avocado/ppc_bamboo.py
+++ b/tests/functional/test_ppc_bamboo.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# Test that Linux kernel boots on the ppc bamboo board and check the console
#
# Copyright (c) 2021 Red Hat
@@ -5,30 +7,24 @@
# This work is licensed under the terms of the GNU GPL, version 2 or
# later. See the COPYING file in the top-level directory.
-from avocado.utils import archive
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import wait_for_console_pattern
-from avocado_qemu import exec_command_and_wait_for_pattern
+from qemu_test.utils import archive_extract
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from qemu_test import exec_command_and_wait_for_pattern
class BambooMachine(QemuSystemTest):
timeout = 90
def test_ppc_bamboo(self):
- """
- :avocado: tags=arch:ppc
- :avocado: tags=machine:bamboo
- :avocado: tags=cpu:440epb
- :avocado: tags=device:rtl8139
- :avocado: tags=accel:tcg
- """
+ self.set_machine('bamboo')
self.require_accelerator("tcg")
self.require_netdev('user')
tar_url = ('http://landley.net/aboriginal/downloads/binaries/'
'system-image-powerpc-440fp.tar.gz')
tar_hash = '53e5f16414b195b82d2c70272f81c2eedb39bad9'
file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
- archive.extract(file_path, self.workdir)
+ archive_extract(file_path, self.workdir)
self.vm.set_console()
self.vm.add_args('-kernel', self.workdir +
'/system-image-powerpc-440fp/linux',
@@ -40,3 +36,6 @@ def test_ppc_bamboo(self):
exec_command_and_wait_for_pattern(self, 'ping 10.0.2.2',
'10.0.2.2 is alive!')
exec_command_and_wait_for_pattern(self, 'halt', 'System Halted')
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
diff --git a/tests/avocado/machine_sparc64_sun4u.py b/tests/functional/test_sparc64_sun4u.py
old mode 100644
new mode 100755
similarity index 60%
rename from tests/avocado/machine_sparc64_sun4u.py
rename to tests/functional/test_sparc64_sun4u.py
index d333c0ae91..cd89d4a1d6
--- a/tests/avocado/machine_sparc64_sun4u.py
+++ b/tests/functional/test_sparc64_sun4u.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# Functional test that boots a Linux kernel and checks the console
#
# Copyright (c) 2020 Red Hat, Inc.
@@ -10,27 +12,28 @@
import os
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import archive
-from boot_linux_console import LinuxKernelTest
+from qemu_test import QemuSystemTest
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import archive_extract
-class Sun4uMachine(LinuxKernelTest):
+class Sun4uMachine(QemuSystemTest):
"""Boots the Linux kernel and checks that the console is operational"""
timeout = 90
def test_sparc64_sun4u(self):
- """
- :avocado: tags=arch:sparc64
- :avocado: tags=machine:sun4u
- """
+ self.set_machine('sun4u')
tar_url = ('https://qemu-advcal.gitlab.io'
'/qac-best-of-multiarch/download/day23.tar.xz')
tar_hash = '142db83cd974ffadc4f75c8a5cad5bcc5722c240'
file_path = self.fetch_asset(tar_url, asset_hash=tar_hash)
- archive.extract(file_path, self.workdir)
+ kernel_name = 'day23/vmlinux'
+ archive_extract(file_path, self.workdir, kernel_name)
self.vm.set_console()
- self.vm.add_args('-kernel', self.workdir + '/day23/vmlinux',
- '-append', self.KERNEL_COMMON_COMMAND_LINE)
+ self.vm.add_args('-kernel', os.path.join(self.workdir, kernel_name),
+ '-append', 'printk.time=0')
self.vm.launch()
wait_for_console_pattern(self, 'Starting logging: OK')
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
--
2.45.2
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 09/11] tests/functional: Set up logging
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
` (7 preceding siblings ...)
2024-07-16 11:26 ` [PATCH 08/11] tests/functional: Convert some avocado tests that needed avocado.utils.archive Thomas Huth
@ 2024-07-16 11:26 ` Thomas Huth
2024-07-16 18:22 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 10/11] tests/functional: Convert the s390x avocado tests into standalone tests Thomas Huth
` (4 subsequent siblings)
13 siblings, 1 reply; 36+ messages in thread
From: Thomas Huth @ 2024-07-16 11:26 UTC (permalink / raw)
To: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé
Cc: Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
Create log files for each test separately, one file that contains
the basic logging and one that contains the console output.
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
tests/functional/qemu_test/__init__.py | 24 +++++++++++++++++++++++-
1 file changed, 23 insertions(+), 1 deletion(-)
diff --git a/tests/functional/qemu_test/__init__.py b/tests/functional/qemu_test/__init__.py
index 77c3b14d34..cc49fd4c94 100644
--- a/tests/functional/qemu_test/__init__.py
+++ b/tests/functional/qemu_test/__init__.py
@@ -208,7 +208,7 @@ class QemuBaseTest(unittest.TestCase):
BUILD_DIR = _build_dir()
workdir = None
- log = logging.getLogger('qemu-test')
+ log = None
def setUp(self, bin_prefix):
self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
@@ -218,6 +218,18 @@ def setUp(self, bin_prefix):
if not os.path.exists(self.workdir):
os.makedirs(self.workdir)
+ self.log = logging.getLogger('qemu-test')
+ self.log.setLevel(logging.DEBUG)
+ self._log_fh = logging.FileHandler(os.path.join(self.workdir,
+ 'base.log'), mode='w')
+ self._log_fh.setLevel(logging.DEBUG)
+ fileFormatter = logging.Formatter('%(asctime)s - %(levelname)s: %(message)s')
+ self._log_fh.setFormatter(fileFormatter)
+ self.log.addHandler(self._log_fh)
+
+ def tearDown(self):
+ self.log.removeHandler(self._log_fh)
+
def check_hash(self, file_name, expected_hash):
if not expected_hash:
return True
@@ -269,6 +281,15 @@ def setUp(self):
super().setUp('qemu-system-')
+ console_log = logging.getLogger('console')
+ console_log.setLevel(logging.DEBUG)
+ self._console_log_fh = logging.FileHandler(os.path.join(self.workdir,
+ 'console.log'), mode='w')
+ self._console_log_fh.setLevel(logging.DEBUG)
+ fileFormatter = logging.Formatter('%(asctime)s: %(message)s')
+ self._console_log_fh.setFormatter(fileFormatter)
+ console_log.addHandler(self._console_log_fh)
+
def set_machine(self, machinename):
# TODO: We should use QMP to get the list of available machines
if not self._machinehelp:
@@ -359,4 +380,5 @@ def set_vm_arg(self, arg, value):
def tearDown(self):
for vm in self._vms.values():
vm.shutdown()
+ logging.getLogger('console').removeHandler(self._console_log_fh)
super().tearDown()
--
2.45.2
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 10/11] tests/functional: Convert the s390x avocado tests into standalone tests
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
` (8 preceding siblings ...)
2024-07-16 11:26 ` [PATCH 09/11] tests/functional: Set up logging Thomas Huth
@ 2024-07-16 11:26 ` Thomas Huth
2024-07-16 18:24 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 11/11] gitlab-ci: Add "check-functional" to the build tests Thomas Huth
` (3 subsequent siblings)
13 siblings, 1 reply; 36+ messages in thread
From: Thomas Huth @ 2024-07-16 11:26 UTC (permalink / raw)
To: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé
Cc: Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
These tests use archive.lzma_uncompress() from the Avocado utils,
so provide a small helper function for this, based on the
standard lzma module from Python instead.
And while we're at it, replace the MD5 hashes in the topology test
with proper SHA256 hashes, since MD5 should not be used anymore
nowadays.
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
MAINTAINERS | 4 +-
tests/functional/meson.build | 6 ++
tests/functional/qemu_test/utils.py | 7 ++
.../test_s390x_ccw_virtio.py} | 32 ++++-----
.../test_s390x_topology.py} | 70 +++++++------------
5 files changed, 52 insertions(+), 67 deletions(-)
rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (95%)
mode change 100644 => 100755
rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (90%)
mode change 100644 => 100755
diff --git a/MAINTAINERS b/MAINTAINERS
index 025227954c..cbefb6fb81 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1739,7 +1739,7 @@ S: Supported
F: hw/s390x/
F: include/hw/s390x/
F: configs/devices/s390x-softmmu/default.mak
-F: tests/avocado/machine_s390_ccw_virtio.py
+F: tests/functional/test_s390x_ccw_virtio.py
T: git https://github.com/borntraeger/qemu.git s390-next
L: qemu-s390x@nongnu.org
@@ -1802,7 +1802,7 @@ F: hw/s390x/cpu-topology.c
F: target/s390x/kvm/stsi-topology.c
F: docs/devel/s390-cpu-topology.rst
F: docs/system/s390x/cpu-topology.rst
-F: tests/avocado/s390_topology.py
+F: tests/functional/test_s390x_topology.py
X86 Machines
------------
diff --git a/tests/functional/meson.build b/tests/functional/meson.build
index c8fc9f6c07..f6de9af8a2 100644
--- a/tests/functional/meson.build
+++ b/tests/functional/meson.build
@@ -13,6 +13,7 @@ endif
test_timeouts = {
'netdev_ethtool' : 180,
'ppc_74xx' : 90,
+ 's390x_ccw_virtio' : 180,
}
tests_generic = [
@@ -47,6 +48,11 @@ tests_ppc_thorough = [
'ppc_bamboo',
]
+tests_s390x_thorough = [
+ 's390x_ccw_virtio',
+ 's390x_topology',
+]
+
tests_sparc64_thorough = [
'sparc64_sun4u',
]
diff --git a/tests/functional/qemu_test/utils.py b/tests/functional/qemu_test/utils.py
index 4eb5e5d5e5..8115d9d1da 100644
--- a/tests/functional/qemu_test/utils.py
+++ b/tests/functional/qemu_test/utils.py
@@ -8,6 +8,8 @@
# This work is licensed under the terms of the GNU GPL, version 2 or
# later. See the COPYING file in the top-level directory.
+import lzma
+import shutil
import tarfile
def archive_extract(archive, dest_dir, member=None):
@@ -19,3 +21,8 @@ def archive_extract(archive, dest_dir, member=None):
tf.extract(member=member, path=dest_dir)
else:
tf.extractall(path=dest_dir)
+
+def lzma_uncompress(xz_path, output_path):
+ with lzma.open(xz_path, 'rb') as lzma_in:
+ with open(output_path, 'wb') as raw_out:
+ shutil.copyfileobj(lzma_in, raw_out)
diff --git a/tests/avocado/machine_s390_ccw_virtio.py b/tests/functional/test_s390x_ccw_virtio.py
old mode 100644
new mode 100755
similarity index 95%
rename from tests/avocado/machine_s390_ccw_virtio.py
rename to tests/functional/test_s390x_ccw_virtio.py
index 26e938c9e9..b2cbaacd21
--- a/tests/avocado/machine_s390_ccw_virtio.py
+++ b/tests/functional/test_s390x_ccw_virtio.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# Functional test that boots an s390x Linux guest with ccw and PCI devices
# attached and checks whether the devices are recognized by Linux
#
@@ -12,11 +14,10 @@
import os
import tempfile
-from avocado import skipUnless
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import exec_command_and_wait_for_pattern
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import archive
+from qemu_test import QemuSystemTest
+from qemu_test import exec_command_and_wait_for_pattern
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import lzma_uncompress
class S390CCWVirtioMachine(QemuSystemTest):
KERNEL_COMMON_COMMAND_LINE = 'printk.time=0 '
@@ -41,11 +42,7 @@ def clear_guest_dmesg(self):
self.dmesg_clear_count += 1
def test_s390x_devices(self):
-
- """
- :avocado: tags=arch:s390x
- :avocado: tags=machine:s390-ccw-virtio
- """
+ self.set_machine('s390-ccw-virtio')
kernel_url = ('https://snapshot.debian.org/archive/debian/'
'20201126T092837Z/dists/buster/main/installer-s390x/'
@@ -160,15 +157,7 @@ def test_s390x_devices(self):
def test_s390x_fedora(self):
-
- """
- :avocado: tags=arch:s390x
- :avocado: tags=machine:s390-ccw-virtio
- :avocado: tags=device:virtio-gpu
- :avocado: tags=device:virtio-crypto
- :avocado: tags=device:virtio-net
- :avocado: tags=flaky
- """
+ self.set_machine('s390-ccw-virtio')
kernel_url = ('https://archives.fedoraproject.org/pub/archive'
'/fedora-secondary/releases/31/Server/s390x/os'
@@ -182,7 +171,7 @@ def test_s390x_fedora(self):
initrd_hash = '3de45d411df5624b8d8ef21cd0b44419ab59b12f'
initrd_path_xz = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
initrd_path = os.path.join(self.workdir, 'initrd-raw.img')
- archive.lzma_uncompress(initrd_path_xz, initrd_path)
+ lzma_uncompress(initrd_path_xz, initrd_path)
self.vm.set_console()
kernel_command_line = (self.KERNEL_COMMON_COMMAND_LINE + ' audit=0 '
@@ -275,3 +264,6 @@ def test_s390x_fedora(self):
exec_command_and_wait_for_pattern(self,
'while ! (dmesg -c | grep Start.virtcrypto_remove) ; do'
' sleep 1 ; done', 'Start virtcrypto_remove.')
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
diff --git a/tests/avocado/s390_topology.py b/tests/functional/test_s390x_topology.py
old mode 100644
new mode 100755
similarity index 90%
rename from tests/avocado/s390_topology.py
rename to tests/functional/test_s390x_topology.py
index 9154ac8776..39c5ce0b3d
--- a/tests/avocado/s390_topology.py
+++ b/tests/functional/test_s390x_topology.py
@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+#
# Functional test that boots a Linux kernel and checks the console
#
# Copyright IBM Corp. 2023
@@ -9,16 +11,13 @@
# later. See the COPYING file in the top-level directory.
import os
-import shutil
import time
-from avocado_qemu import QemuSystemTest
-from avocado_qemu import exec_command
-from avocado_qemu import exec_command_and_wait_for_pattern
-from avocado_qemu import interrupt_interactive_console_until_pattern
-from avocado_qemu import wait_for_console_pattern
-from avocado.utils import process
-from avocado.utils import archive
+from qemu_test import QemuSystemTest
+from qemu_test import exec_command
+from qemu_test import exec_command_and_wait_for_pattern
+from qemu_test import wait_for_console_pattern
+from qemu_test.utils import lzma_uncompress
class S390CPUTopology(QemuSystemTest):
@@ -81,18 +80,16 @@ def kernel_init(self):
kernel_url = ('https://archives.fedoraproject.org/pub/archive'
'/fedora-secondary/releases/35/Server/s390x/os'
'/images/kernel.img')
- kernel_hash = '0d1aaaf303f07cf0160c8c48e56fe638'
- kernel_path = self.fetch_asset(kernel_url, algorithm='md5',
- asset_hash=kernel_hash)
+ kernel_hash = '1f2dddfd11bb1393dd2eb2e784036fbf6fc11057a6d7d27f9eb12d3edc67ef73'
+ kernel_path = self.fetch_asset(kernel_url, asset_hash=kernel_hash)
initrd_url = ('https://archives.fedoraproject.org/pub/archive'
'/fedora-secondary/releases/35/Server/s390x/os'
'/images/initrd.img')
- initrd_hash = 'a122057d95725ac030e2ec51df46e172'
- initrd_path_xz = self.fetch_asset(initrd_url, algorithm='md5',
- asset_hash=initrd_hash)
+ initrd_hash = '1100145fbca00240c8c372ae4b89b48c99844bc189b3dfbc3f481dc60055ca46'
+ initrd_path_xz = self.fetch_asset(initrd_url, asset_hash=initrd_hash)
initrd_path = os.path.join(self.workdir, 'initrd-raw.img')
- archive.lzma_uncompress(initrd_path_xz, initrd_path)
+ lzma_uncompress(initrd_path_xz, initrd_path)
self.vm.set_console()
kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE
@@ -115,10 +112,8 @@ def system_init(self):
def test_single(self):
"""
This test checks the simplest topology with a single CPU.
-
- :avocado: tags=arch:s390x
- :avocado: tags=machine:s390-ccw-virtio
"""
+ self.set_machine('s390-ccw-virtio')
self.kernel_init()
self.vm.launch()
self.wait_until_booted()
@@ -127,10 +122,8 @@ def test_single(self):
def test_default(self):
"""
This test checks the implicit topology.
-
- :avocado: tags=arch:s390x
- :avocado: tags=machine:s390-ccw-virtio
"""
+ self.set_machine('s390-ccw-virtio')
self.kernel_init()
self.vm.add_args('-smp',
'13,drawers=2,books=2,sockets=3,cores=2,maxcpus=24')
@@ -154,10 +147,8 @@ def test_move(self):
"""
This test checks the topology modification by moving a CPU
to another socket: CPU 0 is moved from socket 0 to socket 2.
-
- :avocado: tags=arch:s390x
- :avocado: tags=machine:s390-ccw-virtio
"""
+ self.set_machine('s390-ccw-virtio')
self.kernel_init()
self.vm.add_args('-smp',
'1,drawers=2,books=2,sockets=3,cores=2,maxcpus=24')
@@ -174,10 +165,8 @@ def test_dash_device(self):
"""
This test verifies that a CPU defined with the '-device'
command line option finds its right place inside the topology.
-
- :avocado: tags=arch:s390x
- :avocado: tags=machine:s390-ccw-virtio
"""
+ self.set_machine('s390-ccw-virtio')
self.kernel_init()
self.vm.add_args('-smp',
'1,drawers=2,books=2,sockets=3,cores=2,maxcpus=24')
@@ -221,10 +210,8 @@ def test_polarization(self):
"""
This test verifies that QEMU modifies the entitlement change after
several guest polarization change requests.
-
- :avocado: tags=arch:s390x
- :avocado: tags=machine:s390-ccw-virtio
"""
+ self.set_machine('s390-ccw-virtio')
self.kernel_init()
self.vm.launch()
self.wait_until_booted()
@@ -267,10 +254,8 @@ def test_entitlement(self):
"""
This test verifies that QEMU modifies the entitlement
after a guest request and that the guest sees the change.
-
- :avocado: tags=arch:s390x
- :avocado: tags=machine:s390-ccw-virtio
"""
+ self.set_machine('s390-ccw-virtio')
self.kernel_init()
self.vm.launch()
self.wait_until_booted()
@@ -313,10 +298,8 @@ def test_dedicated(self):
CPU is made dedicated.
QEMU retains the entitlement value when horizontal polarization is in effect.
For the guest, the field shows the effective value of the entitlement.
-
- :avocado: tags=arch:s390x
- :avocado: tags=machine:s390-ccw-virtio
"""
+ self.set_machine('s390-ccw-virtio')
self.kernel_init()
self.vm.launch()
self.wait_until_booted()
@@ -345,10 +328,8 @@ def test_socket_full(self):
This test verifies that QEMU does not accept to overload a socket.
The socket-id 0 on book-id 0 already contains CPUs 0 and 1 and can
not accept any new CPU while socket-id 0 on book-id 1 is free.
-
- :avocado: tags=arch:s390x
- :avocado: tags=machine:s390-ccw-virtio
"""
+ self.set_machine('s390-ccw-virtio')
self.kernel_init()
self.vm.add_args('-smp',
'3,drawers=2,books=2,sockets=3,cores=2,maxcpus=24')
@@ -369,10 +350,8 @@ def test_dedicated_error(self):
"""
This test verifies that QEMU refuses to lower the entitlement
of a dedicated CPU
-
- :avocado: tags=arch:s390x
- :avocado: tags=machine:s390-ccw-virtio
"""
+ self.set_machine('s390-ccw-virtio')
self.kernel_init()
self.vm.launch()
self.wait_until_booted()
@@ -417,10 +396,8 @@ def test_move_error(self):
"""
This test verifies that QEMU refuses to move a CPU to an
nonexistent location
-
- :avocado: tags=arch:s390x
- :avocado: tags=machine:s390-ccw-virtio
"""
+ self.set_machine('s390-ccw-virtio')
self.kernel_init()
self.vm.launch()
self.wait_until_booted()
@@ -437,3 +414,6 @@ def test_move_error(self):
self.assertEqual(res['error']['class'], 'GenericError')
self.check_topology(0, 0, 0, 0, 'medium', False)
+
+if __name__ == '__main__':
+ QemuSystemTest.main()
--
2.45.2
^ permalink raw reply related [flat|nested] 36+ messages in thread
* [PATCH 11/11] gitlab-ci: Add "check-functional" to the build tests
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
` (9 preceding siblings ...)
2024-07-16 11:26 ` [PATCH 10/11] tests/functional: Convert the s390x avocado tests into standalone tests Thomas Huth
@ 2024-07-16 11:26 ` Thomas Huth
2024-07-16 18:25 ` Daniel P. Berrangé
2024-07-16 16:51 ` [PATCH v1 00/11] Convert avocado tests to normal Python unittests Daniel P. Berrangé
` (2 subsequent siblings)
13 siblings, 1 reply; 36+ messages in thread
From: Thomas Huth @ 2024-07-16 11:26 UTC (permalink / raw)
To: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé
Cc: Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
Now that we converted many tests from the "check-avocado" test suite
to the "check-functional" test suite, we should make sure that these
also get tested in the CI.
Signed-off-by: Thomas Huth <thuth@redhat.com>
---
.gitlab-ci.d/buildtest-template.yml | 3 +-
.gitlab-ci.d/buildtest.yml | 60 ++++++++++++++---------------
2 files changed, 32 insertions(+), 31 deletions(-)
diff --git a/.gitlab-ci.d/buildtest-template.yml b/.gitlab-ci.d/buildtest-template.yml
index 8f7ebfaed8..54cae74a8e 100644
--- a/.gitlab-ci.d/buildtest-template.yml
+++ b/.gitlab-ci.d/buildtest-template.yml
@@ -71,7 +71,7 @@
reports:
junit: build/meson-logs/testlog.junit.xml
-.avocado_test_job_template:
+.functional_test_job_template:
extends: .common_test_job_template
cache:
key: "${CI_JOB_NAME}-cache"
@@ -98,6 +98,7 @@
du -chs ${CI_PROJECT_DIR}/avocado-cache ;
fi
- export AVOCADO_ALLOW_UNTRUSTED_CODE=1
+ - export QEMU_TEST_ALLOW_UNTRUSTED_CODE=1
after_script:
- cd build
- du -chs ${CI_PROJECT_DIR}/avocado-cache
diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index e3a0758bd9..8b1d61911c 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -22,14 +22,14 @@ check-system-alpine:
IMAGE: alpine
MAKE_CHECK_ARGS: check-unit check-qtest
-avocado-system-alpine:
- extends: .avocado_test_job_template
+functional-system-alpine:
+ extends: .functional_test_job_template
needs:
- job: build-system-alpine
artifacts: true
variables:
IMAGE: alpine
- MAKE_CHECK_ARGS: check-avocado
+ MAKE_CHECK_ARGS: check-avocado check-functional
AVOCADO_TAGS: arch:avr arch:loongarch64 arch:mips64 arch:mipsel
build-system-ubuntu:
@@ -53,14 +53,14 @@ check-system-ubuntu:
IMAGE: ubuntu2204
MAKE_CHECK_ARGS: check
-avocado-system-ubuntu:
- extends: .avocado_test_job_template
+functional-system-ubuntu:
+ extends: .functional_test_job_template
needs:
- job: build-system-ubuntu
artifacts: true
variables:
IMAGE: ubuntu2204
- MAKE_CHECK_ARGS: check-avocado
+ MAKE_CHECK_ARGS: check-avocado check-functional
AVOCADO_TAGS: arch:alpha arch:microblazeel arch:mips64el
build-system-debian:
@@ -85,14 +85,14 @@ check-system-debian:
IMAGE: debian
MAKE_CHECK_ARGS: check
-avocado-system-debian:
- extends: .avocado_test_job_template
+functional-system-debian:
+ extends: .functional_test_job_template
needs:
- job: build-system-debian
artifacts: true
variables:
IMAGE: debian
- MAKE_CHECK_ARGS: check-avocado
+ MAKE_CHECK_ARGS: check-avocado check-functional
AVOCADO_TAGS: arch:arm arch:i386 arch:riscv64 arch:sh4 arch:sparc arch:xtensa
crash-test-debian:
@@ -129,14 +129,14 @@ check-system-fedora:
IMAGE: fedora
MAKE_CHECK_ARGS: check
-avocado-system-fedora:
- extends: .avocado_test_job_template
+functional-system-fedora:
+ extends: .functional_test_job_template
needs:
- job: build-system-fedora
artifacts: true
variables:
IMAGE: fedora
- MAKE_CHECK_ARGS: check-avocado
+ MAKE_CHECK_ARGS: check-avocado check-functional
AVOCADO_TAGS: arch:microblaze arch:mips arch:xtensa arch:m68k
arch:riscv32 arch:ppc arch:sparc64
@@ -243,14 +243,14 @@ check-system-centos:
IMAGE: centos9
MAKE_CHECK_ARGS: check
-avocado-system-centos:
- extends: .avocado_test_job_template
+functional-system-centos:
+ extends: .functional_test_job_template
needs:
- job: build-system-centos
artifacts: true
variables:
IMAGE: centos9
- MAKE_CHECK_ARGS: check-avocado
+ MAKE_CHECK_ARGS: check-avocado check-functional
AVOCADO_TAGS: arch:ppc64 arch:or1k arch:s390x arch:x86_64 arch:rx
arch:sh4
@@ -274,14 +274,14 @@ check-system-opensuse:
IMAGE: opensuse-leap
MAKE_CHECK_ARGS: check
-avocado-system-opensuse:
- extends: .avocado_test_job_template
+functional-system-opensuse:
+ extends: .functional_test_job_template
needs:
- job: build-system-opensuse
artifacts: true
variables:
IMAGE: opensuse-leap
- MAKE_CHECK_ARGS: check-avocado
+ MAKE_CHECK_ARGS: check-avocado check-functional
AVOCADO_TAGS: arch:s390x arch:x86_64 arch:aarch64
#
@@ -302,15 +302,15 @@ build-system-flaky:
ppc64-softmmu rx-softmmu s390x-softmmu sh4-softmmu x86_64-softmmu
MAKE_CHECK_ARGS: check-build
-avocado-system-flaky:
- extends: .avocado_test_job_template
+functional-system-flaky:
+ extends: .functional_test_job_template
needs:
- job: build-system-flaky
artifacts: true
allow_failure: true
variables:
IMAGE: debian
- MAKE_CHECK_ARGS: check-avocado
+ MAKE_CHECK_ARGS: check-avocado check-functional
QEMU_JOB_OPTIONAL: 1
QEMU_TEST_FLAKY_TESTS: 1
AVOCADO_TAGS: flaky
@@ -484,14 +484,14 @@ check-cfi-aarch64:
IMAGE: fedora
MAKE_CHECK_ARGS: check
-avocado-cfi-aarch64:
- extends: .avocado_test_job_template
+functional-cfi-aarch64:
+ extends: .functional_test_job_template
needs:
- job: build-cfi-aarch64
artifacts: true
variables:
IMAGE: fedora
- MAKE_CHECK_ARGS: check-avocado
+ MAKE_CHECK_ARGS: check-avocado check-functional
build-cfi-ppc64-s390x:
extends:
@@ -522,14 +522,14 @@ check-cfi-ppc64-s390x:
IMAGE: fedora
MAKE_CHECK_ARGS: check
-avocado-cfi-ppc64-s390x:
- extends: .avocado_test_job_template
+functional-cfi-ppc64-s390x:
+ extends: .functional_test_job_template
needs:
- job: build-cfi-ppc64-s390x
artifacts: true
variables:
IMAGE: fedora
- MAKE_CHECK_ARGS: check-avocado
+ MAKE_CHECK_ARGS: check-avocado check-functional
build-cfi-x86_64:
extends:
@@ -556,14 +556,14 @@ check-cfi-x86_64:
IMAGE: fedora
MAKE_CHECK_ARGS: check
-avocado-cfi-x86_64:
- extends: .avocado_test_job_template
+functional-cfi-x86_64:
+ extends: .functional_test_job_template
needs:
- job: build-cfi-x86_64
artifacts: true
variables:
IMAGE: fedora
- MAKE_CHECK_ARGS: check-avocado
+ MAKE_CHECK_ARGS: check-avocado check-functional
tsan-build:
extends: .native_build_job_template
--
2.45.2
^ permalink raw reply related [flat|nested] 36+ messages in thread
* Re: [PATCH 04/11] tests/functional: Add python-based tests to the meson build system
2024-07-16 11:26 ` [PATCH 04/11] tests/functional: Add python-based tests to the meson build system Thomas Huth
@ 2024-07-16 15:15 ` Fabiano Rosas
2024-07-17 9:20 ` Thomas Huth
2024-07-16 16:55 ` Daniel P. Berrangé
1 sibling, 1 reply; 36+ messages in thread
From: Fabiano Rosas @ 2024-07-16 15:15 UTC (permalink / raw)
To: Thomas Huth, Alex Bennée, qemu-devel,
Philippe Mathieu-Daudé
Cc: Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater
Thomas Huth <thuth@redhat.com> writes:
> Integrate the new python-based test framework with the meson build
> system. Since these tests now require the pycotap module, make
> sure that it gets installed in the venv.
>
> The changes to the meson.build files are partly based on an earlier
> patch by Ani Sinha (but heavily modified by Thomas Huth e.g. to use
> pycotap for running the tests instead).
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
Just one touch-up below.
Reviewed-by: Fabiano Rosas <farosas@suse.de>
> ---
> pythondeps.toml | 3 +-
> tests/Makefile.include | 18 ++++++++-
> tests/functional/meson.build | 75 ++++++++++++++++++++++++++++++++++++
> tests/meson.build | 1 +
> 4 files changed, 95 insertions(+), 2 deletions(-)
> create mode 100644 tests/functional/meson.build
>
> diff --git a/pythondeps.toml b/pythondeps.toml
> index f6e590fdd8..c018b4d74a 100644
> --- a/pythondeps.toml
> +++ b/pythondeps.toml
> @@ -26,9 +26,10 @@ meson = { accepted = ">=1.1.0", installed = "1.2.3", canary = "meson" }
> sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
> sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
>
> -[avocado]
> +[tests]
> # Note that qemu.git/python/ is always implicitly installed.
> # Prefer an LTS version when updating the accepted versions of
> # avocado-framework, for example right now the limit is 92.x.
> avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
> pycdlib = { accepted = ">=1.11.0" }
> +pycotap = { accepted = ">=1.1.0" }
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index d39d5dd6a4..2bdf607977 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -9,6 +9,8 @@ check-help:
> @echo "Individual test suites:"
> @echo " $(MAKE) check-qtest-TARGET Run qtest tests for given target"
> @echo " $(MAKE) check-qtest Run qtest tests"
> + @echo " $(MAKE) check-functional Run python-based functional tests"
> + @echo " $(MAKE) check-functional-TARG Run functional tests for
> a given target"
> @echo " $(MAKE) check-unit Run qobject tests"
> @echo " $(MAKE) check-qapi-schema Run QAPI schema tests"
> @echo " $(MAKE) check-block Run block tests"
> @@ -111,7 +113,7 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
>
> $(TESTS_VENV_TOKEN): $(SRC_PATH)/pythondeps.toml
> $(call quiet-venv-pip,install -e "$(SRC_PATH)/python/")
> - $(MKVENV_ENSUREGROUP) $< avocado
> + $(MKVENV_ENSUREGROUP) $< tests
> $(call quiet-command, touch $@)
>
> $(TESTS_RESULTS_DIR):
> @@ -152,6 +154,20 @@ check-acceptance-deprecated-warning:
>
> check-acceptance: check-acceptance-deprecated-warning | check-avocado
>
> +# Make sure that pycotap is installed before running any functional tests:
> +ifneq ($(filter check-func%,$(MAKECMDGOALS))$(filter check,$(MAKECMDGOALS)),)
> +do-meson-check: check-venv
> +endif
> +
> +FUNCTIONAL_TARGETS=$(patsubst %-softmmu,check-functional-%, $(filter %-softmmu,$(TARGETS)))
> +.PHONY: $(FUNCTIONAL_TARGETS)
> +$(FUNCTIONAL_TARGETS):
> + @make SPEED=thorough $(subst -functional,-func,$@)
> +
> +.PHONY: check-functional
> +check-functional:
> + @make SPEED=thorough check-func check-func-quick
I think these^ two should use $(MAKE) instead:
make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent
make rule.
> +
> # Consolidated targets
>
> .PHONY: check check-clean get-vm-images
> diff --git a/tests/functional/meson.build b/tests/functional/meson.build
> new file mode 100644
> index 0000000000..11352b5bb5
> --- /dev/null
> +++ b/tests/functional/meson.build
> @@ -0,0 +1,75 @@
> +# QEMU functional tests:
> +# Tests that are put in the 'quick' category are run by default during
> +# 'make check'. Everything that should not be run during 'make check'
> +# (e.g. tests that fetch assets from the internet) should be put into
> +# the 'thorough' category instead.
> +
> +# Most tests run too slow with TCI enabled, so skip the functional tests there
> +if get_option('tcg_interpreter')
> + subdir_done()
> +endif
> +
> +# Timeouts for individual tests that can be slow e.g. with debugging enabled
> +test_timeouts = {
> + 'ppc_74xx' : 90,
> +}
> +
> +tests_generic = [
> + 'empty_cpu_model',
> + 'info_usernet',
> + 'version',
> +]
> +
> +tests_ppc_quick = [
> + 'ppc_74xx',
> +]
> +
> +tests_x86_64_quick = [
> + 'cpu_queries',
> + 'mem_addr_space',
> + 'pc_cpu_hotplug_props',
> + 'virtio_version',
> +]
> +
> +foreach speed : ['quick', 'thorough']
> + foreach dir : target_dirs
> + if not dir.endswith('-softmmu')
> + continue
> + endif
> +
> + target_base = dir.split('-')[0]
> + test_emulator = emulators['qemu-system-' + target_base]
> +
> + if speed == 'quick'
> + suites = ['func-quick', 'func-' + target_base]
> + target_tests = get_variable('tests_' + target_base + '_quick', []) + tests_generic
> + else
> + suites = ['func-' + speed, 'func-' + target_base + '-' + speed, speed]
> + target_tests = get_variable('tests_' + target_base + '_' + speed, [])
> + endif
> +
> + test_deps = roms
> + test_env = environment()
> + if have_tools
> + test_env.set('QEMU_TEST_QEMU_IMG', meson.global_build_root() / 'qemu-img')
> + test_deps += [qemu_img]
> + endif
> + test_env.set('QEMU_TEST_QEMU_BINARY',
> + meson.global_build_root() / 'qemu-system-' + target_base)
> + test_env.set('QEMU_BUILD_ROOT', meson.project_build_root())
> + test_env.set('PYTHONPATH', meson.project_source_root() / 'python:' +
> + meson.current_source_dir())
> +
> + foreach test : target_tests
> + test('func-@0@/@1@'.format(target_base, test),
> + python,
> + depends: [test_deps, test_emulator, emulator_modules],
> + env: test_env,
> + args: [meson.current_source_dir() / 'test_' + test + '.py'],
> + protocol: 'tap',
> + timeout: test_timeouts.get(test, 60),
> + priority: test_timeouts.get(test, 60),
> + suite: suites)
> + endforeach
2/19 qemu:func-thorough+func-s390x-thorough+thorough / func-s390x/s390x_ccw_virtio OK 48.82s 2 subtests passe
func, thorough, func, thorough, thorough, func
s390x, s390x, s390x
=)
I know, not much we can do...
> + endforeach
> +endforeach
> diff --git a/tests/meson.build b/tests/meson.build
> index acb6807094..3345ad2098 100644
> --- a/tests/meson.build
> +++ b/tests/meson.build
> @@ -85,3 +85,4 @@ subdir('unit')
> subdir('qapi-schema')
> subdir('qtest')
> subdir('migration')
> +subdir('functional')
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
` (10 preceding siblings ...)
2024-07-16 11:26 ` [PATCH 11/11] gitlab-ci: Add "check-functional" to the build tests Thomas Huth
@ 2024-07-16 16:51 ` Daniel P. Berrangé
2024-07-16 17:03 ` Thomas Huth
2024-07-16 17:57 ` Paolo Bonzini
2024-07-25 14:21 ` Cleber Rosa
13 siblings, 1 reply; 36+ messages in thread
From: Daniel P. Berrangé @ 2024-07-16 16:51 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
> The Avocado v88 that we use in QEMU is already on a life support
> system: It is not supported by upstream anymore, and with the latest
> versions of Python, it won't work anymore since it depends on the
> "imp" module that has been removed in Python 3.12.
>
> There have been several attempts to update the test suite in QEMU
> to a newer version of Avocado, but so far no attempt has successfully
> been merged yet.
>
> Additionally, the whole "make check" test suite in QEMU is using the
> meson test runner nowadays, so running the python-based tests via the
> Avocodo test runner looks and feels quite like an oddball, requiring
> the users to deal with the knowledge of multiple test runners in
> parallel (e.g. the timeout settings work completely differently).
>
> So instead of trying to update the python-based test suite in QEMU
> to a newer version of Avocado, we should maybe try to better integrate
> it with the meson test runner instead. Indeed most tests work quite
> nicely without the Avocado framework already, as you can see with
> this patch series - it does not convert all tests, just a subset so
> far, but this already proves that many tests only need small modifi-
> cations to work without Avocado.
>
> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> classes (e.g. based on cloud-init images or using SSH) really depend
> on the Avocado framework, so we'd need a solution for those if we
> want to continue using them. One solution might be to simply use the
> required functions from avocado.utils for these tests, and still run
> them via the meson test runner instead, but that needs some further
> investigation that will be done later.
>
>
> Now if you want to try out these patches: Apply the patches, then
> recompile and then run:
>
> make check-functional
>
> You can also run single targets e.g. with:
>
> make check-functional-ppc
>
> You can also run the tests without any test runner now by
> setting the PYTHONPATH environment variable to the "python" folder
> of your source tree, and by specifying the build directory via
> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>
> export PYTHONPATH=$HOME/qemu/python
> export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
> export PYTHONPATH=$HOME/qemu/build
> ~/qemu/tests/functional/test_virtio_version.py
For the whole series as is
Tested-by: Daniel P. Berrangé <berrange@redhat.com>
as it does what you claim it does here when I tried it.
> The logs of the tests can be found in the build directory under
> tests/functional/<testname> - console log and general logs will
> be put in separate files there.
As an example, one dir name appears to be:
__main__.MemAddrCheck.test_phybits_ok_pentium_pae
I'd rather prefer it if the dir name matched the test script
file name - in this case test_mem_addr_space.py, as I don't
want to have to lookup which class names were defined inside
each test script. We could drop the "test_" prefix from the
method name too
IOW, could we make this dir name be:
test_mem_addr_space.phybits_ok_pentium_pae
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH 04/11] tests/functional: Add python-based tests to the meson build system
2024-07-16 11:26 ` [PATCH 04/11] tests/functional: Add python-based tests to the meson build system Thomas Huth
2024-07-16 15:15 ` Fabiano Rosas
@ 2024-07-16 16:55 ` Daniel P. Berrangé
1 sibling, 0 replies; 36+ messages in thread
From: Daniel P. Berrangé @ 2024-07-16 16:55 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On Tue, Jul 16, 2024 at 01:26:07PM +0200, Thomas Huth wrote:
> Integrate the new python-based test framework with the meson build
> system. Since these tests now require the pycotap module, make
> sure that it gets installed in the venv.
>
> The changes to the meson.build files are partly based on an earlier
> patch by Ani Sinha (but heavily modified by Thomas Huth e.g. to use
> pycotap for running the tests instead).
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> pythondeps.toml | 3 +-
> tests/Makefile.include | 18 ++++++++-
> tests/functional/meson.build | 75 ++++++++++++++++++++++++++++++++++++
> tests/meson.build | 1 +
> 4 files changed, 95 insertions(+), 2 deletions(-)
> create mode 100644 tests/functional/meson.build
Strictly speaking this patch probably ought to be #2, otherwise we
have a bisection window where we've converted some tests but not
run them.
>
> diff --git a/pythondeps.toml b/pythondeps.toml
> index f6e590fdd8..c018b4d74a 100644
> --- a/pythondeps.toml
> +++ b/pythondeps.toml
> @@ -26,9 +26,10 @@ meson = { accepted = ">=1.1.0", installed = "1.2.3", canary = "meson" }
> sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
> sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
>
> -[avocado]
> +[tests]
> # Note that qemu.git/python/ is always implicitly installed.
> # Prefer an LTS version when updating the accepted versions of
> # avocado-framework, for example right now the limit is 92.x.
> avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
> pycdlib = { accepted = ">=1.11.0" }
> +pycotap = { accepted = ">=1.1.0" }
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index d39d5dd6a4..2bdf607977 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -9,6 +9,8 @@ check-help:
> @echo "Individual test suites:"
> @echo " $(MAKE) check-qtest-TARGET Run qtest tests for given target"
> @echo " $(MAKE) check-qtest Run qtest tests"
> + @echo " $(MAKE) check-functional Run python-based functional tests"
> + @echo " $(MAKE) check-functional-TARG Run functional tests for a given target"
We could increase whitespace by 2 to fit TARGET, or shorten all
cases to TGT ?
> @echo " $(MAKE) check-unit Run qobject tests"
> @echo " $(MAKE) check-qapi-schema Run QAPI schema tests"
> @echo " $(MAKE) check-block Run block tests"
> @@ -111,7 +113,7 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
>
The above is a minor non-functional point though so
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests
2024-07-16 16:51 ` [PATCH v1 00/11] Convert avocado tests to normal Python unittests Daniel P. Berrangé
@ 2024-07-16 17:03 ` Thomas Huth
2024-07-17 8:04 ` Thomas Huth
0 siblings, 1 reply; 36+ messages in thread
From: Thomas Huth @ 2024-07-16 17:03 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On 16/07/2024 18.51, Daniel P. Berrangé wrote:
> On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
...
>> So instead of trying to update the python-based test suite in QEMU
>> to a newer version of Avocado, we should maybe try to better integrate
>> it with the meson test runner instead. Indeed most tests work quite
>> nicely without the Avocado framework already, as you can see with
>> this patch series - it does not convert all tests, just a subset so
>> far, but this already proves that many tests only need small modifi-
>> cations to work without Avocado.
...
>> Now if you want to try out these patches: Apply the patches, then
>> recompile and then run:
>>
>> make check-functional
>>
>> You can also run single targets e.g. with:
>>
>> make check-functional-ppc
>>
>> You can also run the tests without any test runner now by
>> setting the PYTHONPATH environment variable to the "python" folder
>> of your source tree, and by specifying the build directory via
>> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
>> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>>
>> export PYTHONPATH=$HOME/qemu/python
>> export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
>> export PYTHONPATH=$HOME/qemu/build
>> ~/qemu/tests/functional/test_virtio_version.py
>
> For the whole series as is
>
> Tested-by: Daniel P. Berrangé <berrange@redhat.com>
>
> as it does what you claim it does here when I tried it.
Thanks!
>> The logs of the tests can be found in the build directory under
>> tests/functional/<testname> - console log and general logs will
>> be put in separate files there.
>
> As an example, one dir name appears to be:
>
> __main__.MemAddrCheck.test_phybits_ok_pentium_pae
>
> I'd rather prefer it if the dir name matched the test script
> file name - in this case test_mem_addr_space.py, as I don't
> want to have to lookup which class names were defined inside
> each test script. We could drop the "test_" prefix from the
> method name too
>
> IOW, could we make this dir name be:
>
> test_mem_addr_space.phybits_ok_pentium_pae
I can try to change that, indeed ... but the boilerplate code will increase
a little bit, I guess, since I cannot simply rely on the unittest.id()
function in that case anymore...
Thomas
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
` (11 preceding siblings ...)
2024-07-16 16:51 ` [PATCH v1 00/11] Convert avocado tests to normal Python unittests Daniel P. Berrangé
@ 2024-07-16 17:57 ` Paolo Bonzini
2024-07-24 11:43 ` Thomas Huth
2024-07-25 14:21 ` Cleber Rosa
13 siblings, 1 reply; 36+ messages in thread
From: Paolo Bonzini @ 2024-07-16 17:57 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Daniel P . Berrange, John Snow,
Cédric Le Goater, Fabiano Rosas
[-- Attachment #1: Type: text/plain, Size: 8684 bytes --]
Il mar 16 lug 2024, 13:26 Thomas Huth <thuth@redhat.com> ha scritto:
> The Avocado v88 that we use in QEMU is already on a life support
> system: It is not supported by upstream anymore, and with the latest
> versions of Python, it won't work anymore since it depends on the
> "imp" module that has been removed in Python 3.12.
>
> There have been several attempts to update the test suite in QEMU
> to a newer version of Avocado, but so far no attempt has successfully
> been merged yet.
>
I think we should take another look at that. Avocado 92 should work, though
I am not sure if it's also depending on "imp", and if I recall correctly
the problem with more recent version was more that it conflicted with
distros that didn't have it packaged. That should also not be a problem
anymore with the pythondeps.toml mechanism.
Additionally, the whole "make check" test suite in QEMU is using the
> meson test runner nowadays, so running the python-based tests via the
> Avocodo test runner looks and feels quite like an oddball, requiring
> the users to deal with the knowledge of multiple test runners in
> parallel (e.g. the timeout settings work completely differently).
>
This is true.
Paolo
> So instead of trying to update the python-based test suite in QEMU
> to a newer version of Avocado, we should maybe try to better integrate
> it with the meson test runner instead. Indeed most tests work quite
> nicely without the Avocado framework already, as you can see with
> this patch series - it does not convert all tests, just a subset so
> far, but this already proves that many tests only need small modifi-
> cations to work without Avocado.
>
> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> classes (e.g. based on cloud-init images or using SSH) really depend
> on the Avocado framework, so we'd need a solution for those if we
> want to continue using them. One solution might be to simply use the
> required functions from avocado.utils for these tests, and still run
> them via the meson test runner instead, but that needs some further
> investigation that will be done later.
>
>
> Now if you want to try out these patches: Apply the patches, then
> recompile and then run:
>
> make check-functional
>
> You can also run single targets e.g. with:
>
> make check-functional-ppc
>
> You can also run the tests without any test runner now by
> setting the PYTHONPATH environment variable to the "python" folder
> of your source tree, and by specifying the build directory via
> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>
> export PYTHONPATH=$HOME/qemu/python
> export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
> export PYTHONPATH=$HOME/qemu/build
> ~/qemu/tests/functional/test_virtio_version.py
>
> The logs of the tests can be found in the build directory under
> tests/functional/<testname> - console log and general logs will
> be put in separate files there.
>
> Still to be done: Update the documentation for this new test framework.
>
> RFC -> v1:
> - Now using pycotap for running the tests instead of "pytest"
> - Change the name from "tests/pytest" to "tests/functional" accordingly
> - Make it possible to run the tests directly
> - Use Python's urllib instead of wget for downloading
> - Lots of makefile / meson integration improvements
> - Converted more tests
> - Update MAINTAINERS file accordingly
> - Added a patch to run check-functional in the gitlab-CI
> - ... lots of other changes I forgot about ... in fact, I changed so
> many things that I also did not dare to pick up the Reviewed-bys
> from the RFC
>
> Thomas Huth (11):
> tests/functional: Add base classes for the upcoming pytest-based tests
> tests/functional: Convert simple avocado tests into standalone python
> tests
> tests/functional: Convert avocado tests that just need a small
> adjustment
> tests/functional: Add python-based tests to the meson build system
> tests/functional: Implement fetch_asset() method for downloading
> assets
> tests/functional: Convert some tests that download files via
> fetch_asset()
> tests/functional: Add a function for extracting files from an archive
> tests/functional: Convert some avocado tests that needed
> avocado.utils.archive
> tests/functional: Set up logging
> tests/functional: Convert the s390x avocado tests into standalone
> tests
> gitlab-ci: Add "check-functional" to the build tests
>
> MAINTAINERS | 22 +-
> .gitlab-ci.d/buildtest-template.yml | 3 +-
> .gitlab-ci.d/buildtest.yml | 60 +--
> pythondeps.toml | 3 +-
> tests/Makefile.include | 18 +-
> tests/functional/meson.build | 112 +++++
> tests/functional/qemu_test/__init__.py | 384 ++++++++++++++++++
> tests/functional/qemu_test/utils.py | 28 ++
> .../test_arm_canona1100.py} | 21 +-
> .../test_arm_n8x0.py} | 25 +-
> .../test_avr_mega2560.py} | 11 +-
> .../test_cpu_queries.py} | 7 +-
> .../test_empty_cpu_model.py} | 7 +-
> .../test_info_usernet.py} | 11 +-
> .../test_loongarch64_virt.py} | 16 +-
> .../test_mem_addr_space.py} | 53 +--
> .../test_mips64el_loongson3v.py} | 26 +-
> .../test_netdev_ethtool.py} | 32 +-
> .../test_pc_cpu_hotplug_props.py} | 11 +-
> .../ppc_405.py => functional/test_ppc_405.py} | 19 +-
> .../test_ppc_74xx.py} | 74 ++--
> .../test_ppc_bamboo.py} | 23 +-
> .../test_s390x_ccw_virtio.py} | 32 +-
> .../test_s390x_topology.py} | 70 ++--
> .../test_sparc64_sun4u.py} | 25 +-
> .../version.py => functional/test_version.py} | 13 +-
> .../test_virtio_version.py} | 8 +-
> tests/meson.build | 1 +
> 28 files changed, 795 insertions(+), 320 deletions(-)
> create mode 100644 tests/functional/meson.build
> create mode 100644 tests/functional/qemu_test/__init__.py
> create mode 100644 tests/functional/qemu_test/utils.py
> rename tests/{avocado/machine_arm_canona1100.py =>
> functional/test_arm_canona1100.py} (71%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py}
> (71%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py}
> (90%)
> mode change 100644 => 100755
> rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py}
> (89%)
> mode change 100644 => 100755
> rename tests/{avocado/empty_cpu_model.py =>
> functional/test_empty_cpu_model.py} (84%)
> mode change 100644 => 100755
> rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py}
> (87%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_loongarch.py =>
> functional/test_loongarch64_virt.py} (89%)
> mode change 100644 => 100755
> rename tests/{avocado/mem-addr-space-check.py =>
> functional/test_mem_addr_space.py} (92%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_mips_loongson3v.py =>
> functional/test_mips64el_loongson3v.py} (55%)
> mode change 100644 => 100755
> rename tests/{avocado/netdev-ethtool.py =>
> functional/test_netdev_ethtool.py} (81%)
> mode change 100644 => 100755
> rename tests/{avocado/pc_cpu_hotplug_props.py =>
> functional/test_pc_cpu_hotplug_props.py} (90%)
> mode change 100644 => 100755
> rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
> mode change 100644 => 100755
> rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
> mode change 100644 => 100755
> rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py}
> (75%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_s390_ccw_virtio.py =>
> functional/test_s390x_ccw_virtio.py} (95%)
> mode change 100644 => 100755
> rename tests/{avocado/s390_topology.py =>
> functional/test_s390x_topology.py} (90%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_sparc64_sun4u.py =>
> functional/test_sparc64_sun4u.py} (60%)
> mode change 100644 => 100755
> rename tests/{avocado/version.py => functional/test_version.py} (78%)
> mode change 100644 => 100755
> rename tests/{avocado/virtio_version.py =>
> functional/test_virtio_version.py} (98%)
> mode change 100644 => 100755
>
> --
> 2.45.2
>
>
[-- Attachment #2: Type: text/html, Size: 10644 bytes --]
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH 02/11] tests/functional: Convert simple avocado tests into standalone python tests
2024-07-16 11:26 ` [PATCH 02/11] tests/functional: Convert simple avocado tests into standalone python tests Thomas Huth
@ 2024-07-16 18:14 ` Daniel P. Berrangé
0 siblings, 0 replies; 36+ messages in thread
From: Daniel P. Berrangé @ 2024-07-16 18:14 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On Tue, Jul 16, 2024 at 01:26:05PM +0200, Thomas Huth wrote:
> These test are rather simple and don't need any modifications apart
> from adjusting the "from avocado_qemu" line. To ease debugging, make
> the files executable and add a shebang line and Python '__main__'
> handling, too, so that these tests can now be run by executing them
> directly.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> .../test_cpu_queries.py} | 7 ++-
> .../test_empty_cpu_model.py} | 7 ++-
> .../test_mem_addr_space.py} | 53 +++----------------
> .../test_pc_cpu_hotplug_props.py} | 11 ++--
> .../test_virtio_version.py} | 8 +--
> 5 files changed, 29 insertions(+), 57 deletions(-)
> rename tests/{avocado/cpu_queries.py => functional/test_cpu_queries.py} (89%)
> mode change 100644 => 100755
> rename tests/{avocado/empty_cpu_model.py => functional/test_empty_cpu_model.py} (84%)
> mode change 100644 => 100755
> rename tests/{avocado/mem-addr-space-check.py => functional/test_mem_addr_space.py} (92%)
> mode change 100644 => 100755
> rename tests/{avocado/pc_cpu_hotplug_props.py => functional/test_pc_cpu_hotplug_props.py} (90%)
> mode change 100644 => 100755
> rename tests/{avocado/virtio_version.py => functional/test_virtio_version.py} (98%)
> mode change 100644 => 100755
>
> diff --git a/tests/avocado/mem-addr-space-check.py b/tests/functional/test_mem_addr_space.py
> old mode 100644
> new mode 100755
> similarity index 92%
> rename from tests/avocado/mem-addr-space-check.py
> rename to tests/functional/test_mem_addr_space.py
> index 85541ea051..bb0cf062ca
> --- a/tests/avocado/mem-addr-space-check.py
> +++ b/tests/functional/test_mem_addr_space.py
> @@ -1,3 +1,5 @@
> +#!/usr/bin/env python3
> +#
> # Check for crash when using memory beyond the available guest processor
> # address space.
> #
> @@ -8,8 +10,7 @@
> #
> # SPDX-License-Identifier: GPL-2.0-or-later
>
> -from avocado_qemu import QemuSystemTest
> -import signal
Nit-pick - cleanup of an unrelated existing bug - 'signal' wasn't
used. Suggest doing it in a separate patch
Aside from that:
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH 03/11] tests/functional: Convert avocado tests that just need a small adjustment
2024-07-16 11:26 ` [PATCH 03/11] tests/functional: Convert avocado tests that just need a small adjustment Thomas Huth
@ 2024-07-16 18:15 ` Daniel P. Berrangé
0 siblings, 0 replies; 36+ messages in thread
From: Daniel P. Berrangé @ 2024-07-16 18:15 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On Tue, Jul 16, 2024 at 01:26:06PM +0200, Thomas Huth wrote:
> These simple tests can be converted to stand-alone tests quite easily,
> e.g. by just setting the machine to 'none' now manually or by adding
> "-cpu" command line parameters, since we don't support the corresponding
> avocado tags in the new python test framework.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> .../test_info_usernet.py} | 11 ++-
> .../test_ppc_74xx.py} | 74 ++++++++-----------
> .../version.py => functional/test_version.py} | 13 ++--
> 3 files changed, 47 insertions(+), 51 deletions(-)
> rename tests/{avocado/info_usernet.py => functional/test_info_usernet.py} (87%)
> mode change 100644 => 100755
> rename tests/{avocado/ppc_74xx.py => functional/test_ppc_74xx.py} (74%)
> mode change 100644 => 100755
> rename tests/{avocado/version.py => functional/test_version.py} (78%)
> mode change 100644 => 100755
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH 05/11] tests/functional: Implement fetch_asset() method for downloading assets
2024-07-16 11:26 ` [PATCH 05/11] tests/functional: Implement fetch_asset() method for downloading assets Thomas Huth
@ 2024-07-16 18:17 ` Daniel P. Berrangé
0 siblings, 0 replies; 36+ messages in thread
From: Daniel P. Berrangé @ 2024-07-16 18:17 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On Tue, Jul 16, 2024 at 01:26:08PM +0200, Thomas Huth wrote:
> In the new python test framework, we cannot use the fetch_asset()
> function from Avocado anymore, so we have to provide our own
> implementation now instead. Thus add such a function based on the
> urllib python module for this purpose.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> tests/functional/qemu_test/__init__.py | 36 ++++++++++++++++++++++++++
> 1 file changed, 36 insertions(+)
>
> diff --git a/tests/functional/qemu_test/__init__.py b/tests/functional/qemu_test/__init__.py
> index e73705d40a..77c3b14d34 100644
> --- a/tests/functional/qemu_test/__init__.py
> +++ b/tests/functional/qemu_test/__init__.py
> @@ -11,6 +11,8 @@
> # This work is licensed under the terms of the GNU GPL, version 2 or
> # later. See the COPYING file in the top-level directory.
>
> +import hashlib
> +import urllib.request
> import logging
> import os
> import pycotap
> @@ -23,6 +25,7 @@
> import unittest
>
> from pathlib import Path
> +from shutil import copyfileobj
> from qemu.machine import QEMUMachine
> from qemu.utils import (get_info_usernet_hostfwd_port, kvm_available,
> tcg_available)
> @@ -215,6 +218,39 @@ def setUp(self, bin_prefix):
> if not os.path.exists(self.workdir):
> os.makedirs(self.workdir)
>
> + def check_hash(self, file_name, expected_hash):
> + if not expected_hash:
> + return True
> + if len(expected_hash) == 40:
> + sum_prog = 'sha1sum'
> + elif len(expected_hash) == 64:
> + sum_prog = 'sha256sum'
> + elif len(expected_hash) == 128:
> + sum_prog = 'sha512sum'
> + else:
> + raise Exception("unknown hash type")
> + checksum = subprocess.check_output([sum_prog, file_name]).split()[0]
> + return expected_hash == checksum.decode("utf-8")
> +
> + def fetch_asset(self, url, asset_hash):
> + cache_dir = os.path.expanduser("~/.cache/qemu/download")
> + if not os.path.exists(cache_dir):
> + os.makedirs(cache_dir)
> + fname = os.path.join(cache_dir,
> + hashlib.sha256(url.encode("utf-8")).hexdigest())
> + if os.path.exists(fname) and self.check_hash(fname, asset_hash):
> + return fname
> + self.log.info("Downloading %s to %s...", url, fname)
> + dl_fname = fname + ".download"
> + with urllib.request.urlopen(url) as src:
> + with open(dl_fname, "wb+") as dst:
> + copyfileobj(src, dst)
For cleanliness we should probably wrap this and delete the file on
error, eg
try:
with open(dl_fname,....) as dst
....
except:
os.remove(dl_fname)
raise
> + if not self.check_hash(dl_fname, asset_hash):
> + os.remove(dl_fname)
> + raise Exception("Hash of " + url + " does not match")
> + os.rename(dl_fname, fname)
> + return fname
> +
> def main():
> tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
> test_output_log = pycotap.LogMode.LogToError)
> --
> 2.45.2
>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH 06/11] tests/functional: Convert some tests that download files via fetch_asset()
2024-07-16 11:26 ` [PATCH 06/11] tests/functional: Convert some tests that download files via fetch_asset() Thomas Huth
@ 2024-07-16 18:19 ` Daniel P. Berrangé
0 siblings, 0 replies; 36+ messages in thread
From: Daniel P. Berrangé @ 2024-07-16 18:19 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On Tue, Jul 16, 2024 at 01:26:09PM +0200, Thomas Huth wrote:
> Now that we've got a working fetch_asset() function, we can convert
> some Avocado tests that use this function for downloading their
> required files.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> MAINTAINERS | 12 +++----
> tests/functional/meson.build | 25 +++++++++++++++
> .../test_arm_n8x0.py} | 25 +++++++--------
> .../test_avr_mega2560.py} | 11 ++++---
> .../test_loongarch64_virt.py} | 16 ++++++----
> .../test_mips64el_loongson3v.py} | 26 +++++++--------
> .../test_netdev_ethtool.py} | 32 ++++++-------------
> .../ppc_405.py => functional/test_ppc_405.py} | 19 ++++++-----
> 8 files changed, 89 insertions(+), 77 deletions(-)
> rename tests/{avocado/machine_arm_n8x0.py => functional/test_arm_n8x0.py} (71%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_avr6.py => functional/test_avr_mega2560.py} (90%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_loongarch.py => functional/test_loongarch64_virt.py} (89%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_mips_loongson3v.py => functional/test_mips64el_loongson3v.py} (55%)
> mode change 100644 => 100755
> rename tests/{avocado/netdev-ethtool.py => functional/test_netdev_ethtool.py} (81%)
> mode change 100644 => 100755
> rename tests/{avocado/ppc_405.py => functional/test_ppc_405.py} (73%)
> mode change 100644 => 100755
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH 07/11] tests/functional: Add a function for extracting files from an archive
2024-07-16 11:26 ` [PATCH 07/11] tests/functional: Add a function for extracting files from an archive Thomas Huth
@ 2024-07-16 18:20 ` Daniel P. Berrangé
0 siblings, 0 replies; 36+ messages in thread
From: Daniel P. Berrangé @ 2024-07-16 18:20 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On Tue, Jul 16, 2024 at 01:26:10PM +0200, Thomas Huth wrote:
> Some Avocado-based tests use the "archive" module from avocado.utils
> to extract files from an archive. To be able to use these tests
> without Avocado, we have to provide our own function for extracting
> files. Fortunately, there is already the tarfile module that will
> provide us with this functionality, so let's just add a nice wrapper
> function around that.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> tests/functional/qemu_test/utils.py | 21 +++++++++++++++++++++
> 1 file changed, 21 insertions(+)
> create mode 100644 tests/functional/qemu_test/utils.py
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH 08/11] tests/functional: Convert some avocado tests that needed avocado.utils.archive
2024-07-16 11:26 ` [PATCH 08/11] tests/functional: Convert some avocado tests that needed avocado.utils.archive Thomas Huth
@ 2024-07-16 18:20 ` Daniel P. Berrangé
0 siblings, 0 replies; 36+ messages in thread
From: Daniel P. Berrangé @ 2024-07-16 18:20 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On Tue, Jul 16, 2024 at 01:26:11PM +0200, Thomas Huth wrote:
> Instead of using the "archive" module from avocado.utils, switch
> these tests to use the new wrapper function that is based on the
> "tarfile" module instead.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> MAINTAINERS | 6 ++---
> tests/functional/meson.build | 6 +++++
> .../test_arm_canona1100.py} | 21 +++++++++-------
> .../test_ppc_bamboo.py} | 23 ++++++++---------
> .../test_sparc64_sun4u.py} | 25 +++++++++++--------
> 5 files changed, 46 insertions(+), 35 deletions(-)
> rename tests/{avocado/machine_arm_canona1100.py => functional/test_arm_canona1100.py} (71%)
> mode change 100644 => 100755
> rename tests/{avocado/ppc_bamboo.py => functional/test_ppc_bamboo.py} (75%)
> mode change 100644 => 100755
> rename tests/{avocado/machine_sparc64_sun4u.py => functional/test_sparc64_sun4u.py} (60%)
> mode change 100644 => 100755
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH 09/11] tests/functional: Set up logging
2024-07-16 11:26 ` [PATCH 09/11] tests/functional: Set up logging Thomas Huth
@ 2024-07-16 18:22 ` Daniel P. Berrangé
0 siblings, 0 replies; 36+ messages in thread
From: Daniel P. Berrangé @ 2024-07-16 18:22 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On Tue, Jul 16, 2024 at 01:26:12PM +0200, Thomas Huth wrote:
> Create log files for each test separately, one file that contains
> the basic logging and one that contains the console output.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> tests/functional/qemu_test/__init__.py | 24 +++++++++++++++++++++++-
> 1 file changed, 23 insertions(+), 1 deletion(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH 10/11] tests/functional: Convert the s390x avocado tests into standalone tests
2024-07-16 11:26 ` [PATCH 10/11] tests/functional: Convert the s390x avocado tests into standalone tests Thomas Huth
@ 2024-07-16 18:24 ` Daniel P. Berrangé
0 siblings, 0 replies; 36+ messages in thread
From: Daniel P. Berrangé @ 2024-07-16 18:24 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On Tue, Jul 16, 2024 at 01:26:13PM +0200, Thomas Huth wrote:
> These tests use archive.lzma_uncompress() from the Avocado utils,
> so provide a small helper function for this, based on the
> standard lzma module from Python instead.
>
> And while we're at it, replace the MD5 hashes in the topology test
> with proper SHA256 hashes, since MD5 should not be used anymore
> nowadays.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> MAINTAINERS | 4 +-
> tests/functional/meson.build | 6 ++
> tests/functional/qemu_test/utils.py | 7 ++
> .../test_s390x_ccw_virtio.py} | 32 ++++-----
> .../test_s390x_topology.py} | 70 +++++++------------
> 5 files changed, 52 insertions(+), 67 deletions(-)
> rename tests/{avocado/machine_s390_ccw_virtio.py => functional/test_s390x_ccw_virtio.py} (95%)
> mode change 100644 => 100755
> rename tests/{avocado/s390_topology.py => functional/test_s390x_topology.py} (90%)
> mode change 100644 => 100755
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 025227954c..cbefb6fb81 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -1739,7 +1739,7 @@ S: Supported
> F: hw/s390x/
> F: include/hw/s390x/
> F: configs/devices/s390x-softmmu/default.mak
> -F: tests/avocado/machine_s390_ccw_virtio.py
> +F: tests/functional/test_s390x_ccw_virtio.py
> T: git https://github.com/borntraeger/qemu.git s390-next
> L: qemu-s390x@nongnu.org
>
> @@ -1802,7 +1802,7 @@ F: hw/s390x/cpu-topology.c
> F: target/s390x/kvm/stsi-topology.c
> F: docs/devel/s390-cpu-topology.rst
> F: docs/system/s390x/cpu-topology.rst
> -F: tests/avocado/s390_topology.py
> +F: tests/functional/test_s390x_topology.py
>
> X86 Machines
> ------------
> diff --git a/tests/functional/meson.build b/tests/functional/meson.build
> index c8fc9f6c07..f6de9af8a2 100644
> --- a/tests/functional/meson.build
> +++ b/tests/functional/meson.build
> @@ -13,6 +13,7 @@ endif
> test_timeouts = {
> 'netdev_ethtool' : 180,
> 'ppc_74xx' : 90,
> + 's390x_ccw_virtio' : 180,
> }
>
> tests_generic = [
> @@ -47,6 +48,11 @@ tests_ppc_thorough = [
> 'ppc_bamboo',
> ]
>
> +tests_s390x_thorough = [
> + 's390x_ccw_virtio',
> + 's390x_topology',
> +]
> +
> tests_sparc64_thorough = [
> 'sparc64_sun4u',
> ]
> diff --git a/tests/functional/qemu_test/utils.py b/tests/functional/qemu_test/utils.py
> index 4eb5e5d5e5..8115d9d1da 100644
> --- a/tests/functional/qemu_test/utils.py
> +++ b/tests/functional/qemu_test/utils.py
> @@ -8,6 +8,8 @@
> # This work is licensed under the terms of the GNU GPL, version 2 or
> # later. See the COPYING file in the top-level directory.
>
> +import lzma
> +import shutil
> import tarfile
>
> def archive_extract(archive, dest_dir, member=None):
> @@ -19,3 +21,8 @@ def archive_extract(archive, dest_dir, member=None):
> tf.extract(member=member, path=dest_dir)
> else:
> tf.extractall(path=dest_dir)
> +
> +def lzma_uncompress(xz_path, output_path):
> + with lzma.open(xz_path, 'rb') as lzma_in:
> + with open(output_path, 'wb') as raw_out:
> + shutil.copyfileobj(lzma_in, raw_out)
Avocado short-circuited if output_path already existed
for speed. Worth doing the same.
The inner 'with' should be surrounded by a try/except
that does os.remove(output_path) on error. Avocado
didn't have this safety net either, but we should add
it.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH 11/11] gitlab-ci: Add "check-functional" to the build tests
2024-07-16 11:26 ` [PATCH 11/11] gitlab-ci: Add "check-functional" to the build tests Thomas Huth
@ 2024-07-16 18:25 ` Daniel P. Berrangé
0 siblings, 0 replies; 36+ messages in thread
From: Daniel P. Berrangé @ 2024-07-16 18:25 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On Tue, Jul 16, 2024 at 01:26:14PM +0200, Thomas Huth wrote:
> Now that we converted many tests from the "check-avocado" test suite
> to the "check-functional" test suite, we should make sure that these
> also get tested in the CI.
I was thinking of suggesting we make 'check-functional' run both
avocado and new tests, but that's pointless extra work if we fully
convert all the avocado tests to the new approach in a short time
frame.
>
> Signed-off-by: Thomas Huth <thuth@redhat.com>
> ---
> .gitlab-ci.d/buildtest-template.yml | 3 +-
> .gitlab-ci.d/buildtest.yml | 60 ++++++++++++++---------------
> 2 files changed, 32 insertions(+), 31 deletions(-)
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests
2024-07-16 17:03 ` Thomas Huth
@ 2024-07-17 8:04 ` Thomas Huth
2024-07-17 8:37 ` Daniel P. Berrangé
0 siblings, 1 reply; 36+ messages in thread
From: Thomas Huth @ 2024-07-17 8:04 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On 16/07/2024 19.03, Thomas Huth wrote:
> On 16/07/2024 18.51, Daniel P. Berrangé wrote:
>> On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
> ...
>>> So instead of trying to update the python-based test suite in QEMU
>>> to a newer version of Avocado, we should maybe try to better integrate
>>> it with the meson test runner instead. Indeed most tests work quite
>>> nicely without the Avocado framework already, as you can see with
>>> this patch series - it does not convert all tests, just a subset so
>>> far, but this already proves that many tests only need small modifi-
>>> cations to work without Avocado.
> ...
>>> Now if you want to try out these patches: Apply the patches, then
>>> recompile and then run:
>>>
>>> make check-functional
>>>
>>> You can also run single targets e.g. with:
>>>
>>> make check-functional-ppc
>>>
>>> You can also run the tests without any test runner now by
>>> setting the PYTHONPATH environment variable to the "python" folder
>>> of your source tree, and by specifying the build directory via
>>> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
>>> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>>>
>>> export PYTHONPATH=$HOME/qemu/python
>>> export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
>>> export PYTHONPATH=$HOME/qemu/build
>>> ~/qemu/tests/functional/test_virtio_version.py
>>
>> For the whole series as is
>>
>> Tested-by: Daniel P. Berrangé <berrange@redhat.com>
>>
>> as it does what you claim it does here when I tried it.
>
> Thanks!
>
>>> The logs of the tests can be found in the build directory under
>>> tests/functional/<testname> - console log and general logs will
>>> be put in separate files there.
>>
>> As an example, one dir name appears to be:
>>
>> __main__.MemAddrCheck.test_phybits_ok_pentium_pae
>>
>> I'd rather prefer it if the dir name matched the test script
>> file name - in this case test_mem_addr_space.py, as I don't
>> want to have to lookup which class names were defined inside
>> each test script. We could drop the "test_" prefix from the
>> method name too
>>
>> IOW, could we make this dir name be:
>>
>> test_mem_addr_space.phybits_ok_pentium_pae
>
> I can try to change that, indeed ... but the boilerplate code will increase
> a little bit, I guess, since I cannot simply rely on the unittest.id()
> function in that case anymore...
After looking at this for a while, I think it's maybe best to ditch the idea
of making the .py files directly runnable and run the tests via a simple
pycotap runner instead. Then you get proper module names:
$ pyvenv/bin/python3 -m pycotap test_virtio_version
TAP version 13
ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs
ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs
1..2
Thomas
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests
2024-07-17 8:04 ` Thomas Huth
@ 2024-07-17 8:37 ` Daniel P. Berrangé
2024-07-17 8:53 ` Thomas Huth
0 siblings, 1 reply; 36+ messages in thread
From: Daniel P. Berrangé @ 2024-07-17 8:37 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On Wed, Jul 17, 2024 at 10:04:19AM +0200, Thomas Huth wrote:
> On 16/07/2024 19.03, Thomas Huth wrote:
> > On 16/07/2024 18.51, Daniel P. Berrangé wrote:
> > > On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
> > ...
> > > > So instead of trying to update the python-based test suite in QEMU
> > > > to a newer version of Avocado, we should maybe try to better integrate
> > > > it with the meson test runner instead. Indeed most tests work quite
> > > > nicely without the Avocado framework already, as you can see with
> > > > this patch series - it does not convert all tests, just a subset so
> > > > far, but this already proves that many tests only need small modifi-
> > > > cations to work without Avocado.
> > ...
> > > > Now if you want to try out these patches: Apply the patches, then
> > > > recompile and then run:
> > > >
> > > > make check-functional
> > > >
> > > > You can also run single targets e.g. with:
> > > >
> > > > make check-functional-ppc
> > > >
> > > > You can also run the tests without any test runner now by
> > > > setting the PYTHONPATH environment variable to the "python" folder
> > > > of your source tree, and by specifying the build directory via
> > > > QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
> > > > QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
> > > >
> > > > export PYTHONPATH=$HOME/qemu/python
> > > > export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
> > > > export PYTHONPATH=$HOME/qemu/build
> > > > ~/qemu/tests/functional/test_virtio_version.py
> > >
> > > For the whole series as is
> > >
> > > Tested-by: Daniel P. Berrangé <berrange@redhat.com>
> > >
> > > as it does what you claim it does here when I tried it.
> >
> > Thanks!
> >
> > > > The logs of the tests can be found in the build directory under
> > > > tests/functional/<testname> - console log and general logs will
> > > > be put in separate files there.
> > >
> > > As an example, one dir name appears to be:
> > >
> > > __main__.MemAddrCheck.test_phybits_ok_pentium_pae
> > >
> > > I'd rather prefer it if the dir name matched the test script
> > > file name - in this case test_mem_addr_space.py, as I don't
> > > want to have to lookup which class names were defined inside
> > > each test script. We could drop the "test_" prefix from the
> > > method name too
> > >
> > > IOW, could we make this dir name be:
> > >
> > > test_mem_addr_space.phybits_ok_pentium_pae
> >
> > I can try to change that, indeed ... but the boilerplate code will
> > increase a little bit, I guess, since I cannot simply rely on the
> > unittest.id() function in that case anymore...
>
> After looking at this for a while, I think it's maybe best to ditch the idea
> of making the .py files directly runnable and run the tests via a simple
> pycotap runner instead. Then you get proper module names:
I'd really not want to loose that. To me, eliminating the test harness
entirely when debugging is the single biggest improvement of this new
approach, especially when I want to 'strace' the test without
extraneous processes.
> $ pyvenv/bin/python3 -m pycotap test_virtio_version
> TAP version 13
> ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs
> ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs
> 1..
With the following change, you get the same output with direct
execution, by making argv look the same as you'd get when
running your pycotap example.
diff --git a/tests/functional/qemu_test/__init__.py b/tests/functional/qemu_test/__init__.py
index cc49fd4c94..3a3e65252d 100644
--- a/tests/functional/qemu_test/__init__.py
+++ b/tests/functional/qemu_test/__init__.py
@@ -266,7 +266,10 @@ def fetch_asset(self, url, asset_hash):
def main():
tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
test_output_log = pycotap.LogMode.LogToError)
- unittest.main(testRunner = tr)
+ import sys
+ import os.path
+ path = os.path.basename(sys.argv[0])[:-3]
+ unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])
class QemuSystemTest(QemuBaseTest):
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply related [flat|nested] 36+ messages in thread
* Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests
2024-07-17 8:37 ` Daniel P. Berrangé
@ 2024-07-17 8:53 ` Thomas Huth
0 siblings, 0 replies; 36+ messages in thread
From: Thomas Huth @ 2024-07-17 8:53 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, John Snow,
Cédric Le Goater, Fabiano Rosas
On 17/07/2024 10.37, Daniel P. Berrangé wrote:
> On Wed, Jul 17, 2024 at 10:04:19AM +0200, Thomas Huth wrote:
>> On 16/07/2024 19.03, Thomas Huth wrote:
>>> On 16/07/2024 18.51, Daniel P. Berrangé wrote:
>>>> On Tue, Jul 16, 2024 at 01:26:03PM +0200, Thomas Huth wrote:
>>> ...
>>>>> So instead of trying to update the python-based test suite in QEMU
>>>>> to a newer version of Avocado, we should maybe try to better integrate
>>>>> it with the meson test runner instead. Indeed most tests work quite
>>>>> nicely without the Avocado framework already, as you can see with
>>>>> this patch series - it does not convert all tests, just a subset so
>>>>> far, but this already proves that many tests only need small modifi-
>>>>> cations to work without Avocado.
>>> ...
>>>>> Now if you want to try out these patches: Apply the patches, then
>>>>> recompile and then run:
>>>>>
>>>>> make check-functional
>>>>>
>>>>> You can also run single targets e.g. with:
>>>>>
>>>>> make check-functional-ppc
>>>>>
>>>>> You can also run the tests without any test runner now by
>>>>> setting the PYTHONPATH environment variable to the "python" folder
>>>>> of your source tree, and by specifying the build directory via
>>>>> QEMU_BUILD_ROOT (if autodetection fails) and by specifying the
>>>>> QEMU binary via QEMU_TEST_QEMU_BINARY. For example:
>>>>>
>>>>> export PYTHONPATH=$HOME/qemu/python
>>>>> export QEMU_TEST_QEMU_BINARY=qemu-system-x86_64
>>>>> export PYTHONPATH=$HOME/qemu/build
>>>>> ~/qemu/tests/functional/test_virtio_version.py
>>>>
>>>> For the whole series as is
>>>>
>>>> Tested-by: Daniel P. Berrangé <berrange@redhat.com>
>>>>
>>>> as it does what you claim it does here when I tried it.
>>>
>>> Thanks!
>>>
>>>>> The logs of the tests can be found in the build directory under
>>>>> tests/functional/<testname> - console log and general logs will
>>>>> be put in separate files there.
>>>>
>>>> As an example, one dir name appears to be:
>>>>
>>>> __main__.MemAddrCheck.test_phybits_ok_pentium_pae
>>>>
>>>> I'd rather prefer it if the dir name matched the test script
>>>> file name - in this case test_mem_addr_space.py, as I don't
>>>> want to have to lookup which class names were defined inside
>>>> each test script. We could drop the "test_" prefix from the
>>>> method name too
>>>>
>>>> IOW, could we make this dir name be:
>>>>
>>>> test_mem_addr_space.phybits_ok_pentium_pae
>>>
>>> I can try to change that, indeed ... but the boilerplate code will
>>> increase a little bit, I guess, since I cannot simply rely on the
>>> unittest.id() function in that case anymore...
>>
>> After looking at this for a while, I think it's maybe best to ditch the idea
>> of making the .py files directly runnable and run the tests via a simple
>> pycotap runner instead. Then you get proper module names:
>
> I'd really not want to loose that. To me, eliminating the test harness
> entirely when debugging is the single biggest improvement of this new
> approach, especially when I want to 'strace' the test without
> extraneous processes.
>
>> $ pyvenv/bin/python3 -m pycotap test_virtio_version
>> TAP version 13
>> ok 1 test_virtio_version.VirtioVersionCheck.test_conventional_devs
>> ok 2 test_virtio_version.VirtioVersionCheck.test_modern_only_devs
>> 1..
>
> With the following change, you get the same output with direct
> execution, by making argv look the same as you'd get when
> running your pycotap example.
>
> diff --git a/tests/functional/qemu_test/__init__.py b/tests/functional/qemu_test/__init__.py
> index cc49fd4c94..3a3e65252d 100644
> --- a/tests/functional/qemu_test/__init__.py
> +++ b/tests/functional/qemu_test/__init__.py
> @@ -266,7 +266,10 @@ def fetch_asset(self, url, asset_hash):
> def main():
> tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
> test_output_log = pycotap.LogMode.LogToError)
> - unittest.main(testRunner = tr)
> + import sys
> + import os.path
> + path = os.path.basename(sys.argv[0])[:-3]
> + unittest.main(module = None, testRunner = tr, argv=["__dummy__", path])
Sweet, thank you very much, looks like this will do the job!
Thomas
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH 04/11] tests/functional: Add python-based tests to the meson build system
2024-07-16 15:15 ` Fabiano Rosas
@ 2024-07-17 9:20 ` Thomas Huth
0 siblings, 0 replies; 36+ messages in thread
From: Thomas Huth @ 2024-07-17 9:20 UTC (permalink / raw)
To: Fabiano Rosas, Alex Bennée, qemu-devel,
Philippe Mathieu-Daudé
Cc: Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater
On 16/07/2024 17.15, Fabiano Rosas wrote:
> Thomas Huth <thuth@redhat.com> writes:
>
>> Integrate the new python-based test framework with the meson build
>> system. Since these tests now require the pycotap module, make
>> sure that it gets installed in the venv.
>>
>> The changes to the meson.build files are partly based on an earlier
>> patch by Ani Sinha (but heavily modified by Thomas Huth e.g. to use
>> pycotap for running the tests instead).
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>
> Just one touch-up below.
>
> Reviewed-by: Fabiano Rosas <farosas@suse.de>
>
>> ---
>> pythondeps.toml | 3 +-
>> tests/Makefile.include | 18 ++++++++-
>> tests/functional/meson.build | 75 ++++++++++++++++++++++++++++++++++++
>> tests/meson.build | 1 +
>> 4 files changed, 95 insertions(+), 2 deletions(-)
>> create mode 100644 tests/functional/meson.build
>>
>> diff --git a/pythondeps.toml b/pythondeps.toml
>> index f6e590fdd8..c018b4d74a 100644
>> --- a/pythondeps.toml
>> +++ b/pythondeps.toml
>> @@ -26,9 +26,10 @@ meson = { accepted = ">=1.1.0", installed = "1.2.3", canary = "meson" }
>> sphinx = { accepted = ">=3.4.3", installed = "5.3.0", canary = "sphinx-build" }
>> sphinx_rtd_theme = { accepted = ">=0.5", installed = "1.1.1" }
>>
>> -[avocado]
>> +[tests]
>> # Note that qemu.git/python/ is always implicitly installed.
>> # Prefer an LTS version when updating the accepted versions of
>> # avocado-framework, for example right now the limit is 92.x.
>> avocado-framework = { accepted = "(>=88.1, <93.0)", installed = "88.1", canary = "avocado" }
>> pycdlib = { accepted = ">=1.11.0" }
>> +pycotap = { accepted = ">=1.1.0" }
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index d39d5dd6a4..2bdf607977 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -9,6 +9,8 @@ check-help:
>> @echo "Individual test suites:"
>> @echo " $(MAKE) check-qtest-TARGET Run qtest tests for given target"
>> @echo " $(MAKE) check-qtest Run qtest tests"
>> + @echo " $(MAKE) check-functional Run python-based functional tests"
>> + @echo " $(MAKE) check-functional-TARG Run functional tests for
>> a given target"
>> @echo " $(MAKE) check-unit Run qobject tests"
>> @echo " $(MAKE) check-qapi-schema Run QAPI schema tests"
>> @echo " $(MAKE) check-block Run block tests"
>> @@ -111,7 +113,7 @@ quiet-venv-pip = $(quiet-@)$(call quiet-command-run, \
>>
>> $(TESTS_VENV_TOKEN): $(SRC_PATH)/pythondeps.toml
>> $(call quiet-venv-pip,install -e "$(SRC_PATH)/python/")
>> - $(MKVENV_ENSUREGROUP) $< avocado
>> + $(MKVENV_ENSUREGROUP) $< tests
>> $(call quiet-command, touch $@)
>>
>> $(TESTS_RESULTS_DIR):
>> @@ -152,6 +154,20 @@ check-acceptance-deprecated-warning:
>>
>> check-acceptance: check-acceptance-deprecated-warning | check-avocado
>>
>> +# Make sure that pycotap is installed before running any functional tests:
>> +ifneq ($(filter check-func%,$(MAKECMDGOALS))$(filter check,$(MAKECMDGOALS)),)
>> +do-meson-check: check-venv
>> +endif
>> +
>> +FUNCTIONAL_TARGETS=$(patsubst %-softmmu,check-functional-%, $(filter %-softmmu,$(TARGETS)))
>> +.PHONY: $(FUNCTIONAL_TARGETS)
>> +$(FUNCTIONAL_TARGETS):
>> + @make SPEED=thorough $(subst -functional,-func,$@)
>> +
>> +.PHONY: check-functional
>> +check-functional:
>> + @make SPEED=thorough check-func check-func-quick
>
> I think these^ two should use $(MAKE) instead:
>
> make[1]: warning: jobserver unavailable: using -j1. Add '+' to parent
> make rule.
Right, thanks! I'll change it in the next version.
Thomas
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests
2024-07-16 17:57 ` Paolo Bonzini
@ 2024-07-24 11:43 ` Thomas Huth
0 siblings, 0 replies; 36+ messages in thread
From: Thomas Huth @ 2024-07-24 11:43 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Daniel P . Berrange, John Snow,
Cédric Le Goater, Fabiano Rosas
On 16/07/2024 19.57, Paolo Bonzini wrote:
>
>
> Il mar 16 lug 2024, 13:26 Thomas Huth <thuth@redhat.com
> <mailto:thuth@redhat.com>> ha scritto:
>
> The Avocado v88 that we use in QEMU is already on a life support
> system: It is not supported by upstream anymore, and with the latest
> versions of Python, it won't work anymore since it depends on the
> "imp" module that has been removed in Python 3.12.
>
> There have been several attempts to update the test suite in QEMU
> to a newer version of Avocado, but so far no attempt has successfully
> been merged yet.
>
>
> I think we should take another look at that. Avocado 92 should work, though
> I am not sure if it's also depending on "imp"
FWIW, the "imp" problem had been fixed here:
https://github.com/avocado-framework/avocado/commit/dfc340f1a0b0aeddbf6699e49e94b75ed080137c
That had just been released with Avocado version 99, so updating to 92 does
not really help.
Thomas
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
` (12 preceding siblings ...)
2024-07-16 17:57 ` Paolo Bonzini
@ 2024-07-25 14:21 ` Cleber Rosa
2024-07-26 10:07 ` Thomas Huth
2024-07-29 12:44 ` Daniel P. Berrangé
13 siblings, 2 replies; 36+ messages in thread
From: Cleber Rosa @ 2024-07-25 14:21 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote:
>
> The Avocado v88 that we use in QEMU is already on a life support
> system: It is not supported by upstream anymore, and with the latest
> versions of Python, it won't work anymore since it depends on the
> "imp" module that has been removed in Python 3.12.
>
Hi Thomas,
Let me start by saying that this is my attempt to explain the context
of the lack of updates on QEMU when it comes to the Avocado update,
and point to the actual updates.
> There have been several attempts to update the test suite in QEMU
> to a newer version of Avocado, but so far no attempt has successfully
> been merged yet.
>
So, we've seen in the past an attempt to update Avocado from 88.1 to a
regular release, and the troubles it caused, including a revert. My
take was that a LTS version should be used, but during this time,
Avocado experienced a rewrite and having it replacing the old
implementation in a production level project such as QEMU was tricky.
Then 103.0 LTS was released, and there was extensive work to test the
QEMU tests before that release was cut. Additionally, there was
further work, but unfortunately not posted yet, to make use of 103.0
features in the existing tests[2]. I've tested on GitLab with tests
running in parallel, cutting job times in 1/3[2]. A side node is
that, because 103.0 is an LTS release, it will receive the needed bug
fixes and updates that we deem necessary, including things we find in
QEMU tests. In fact, 103.1[3] is in the works.
> Additionally, the whole "make check" test suite in QEMU is using the
> meson test runner nowadays, so running the python-based tests via the
> Avocodo test runner looks and feels quite like an oddball, requiring
> the users to deal with the knowledge of multiple test runners in
> parallel (e.g. the timeout settings work completely differently).
>
Now I believe we can be very much in sync here. I've thought for a
while that there's no reason for Avocado to cooperate or be compatible
with Meson. There's no reason why users can't simply pick how the
test gets run. In fact, with the new Avocado architecture, you don't
even need to run "avocado" to run an "avocado-instrumented" test. You
could pretty much run "avocado-runner-avocado-instrumented" with the
right parameters through Meson.
> So instead of trying to update the python-based test suite in QEMU
> to a newer version of Avocado, we should maybe try to better integrate
> it with the meson test runner instead. Indeed most tests work quite
> nicely without the Avocado framework already, as you can see with
> this patch series - it does not convert all tests, just a subset so
> far, but this already proves that many tests only need small modifi-
> cations to work without Avocado.
>
> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> classes (e.g. based on cloud-init images or using SSH) really depend
> on the Avocado framework, so we'd need a solution for those if we
> want to continue using them. One solution might be to simply use the
> required functions from avocado.utils for these tests, and still run
> them via the meson test runner instead, but that needs some further
> investigation that will be done later.
>
So, I believe this type of higher level testing is something that
needs to remain, and even grow. Speaking for Red Hat, I see the
movement of QE contributing more Avocado-VT style tests into QEMU
itself. This means way more libraries and features that go into a
common set of utilities and features (more on that later) than it
currently exists in avocado.utils.
This brings the autils[4] initiative into the picture. We're about
80% done with the project structure, and after that, it will be a
common utility project (such as the cloudinit and ssh) which can be
released automatically when the maintainer votes (through GitHub) that
a new release is needed. With that, we can have all the existing QEMU
tests (and more) running in either meson or Avocado (for users that
want Avocado runner only features, such as dependency resolution,
container support, etc).
Now, if you think any of this makes sense, I'd love to cooperate with
you on an update that does both: update Avocado, and allow for the
execution of all tests through meson.
Thanks and I'm very very sorry for the delay in this update.
- Cleber.
[1] https://gitlab.com/cleber.gnu/qemu/-/commits/WIP/avocado_test_updates_lts?ref_type=heads
[2] https://gitlab.com/cleber.gnu/qemu/-/jobs/7085879478
[3] https://github.com/avocado-framework/avocado/pull/5953
[4] https://github.com/avocado-framework/autils
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests
2024-07-25 14:21 ` Cleber Rosa
@ 2024-07-26 10:07 ` Thomas Huth
2024-07-26 13:56 ` Cleber Rosa
2024-07-29 12:44 ` Daniel P. Berrangé
1 sibling, 1 reply; 36+ messages in thread
From: Thomas Huth @ 2024-07-26 10:07 UTC (permalink / raw)
To: Cleber Rosa
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
On 25/07/2024 16.21, Cleber Rosa wrote:
> On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote:
...
>> There have been several attempts to update the test suite in QEMU
>> to a newer version of Avocado, but so far no attempt has successfully
>> been merged yet.
>
> So, we've seen in the past an attempt to update Avocado from 88.1 to a
> regular release, and the troubles it caused, including a revert. My
> take was that a LTS version should be used, but during this time,
> Avocado experienced a rewrite and having it replacing the old
> implementation in a production level project such as QEMU was tricky.
> Then 103.0 LTS was released, and there was extensive work to test the
> QEMU tests before that release was cut. Additionally, there was
> further work, but unfortunately not posted yet, to make use of 103.0
> features in the existing tests[2]. I've tested on GitLab with tests
> running in parallel, cutting job times in 1/3[2]. A side node is
> that, because 103.0 is an LTS release, it will receive the needed bug
> fixes and updates that we deem necessary, including things we find in
> QEMU tests. In fact, 103.1[3] is in the works.
Hi Cleber,
thanks for the explanation, but we really need to replace v88 rather *now*
since v88 does not work with the latest versions of Python anymore (there is
a work-around on Fedora fortunately, but it's completely broken on Ubuntu
24.04 as far as I know). So even a single-threaded execution with v103 would
have been better than waiting forever for your update to land. The problem
with v88 being broken has been raised a couple of times already, but it's
incredibly hard to get a response from you Avocado folks, so with hardly any
help from the Avocado side, and nobody on the QEMU side being really
familiar with the Avocado stuff, and with the meson test runner being used
by all other subsystems in QEMU already, I think it's best if we continue
with this series here.
>> Additionally, the whole "make check" test suite in QEMU is using the
>> meson test runner nowadays, so running the python-based tests via the
>> Avocodo test runner looks and feels quite like an oddball, requiring
>> the users to deal with the knowledge of multiple test runners in
>> parallel (e.g. the timeout settings work completely differently).
>>
>
> Now I believe we can be very much in sync here. I've thought for a
> while that there's no reason for Avocado to cooperate or be compatible
> with Meson. There's no reason why users can't simply pick how the
> test gets run. In fact, with the new Avocado architecture, you don't
> even need to run "avocado" to run an "avocado-instrumented" test. You
> could pretty much run "avocado-runner-avocado-instrumented" with the
> right parameters through Meson.
Ok, good to know, we could maybe use that eventually for the tests that
really require the Avocado framework (i.e. the cloud-init based tests).
For the others, as Daniel said in an earlier mail, it's much more convenient
if you can also run the tests directly instead, without such a layer in
between - that makes debugging way easier.
>> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
>> classes (e.g. based on cloud-init images or using SSH) really depend
>> on the Avocado framework, so we'd need a solution for those if we
>> want to continue using them. One solution might be to simply use the
>> required functions from avocado.utils for these tests, and still run
>> them via the meson test runner instead, but that needs some further
>> investigation that will be done later.
>
> So, I believe this type of higher level testing is something that
> needs to remain, and even grow. Speaking for Red Hat, I see the
> movement of QE contributing more Avocado-VT style tests into QEMU
> itself.
I didn't really see such a movement in recent times yet ... Could you point
to an example?
> This means way more libraries and features that go into a
> common set of utilities and features (more on that later) than it
> currently exists in avocado.utils.
>
> This brings the autils[4] initiative into the picture. We're about
> 80% done with the project structure, and after that, it will be a
> common utility project (such as the cloudinit and ssh) which can be
> released automatically when the maintainer votes (through GitHub) that
> a new release is needed.
autils sounds promising, but I just had a look at the repository, and there
does not seem to be that much code available there yet, so I guess it will
take still quite a long time 'til that's ready?
Thomas
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests
2024-07-26 10:07 ` Thomas Huth
@ 2024-07-26 13:56 ` Cleber Rosa
0 siblings, 0 replies; 36+ messages in thread
From: Cleber Rosa @ 2024-07-26 13:56 UTC (permalink / raw)
To: Thomas Huth
Cc: Alex Bennée, qemu-devel, Philippe Mathieu-Daudé,
Ani Sinha, Richard Henderson, Paolo Bonzini, Daniel P . Berrange,
John Snow, Cédric Le Goater, Fabiano Rosas
On Fri, Jul 26, 2024 at 6:07 AM Thomas Huth <thuth@redhat.com> wrote:
>
> On 25/07/2024 16.21, Cleber Rosa wrote:
> > On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote:
> ...
> >> There have been several attempts to update the test suite in QEMU
> >> to a newer version of Avocado, but so far no attempt has successfully
> >> been merged yet.
> >
> > So, we've seen in the past an attempt to update Avocado from 88.1 to a
> > regular release, and the troubles it caused, including a revert. My
> > take was that a LTS version should be used, but during this time,
> > Avocado experienced a rewrite and having it replacing the old
> > implementation in a production level project such as QEMU was tricky.
> > Then 103.0 LTS was released, and there was extensive work to test the
> > QEMU tests before that release was cut. Additionally, there was
> > further work, but unfortunately not posted yet, to make use of 103.0
> > features in the existing tests[2]. I've tested on GitLab with tests
> > running in parallel, cutting job times in 1/3[2]. A side node is
> > that, because 103.0 is an LTS release, it will receive the needed bug
> > fixes and updates that we deem necessary, including things we find in
> > QEMU tests. In fact, 103.1[3] is in the works.
>
> Hi Cleber,
>
> thanks for the explanation, but we really need to replace v88 rather *now*
> since v88 does not work with the latest versions of Python anymore (there is
> a work-around on Fedora fortunately, but it's completely broken on Ubuntu
> 24.04 as far as I know). So even a single-threaded execution with v103 would
> have been better than waiting forever for your update to land. The problem
> with v88 being broken has been raised a couple of times already, but it's
> incredibly hard to get a response from you Avocado folks, so with hardly any
> help from the Avocado side, and nobody on the QEMU side being really
> familiar with the Avocado stuff, and with the meson test runner being used
> by all other subsystems in QEMU already, I think it's best if we continue
> with this series here.
>
Hi Thomas,
I agree with the urgency. I've posted the updates here:
https://lists.gnu.org/archive/html/qemu-devel/2024-07/msg06236.html
Sorry for the delay, and I hope this gives more time to make the
evolution towards a solution that better suits QEMU.
Regards,
- Cleber.
> >> Additionally, the whole "make check" test suite in QEMU is using the
> >> meson test runner nowadays, so running the python-based tests via the
> >> Avocodo test runner looks and feels quite like an oddball, requiring
> >> the users to deal with the knowledge of multiple test runners in
> >> parallel (e.g. the timeout settings work completely differently).
> >>
> >
> > Now I believe we can be very much in sync here. I've thought for a
> > while that there's no reason for Avocado to cooperate or be compatible
> > with Meson. There's no reason why users can't simply pick how the
> > test gets run. In fact, with the new Avocado architecture, you don't
> > even need to run "avocado" to run an "avocado-instrumented" test. You
> > could pretty much run "avocado-runner-avocado-instrumented" with the
> > right parameters through Meson.
>
> Ok, good to know, we could maybe use that eventually for the tests that
> really require the Avocado framework (i.e. the cloud-init based tests).
>
> For the others, as Daniel said in an earlier mail, it's much more convenient
> if you can also run the tests directly instead, without such a layer in
> between - that makes debugging way easier.
>
> >> Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> >> classes (e.g. based on cloud-init images or using SSH) really depend
> >> on the Avocado framework, so we'd need a solution for those if we
> >> want to continue using them. One solution might be to simply use the
> >> required functions from avocado.utils for these tests, and still run
> >> them via the meson test runner instead, but that needs some further
> >> investigation that will be done later.
> >
> > So, I believe this type of higher level testing is something that
> > needs to remain, and even grow. Speaking for Red Hat, I see the
> > movement of QE contributing more Avocado-VT style tests into QEMU
> > itself.
>
> I didn't really see such a movement in recent times yet ... Could you point
> to an example?
>
Indeed, there is no public evidence of that, yet.
> > This means way more libraries and features that go into a
> > common set of utilities and features (more on that later) than it
> > currently exists in avocado.utils.
> >
> > This brings the autils[4] initiative into the picture. We're about
> > 80% done with the project structure, and after that, it will be a
> > common utility project (such as the cloudinit and ssh) which can be
> > released automatically when the maintainer votes (through GitHub) that
> > a new release is needed.
>
> autils sounds promising, but I just had a look at the repository, and there
> does not seem to be that much code available there yet, so I guess it will
> take still quite a long time 'til that's ready?
>
Like I said previously, we were working on the project infrastructure.
We haven't ported much code over there yet. There are very few things
missing, and all of them have been assigned to be worked on. It's
been tracked here:
https://github.com/avocado-framework/autils/issues/1
My estimation is that 2-4 weeks for the project infrastructure to be complete.
Regards,
- Cleber.
> Thomas
>
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests
2024-07-25 14:21 ` Cleber Rosa
2024-07-26 10:07 ` Thomas Huth
@ 2024-07-29 12:44 ` Daniel P. Berrangé
2024-07-29 14:01 ` Cleber Rosa
1 sibling, 1 reply; 36+ messages in thread
From: Daniel P. Berrangé @ 2024-07-29 12:44 UTC (permalink / raw)
To: Cleber Rosa
Cc: Thomas Huth, Alex Bennée, qemu-devel,
Philippe Mathieu-Daudé, Ani Sinha, Richard Henderson,
Paolo Bonzini, John Snow, Cédric Le Goater, Fabiano Rosas
On Thu, Jul 25, 2024 at 10:21:54AM -0400, Cleber Rosa wrote:
> On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote:
> > There have been several attempts to update the test suite in QEMU
> > to a newer version of Avocado, but so far no attempt has successfully
> > been merged yet.
> >
>
> So, we've seen in the past an attempt to update Avocado from 88.1 to a
> regular release, and the troubles it caused, including a revert. My
> take was that a LTS version should be used, but during this time,
> Avocado experienced a rewrite and having it replacing the old
> implementation in a production level project such as QEMU was tricky.
IMHO the problems with updating avocado in QEMU are a sign that we're
suffering from excess complexity, as it shouldn't be this difficult to
update the test harness. That we need to stick on an LTS release to
protect us from further instability further suggests that we would
benefit from a much simpler solution here.
> > Additionally, the whole "make check" test suite in QEMU is using the
> > meson test runner nowadays, so running the python-based tests via the
> > Avocodo test runner looks and feels quite like an oddball, requiring
> > the users to deal with the knowledge of multiple test runners in
> > parallel (e.g. the timeout settings work completely differently).
> >
>
> Now I believe we can be very much in sync here. I've thought for a
> while that there's no reason for Avocado to cooperate or be compatible
> with Meson. There's no reason why users can't simply pick how the
> test gets run. In fact, with the new Avocado architecture, you don't
> even need to run "avocado" to run an "avocado-instrumented" test. You
> could pretty much run "avocado-runner-avocado-instrumented" with the
> right parameters through Meson.
Looking back to when we introduced avocado, we still had our original
home grown build system, and multiple different ways to run integrate
testing betwen our unit tests / qtests and block iotests. Adding
avocado to the mix introduced a third way to run tests. All round it
was rather a mess of inconsistent approaches to testing, but that was
a fact of life with our old home grown build/test process.
Today we have adopted a much more standardized build system via meson.
While we do still have the block iotests harness, we now expose every
test to meson individually to reduce what the intermediate harness
needs to be involved in doing. Thomas' series here is bringing the same
simplified integration to our functional tests, such that each functional
test is known to meson individually. Overall we'll now have a consistent
approach to integrating tests into meson, without intermediate harnesses
adding their own logic.
I think this will make it easier for QEMU maintainers to understand how
the tests are working, and make it easier to maintain them and troubleshoot
failures.
So overall I'm not seeing a compelling reason for us to offer users a
choice between meson & avocado how they run tests. Such choice is a maint
burden over the long term, as we need to validate that both options keep
working. Focusing on a single option reduces the complexity and maint
work and gives us more flexiblity to evolve the code over time.
> > So instead of trying to update the python-based test suite in QEMU
> > to a newer version of Avocado, we should maybe try to better integrate
> > it with the meson test runner instead. Indeed most tests work quite
> > nicely without the Avocado framework already, as you can see with
> > this patch series - it does not convert all tests, just a subset so
> > far, but this already proves that many tests only need small modifi-
> > cations to work without Avocado.
> >
> > Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> > classes (e.g. based on cloud-init images or using SSH) really depend
> > on the Avocado framework, so we'd need a solution for those if we
> > want to continue using them. One solution might be to simply use the
> > required functions from avocado.utils for these tests, and still run
> > them via the meson test runner instead, but that needs some further
> > investigation that will be done later.
> >
>
> So, I believe this type of higher level testing is something that
> needs to remain, and even grow. Speaking for Red Hat, I see the
> movement of QE contributing more Avocado-VT style tests into QEMU
> itself. This means way more libraries and features that go into a
> common set of utilities and features (more on that later) than it
> currently exists in avocado.utils.
The series here is showing how simple it is to adapt our existing
tests to the new execution approach, so shouldn't have a significant
impact on ability of people to contribute futher tests in future.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 36+ messages in thread
* Re: [PATCH v1 00/11] Convert avocado tests to normal Python unittests
2024-07-29 12:44 ` Daniel P. Berrangé
@ 2024-07-29 14:01 ` Cleber Rosa
0 siblings, 0 replies; 36+ messages in thread
From: Cleber Rosa @ 2024-07-29 14:01 UTC (permalink / raw)
To: Daniel P. Berrangé
Cc: Thomas Huth, Alex Bennée, qemu-devel,
Philippe Mathieu-Daudé, Ani Sinha, Richard Henderson,
Paolo Bonzini, John Snow, Cédric Le Goater, Fabiano Rosas
On Mon, Jul 29, 2024 at 8:44 AM Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> On Thu, Jul 25, 2024 at 10:21:54AM -0400, Cleber Rosa wrote:
> > On Tue, Jul 16, 2024 at 7:28 AM Thomas Huth <thuth@redhat.com> wrote:
> > > There have been several attempts to update the test suite in QEMU
> > > to a newer version of Avocado, but so far no attempt has successfully
> > > been merged yet.
> > >
> >
> > So, we've seen in the past an attempt to update Avocado from 88.1 to a
> > regular release, and the troubles it caused, including a revert. My
> > take was that a LTS version should be used, but during this time,
> > Avocado experienced a rewrite and having it replacing the old
> > implementation in a production level project such as QEMU was tricky.
>
> IMHO the problems with updating avocado in QEMU are a sign that we're
> suffering from excess complexity, as it shouldn't be this difficult to
> update the test harness. That we need to stick on an LTS release to
> protect us from further instability further suggests that we would
> benefit from a much simpler solution here.
>
Hi Daniel,
Some other projects using Avocado have picked different, non-LTS
versions, to use as their pinned versions. The consequences of using
rolling releases versus LTS-like releases is not something specific to
one software package IMO. The suggestion to use LTS was to minimize
the impact on users writing and running tests, while allowing for only
needed fixes or very important features (say supporting a new Python
version) to be addressed.
> > > Additionally, the whole "make check" test suite in QEMU is using the
> > > meson test runner nowadays, so running the python-based tests via the
> > > Avocodo test runner looks and feels quite like an oddball, requiring
> > > the users to deal with the knowledge of multiple test runners in
> > > parallel (e.g. the timeout settings work completely differently).
> > >
> >
> > Now I believe we can be very much in sync here. I've thought for a
> > while that there's no reason for Avocado to cooperate or be compatible
> > with Meson. There's no reason why users can't simply pick how the
> > test gets run. In fact, with the new Avocado architecture, you don't
> > even need to run "avocado" to run an "avocado-instrumented" test. You
> > could pretty much run "avocado-runner-avocado-instrumented" with the
> > right parameters through Meson.
>
> Looking back to when we introduced avocado, we still had our original
> home grown build system, and multiple different ways to run integrate
> testing betwen our unit tests / qtests and block iotests. Adding
> avocado to the mix introduced a third way to run tests. All round it
> was rather a mess of inconsistent approaches to testing, but that was
> a fact of life with our old home grown build/test process.
>
Agreed.
> Today we have adopted a much more standardized build system via meson.
> While we do still have the block iotests harness, we now expose every
> test to meson individually to reduce what the intermediate harness
> needs to be involved in doing. Thomas' series here is bringing the same
> simplified integration to our functional tests, such that each functional
> test is known to meson individually. Overall we'll now have a consistent
> approach to integrating tests into meson, without intermediate harnesses
> adding their own logic.
>
> I think this will make it easier for QEMU maintainers to understand how
> the tests are working, and make it easier to maintain them and troubleshoot
> failures.
>
I think this is a very good way forward. I don't think anything
(simple tests here) should depend on what they don't need to depend
on.
> So overall I'm not seeing a compelling reason for us to offer users a
> choice between meson & avocado how they run tests. Such choice is a maint
> burden over the long term, as we need to validate that both options keep
> working. Focusing on a single option reduces the complexity and maint
> work and gives us more flexiblity to evolve the code over time.
>
>
If meson is capable of doing all that can be done right now, then I
believe you're right. No reason to use duplicate software. Now, If
you mean that having "single option" may imply duplicating code that
already exists and is maintained elsewhere, into the QEMU repo, then
you're wrong. This can be easily judged by how easy it is to convert
all current tests and their features (timeout, download management,
etc).
Another level of judgement that I believe would be helpful to the
decision making process here is how this impacts growing the test
coverage. Tests that manipulate both QEMU and the guest are still
very very simple compared to what can be found on what I'll call "real
life integration testing" of QEMU (read Avocado-VT like tests). But
they present a glimpse of the problem. Having such tests requires a
good amount of code to either be used from a project/repo that
addresses that specifically, or be duplicated inside QEMU.
> > > So instead of trying to update the python-based test suite in QEMU
> > > to a newer version of Avocado, we should maybe try to better integrate
> > > it with the meson test runner instead. Indeed most tests work quite
> > > nicely without the Avocado framework already, as you can see with
> > > this patch series - it does not convert all tests, just a subset so
> > > far, but this already proves that many tests only need small modifi-
> > > cations to work without Avocado.
> > >
> > > Only tests that use the LinuxTest / LinuxDistro and LinuxSSHMixIn
> > > classes (e.g. based on cloud-init images or using SSH) really depend
> > > on the Avocado framework, so we'd need a solution for those if we
> > > want to continue using them. One solution might be to simply use the
> > > required functions from avocado.utils for these tests, and still run
> > > them via the meson test runner instead, but that needs some further
> > > investigation that will be done later.
> > >
> >
> > So, I believe this type of higher level testing is something that
> > needs to remain, and even grow. Speaking for Red Hat, I see the
> > movement of QE contributing more Avocado-VT style tests into QEMU
> > itself. This means way more libraries and features that go into a
> > common set of utilities and features (more on that later) than it
> > currently exists in avocado.utils.
>
> The series here is showing how simple it is to adapt our existing
> tests to the new execution approach, so shouldn't have a significant
> impact on ability of people to contribute futher tests in future.
>
I am on the side of the unified execution approach. But, will it
really be simple to convert all tests? "Real life integration" tests
too? I would love to see that, but I can't see how that can be done
without the woes mentioned before (duplicating other code or limiting
the further toolset and capabilities of test writers).
Best regards,
- Cleber.
^ permalink raw reply [flat|nested] 36+ messages in thread
end of thread, other threads:[~2024-07-29 14:02 UTC | newest]
Thread overview: 36+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-07-16 11:26 [PATCH v1 00/11] Convert avocado tests to normal Python unittests Thomas Huth
2024-07-16 11:26 ` [PATCH 01/11] tests/functional: Add base classes for the upcoming pytest-based tests Thomas Huth
2024-07-16 11:26 ` [PATCH 02/11] tests/functional: Convert simple avocado tests into standalone python tests Thomas Huth
2024-07-16 18:14 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 03/11] tests/functional: Convert avocado tests that just need a small adjustment Thomas Huth
2024-07-16 18:15 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 04/11] tests/functional: Add python-based tests to the meson build system Thomas Huth
2024-07-16 15:15 ` Fabiano Rosas
2024-07-17 9:20 ` Thomas Huth
2024-07-16 16:55 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 05/11] tests/functional: Implement fetch_asset() method for downloading assets Thomas Huth
2024-07-16 18:17 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 06/11] tests/functional: Convert some tests that download files via fetch_asset() Thomas Huth
2024-07-16 18:19 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 07/11] tests/functional: Add a function for extracting files from an archive Thomas Huth
2024-07-16 18:20 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 08/11] tests/functional: Convert some avocado tests that needed avocado.utils.archive Thomas Huth
2024-07-16 18:20 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 09/11] tests/functional: Set up logging Thomas Huth
2024-07-16 18:22 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 10/11] tests/functional: Convert the s390x avocado tests into standalone tests Thomas Huth
2024-07-16 18:24 ` Daniel P. Berrangé
2024-07-16 11:26 ` [PATCH 11/11] gitlab-ci: Add "check-functional" to the build tests Thomas Huth
2024-07-16 18:25 ` Daniel P. Berrangé
2024-07-16 16:51 ` [PATCH v1 00/11] Convert avocado tests to normal Python unittests Daniel P. Berrangé
2024-07-16 17:03 ` Thomas Huth
2024-07-17 8:04 ` Thomas Huth
2024-07-17 8:37 ` Daniel P. Berrangé
2024-07-17 8:53 ` Thomas Huth
2024-07-16 17:57 ` Paolo Bonzini
2024-07-24 11:43 ` Thomas Huth
2024-07-25 14:21 ` Cleber Rosa
2024-07-26 10:07 ` Thomas Huth
2024-07-26 13:56 ` Cleber Rosa
2024-07-29 12:44 ` Daniel P. Berrangé
2024-07-29 14:01 ` Cleber Rosa
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).