[PATCH v3 0/4] tests/functional: add CLI args

[PATCH v3 0/4] tests/functional: add CLI args

[Manos Pitsidianakis]

This series adds extra CLI args for functional tests, useful for
developers that run test files directly.

Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
---
Changes in v3:
- Merged with other patch series
- Made argument values into class attributes, and parse_args into a
  class method (staticmethod) (suggested by Thomas)
- Link to v2: https://lore.kernel.org/qemu-devel/20250717-functional_tests_debug_arg-v2-1-4f0d991e16f7@linaro.org

Changes in v2:
- Store stdout handler in `self` object (thanks Daniel)
- Deduplicate handler removal code (Daniel)
- Amend commit description to mention PYTHONPATH (thanks Alex)
- Link to v1: https://lore.kernel.org/qemu-devel/20250716-functional_tests_debug_arg-v1-1-6a9cd68318bb@linaro.org

---
Manos Pitsidianakis (4):
      tests/functional: add --debug CLI arg
      tests/functional: add --keep-scratch CLI arg
      tests/functional: add --list-tests CLI arg
      tests/functional: add -k TEST_NAME_PATTERN CLI arg

 docs/devel/testing/functional.rst      |   2 +
 tests/functional/qemu_test/testcase.py | 104 ++++++++++++++++++++++++++++++---
 2 files changed, 99 insertions(+), 7 deletions(-)
---
base-commit: 4bc8fb0135ac48cd1c8f0698976b7d73c60b7caa
change-id: 20250716-functional_tests_debug_arg-aa0a5f6b9375

--
γαῖα πυρί μιχθήτω

[PATCH v3 1/4] tests/functional: add --debug CLI arg

[Manos Pitsidianakis]

Add argument parsing to functional tests to improve developer experience
when running individual tests. All logs are printed to stdout
interspersed with TAP output.

Example usage, assuming current build directory with qemu source code in
the parent directory (see docs/devel/testing/functional.rst for details):

  $ export PYTHONPATH=../python:../tests/functional
  $ export QEMU_TEST_QEMU_BINARY="$(pwd)/qemu-system-aarch64"
  $ ./pyvenv/bin/python3 ../tests/functional/test_aarch64_virt.py --help
  usage: test_aarch64_virt [-h] [-d]

  QEMU Functional test

  options:
    -h, --help   show this help message and exit
    -d, --debug  Also print test and console logs on stdout. This will
                 make the TAP output invalid and is meant for debugging
                 only.

Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
---
 docs/devel/testing/functional.rst      |  2 ++
 tests/functional/qemu_test/testcase.py | 54 +++++++++++++++++++++++++++++++---
 2 files changed, 52 insertions(+), 4 deletions(-)

diff --git a/docs/devel/testing/functional.rst b/docs/devel/testing/functional.rst
index 3728bab6c0c4b0cbacd00744bdb5c1462c71d7a8..2725633e09104db3912ec6167bbda652f40aa969 100644
--- a/docs/devel/testing/functional.rst
+++ b/docs/devel/testing/functional.rst
@@ -63,6 +63,8 @@ directory should be your build folder. For example::
   $ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64
   $ pyvenv/bin/python3 ../tests/functional/test_file.py
 
+By default, functional tests redirect informational logs and console output to
+log files. Specify the ``--debug`` flag to also print those to standard output.
 The test framework will automatically purge any scratch files created during
 the tests. If needing to debug a failed test, it is possible to keep these
 files around on disk by setting ``QEMU_TEST_KEEP_SCRATCH=1`` as an env
diff --git a/tests/functional/qemu_test/testcase.py b/tests/functional/qemu_test/testcase.py
index 2a78e735f1604f21efd18e38ee0d586496d6b38c..eedca7f1ad29c9e654cf56535acf9639d679f5c4 100644
--- a/tests/functional/qemu_test/testcase.py
+++ b/tests/functional/qemu_test/testcase.py
@@ -11,6 +11,7 @@
 # This work is licensed under the terms of the GNU GPL, version 2 or
 # later.  See the COPYING file in the top-level directory.
 
+import argparse
 import logging
 import os
 from pathlib import Path
@@ -33,6 +34,28 @@
 
 
 class QemuBaseTest(unittest.TestCase):
+    debug: bool = False
+
+    """
+    Class method that initializes class attributes from given command-line
+    arguments.
+    """
+    @staticmethod
+    def parse_args():
+        test_name = os.path.basename(sys.argv[0])[:-3]
+        parser = argparse.ArgumentParser(
+            prog=test_name, description="QEMU Functional test"
+        )
+        parser.add_argument(
+            "-d",
+            "--debug",
+            action="store_true",
+            help="Also print test and console logs on stdout. This will make "
+            "the TAP output invalid and is meant for debugging only.",
+        )
+        args = parser.parse_args()
+        QemuBaseTest.debug = args.debug
+        return
 
     '''
     @params compressed: filename, Asset, or file-like object to uncompress
@@ -197,6 +220,14 @@ def assets_available(self):
         return True
 
     def setUp(self):
+        self.stdout_handler = None
+        if QemuBaseTest.debug:
+            self.stdout_handler = logging.StreamHandler(sys.stdout)
+            self.stdout_handler.setLevel(logging.DEBUG)
+            formatter = logging.Formatter(
+                "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
+            )
+            self.stdout_handler.setFormatter(formatter)
         self.qemu_bin = os.getenv('QEMU_TEST_QEMU_BINARY')
         self.assertIsNotNone(self.qemu_bin, 'QEMU_TEST_QEMU_BINARY must be set')
         self.arch = self.qemu_bin.split('-')[-1]
@@ -216,12 +247,17 @@ def setUp(self):
             '%(asctime)s - %(levelname)s: %(message)s')
         self._log_fh.setFormatter(fileFormatter)
         self.log.addHandler(self._log_fh)
+        if self.stdout_handler:
+            self.log.addHandler(self.stdout_handler)
 
         # Capture QEMUMachine logging
         self.machinelog = logging.getLogger('qemu.machine')
         self.machinelog.setLevel(logging.DEBUG)
         self.machinelog.addHandler(self._log_fh)
 
+        if self.stdout_handler:
+            self.machinelog.addHandler(self.stdout_handler)
+
         if not self.assets_available():
             self.skipTest('One or more assets is not available')
 
@@ -231,15 +267,19 @@ def tearDown(self):
         if self.socketdir is not None:
             shutil.rmtree(self.socketdir.name)
             self.socketdir = None
-        self.machinelog.removeHandler(self._log_fh)
-        self.log.removeHandler(self._log_fh)
-        self._log_fh.close()
+        for handler in [self._log_fh, self.stdout_handler]:
+            if handler is None:
+                continue
+            self.machinelog.removeHandler(handler)
+            self.log.removeHandler(handler)
+            handler.close()
 
     def main():
         warnings.simplefilter("default")
         os.environ["PYTHONWARNINGS"] = "default"
 
         path = os.path.basename(sys.argv[0])[:-3]
+        QemuBaseTest.parse_args()
 
         cache = os.environ.get("QEMU_TEST_PRECACHE", None)
         if cache is not None:
@@ -297,6 +337,8 @@ def setUp(self):
         fileFormatter = logging.Formatter('%(asctime)s: %(message)s')
         self._console_log_fh.setFormatter(fileFormatter)
         console_log.addHandler(self._console_log_fh)
+        if self.stdout_handler:
+            console_log.addHandler(self.stdout_handler)
 
     def set_machine(self, machinename):
         # TODO: We should use QMP to get the list of available machines
@@ -403,6 +445,10 @@ def set_vm_arg(self, arg, value):
     def tearDown(self):
         for vm in self._vms.values():
             vm.shutdown()
-        logging.getLogger('console').removeHandler(self._console_log_fh)
+        console_log = logging.getLogger("console")
+        console_log.removeHandler(self._console_log_fh)
         self._console_log_fh.close()
+        if self.stdout_handler:
+            console_log.removeHandler(self.stdout_handler)
+            self.stdout_handler.close()
         super().tearDown()

-- 
2.47.2

Re: [PATCH v3 1/4] tests/functional: add --debug CLI arg

[Alex Bennée]

Manos Pitsidianakis <manos.pitsidianakis@linaro.org> writes:

> Add argument parsing to functional tests to improve developer experience
> when running individual tests. All logs are printed to stdout
> interspersed with TAP output.
>
> Example usage, assuming current build directory with qemu source code in
> the parent directory (see docs/devel/testing/functional.rst for details):
>
>   $ export PYTHONPATH=../python:../tests/functional
>   $ export QEMU_TEST_QEMU_BINARY="$(pwd)/qemu-system-aarch64"
>   $ ./pyvenv/bin/python3 ../tests/functional/test_aarch64_virt.py --help
>   usage: test_aarch64_virt [-h] [-d]
>
>   QEMU Functional test
>
>   options:
>     -h, --help   show this help message and exit
>     -d, --debug  Also print test and console logs on stdout. This will
>                  make the TAP output invalid and is meant for debugging
>                  only.
>
> Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>



Tested-by: Alex Bennée <alex.bennee@linaro.org>

<snip>

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

[PATCH v3 2/4] tests/functional: add --keep-scratch CLI arg

[Manos Pitsidianakis]

Add CLI arg to keep scratch files after test execution, equivalent to
setting QEMU_TEST_KEEP_SCRATCH env var.

Suggested-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
---
 tests/functional/qemu_test/testcase.py | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/tests/functional/qemu_test/testcase.py b/tests/functional/qemu_test/testcase.py
index eedca7f1ad29c9e654cf56535acf9639d679f5c4..f7e306cf749e8b24a712b09dfe8673660cbb5085 100644
--- a/tests/functional/qemu_test/testcase.py
+++ b/tests/functional/qemu_test/testcase.py
@@ -35,6 +35,7 @@
 
 class QemuBaseTest(unittest.TestCase):
     debug: bool = False
+    keep_scratch: bool = "QEMU_TEST_KEEP_SCRATCH" in os.environ
 
     """
     Class method that initializes class attributes from given command-line
@@ -53,8 +54,16 @@ def parse_args():
             help="Also print test and console logs on stdout. This will make "
             "the TAP output invalid and is meant for debugging only.",
         )
+        parser.add_argument(
+            "--keep-scratch",
+            action="store_true",
+            help="Do not purge any scratch files created during the tests. "
+            "This is equivalent to setting QEMU_TEST_KEEP_SCRATCH=1 in the "
+            "environment.",
+        )
         args = parser.parse_args()
         QemuBaseTest.debug = args.debug
+        QemuBaseTest.keep_scratch |= args.keep_scratch
         return
 
     '''
@@ -262,8 +271,10 @@ def setUp(self):
             self.skipTest('One or more assets is not available')
 
     def tearDown(self):
-        if "QEMU_TEST_KEEP_SCRATCH" not in os.environ:
+        if not QemuBaseTest.keep_scratch:
             shutil.rmtree(self.workdir)
+        else:
+            self.log.info(f"Kept scratch files in {self.workdir}")
         if self.socketdir is not None:
             shutil.rmtree(self.socketdir.name)
             self.socketdir = None

-- 
2.47.2

[PATCH v3 3/4] tests/functional: add --list-tests CLI arg

[Manos Pitsidianakis]

Add CLI argument to list tests and exit.

Example output (current dir is build directory under root dir):

  $ export PYTHONPATH=../python:../tests/functional
  $ export QEMU_TEST_QEMU_BINARY="$(pwd)/qemu-system-aarch64"
  $ ./pyvenv/bin/python3 ../tests/functional/test_aarch64_virt.py --list-tests
  test_aarch64_virt_gicv2 (test_aarch64_virt.Aarch64VirtMachine.test_aarch64_virt_gicv2)
  test_aarch64_virt_gicv3 (test_aarch64_virt.Aarch64VirtMachine.test_aarch64_virt_gicv3)
  test_alpine_virt_tcg_gic_max (test_aarch64_virt.Aarch64VirtMachine.test_alpine_virt_tcg_gic_max)

Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
---
 tests/functional/qemu_test/testcase.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/tests/functional/qemu_test/testcase.py b/tests/functional/qemu_test/testcase.py
index f7e306cf749e8b24a712b09dfe8673660cbb5085..ab564f873c303bcc28c3bf7bec8c8c4569fae91c 100644
--- a/tests/functional/qemu_test/testcase.py
+++ b/tests/functional/qemu_test/testcase.py
@@ -36,6 +36,7 @@
 class QemuBaseTest(unittest.TestCase):
     debug: bool = False
     keep_scratch: bool = "QEMU_TEST_KEEP_SCRATCH" in os.environ
+    list_tests: bool = False
 
     """
     Class method that initializes class attributes from given command-line
@@ -61,9 +62,15 @@ def parse_args():
             "This is equivalent to setting QEMU_TEST_KEEP_SCRATCH=1 in the "
             "environment.",
         )
+        parser.add_argument(
+            "--list-tests",
+            action="store_true",
+            help="List all tests that would be executed and exit.",
+        )
         args = parser.parse_args()
         QemuBaseTest.debug = args.debug
         QemuBaseTest.keep_scratch |= args.keep_scratch
+        QemuBaseTest.list_tests = args.list_tests
         return
 
     '''
@@ -292,6 +299,13 @@ def main():
         path = os.path.basename(sys.argv[0])[:-3]
         QemuBaseTest.parse_args()
 
+        if QemuBaseTest.list_tests:
+            loader = unittest.TestLoader()
+            for test_suite in loader.loadTestsFromName(path):
+                for test in test_suite:
+                    print(test)
+            return
+
         cache = os.environ.get("QEMU_TEST_PRECACHE", None)
         if cache is not None:
             Asset.precache_suites(path, cache)

-- 
2.47.2

[PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg

[Manos Pitsidianakis]

Add a CLI argument that takes fnmatch(3)-style patterns as value and can
be specified many times. Only tests that match the pattern will be
executed. This argument is passed to unittest.main which takes the same
argument.

Acked-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
---
 tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/tests/functional/qemu_test/testcase.py b/tests/functional/qemu_test/testcase.py
index ab564f873c303bcc28c3bf7bec8c8c4569fae91c..b045d82caa79d9d161fb868b0b0748ad7de453d9 100644
--- a/tests/functional/qemu_test/testcase.py
+++ b/tests/functional/qemu_test/testcase.py
@@ -16,6 +16,7 @@
 import os
 from pathlib import Path
 import pycotap
+import itertools
 import shutil
 from subprocess import run
 import sys
@@ -37,6 +38,7 @@ class QemuBaseTest(unittest.TestCase):
     debug: bool = False
     keep_scratch: bool = "QEMU_TEST_KEEP_SCRATCH" in os.environ
     list_tests: bool = False
+    test_name_patterns: list[str] = []
 
     """
     Class method that initializes class attributes from given command-line
@@ -67,10 +69,19 @@ def parse_args():
             action="store_true",
             help="List all tests that would be executed and exit.",
         )
+        parser.add_argument(
+            "-k",
+            dest="test_name_patterns",
+            action="append",
+            type=str,
+            help="Only run tests which match the given substring. "
+            "This argument is passed to unittest.main verbatim.",
+        )
         args = parser.parse_args()
         QemuBaseTest.debug = args.debug
         QemuBaseTest.keep_scratch |= args.keep_scratch
         QemuBaseTest.list_tests = args.list_tests
+        QemuBaseTest.test_name_patterns = args.test_name_patterns
         return
 
     '''
@@ -313,8 +324,16 @@ def main():
 
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
-        res = unittest.main(module = None, testRunner = tr, exit = False,
-                            argv=["__dummy__", path])
+        argv = ["__dummy__", path] + (
+            list(
+                itertools.chain.from_iterable(
+                    ["-k", x] for x in QemuBaseTest.test_name_patterns
+                )
+            )
+            if QemuBaseTest.test_name_patterns
+            else []
+        )
+        res = unittest.main(module=None, testRunner=tr, exit=False, argv=argv)
         for (test, message) in res.result.errors + res.result.failures:
 
             if hasattr(test, "log_filename"):

-- 
2.47.2

Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg

[Alex Bennée]

Manos Pitsidianakis <manos.pitsidianakis@linaro.org> writes:

> Add a CLI argument that takes fnmatch(3)-style patterns as value and can
> be specified many times. Only tests that match the pattern will be
> executed. This argument is passed to unittest.main which takes the same
> argument.
>
> Acked-by: Thomas Huth <thuth@redhat.com>
> Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>

Tested-by: Alex Bennée <alex.bennee@linaro.org>

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro

Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg

[Daniel P. Berrangé]

On Fri, Jul 25, 2025 at 12:41:25PM +0300, Manos Pitsidianakis wrote:
> Add a CLI argument that takes fnmatch(3)-style patterns as value and can
> be specified many times. Only tests that match the pattern will be
> executed. This argument is passed to unittest.main which takes the same
> argument.
> 
> Acked-by: Thomas Huth <thuth@redhat.com>
> Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
> ---
>  tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++--
>  1 file changed, 21 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/functional/qemu_test/testcase.py b/tests/functional/qemu_test/testcase.py
> index ab564f873c303bcc28c3bf7bec8c8c4569fae91c..b045d82caa79d9d161fb868b0b0748ad7de453d9 100644
> --- a/tests/functional/qemu_test/testcase.py
> +++ b/tests/functional/qemu_test/testcase.py
> @@ -16,6 +16,7 @@
>  import os
>  from pathlib import Path
>  import pycotap
> +import itertools
>  import shutil
>  from subprocess import run
>  import sys
> @@ -37,6 +38,7 @@ class QemuBaseTest(unittest.TestCase):
>      debug: bool = False
>      keep_scratch: bool = "QEMU_TEST_KEEP_SCRATCH" in os.environ
>      list_tests: bool = False
> +    test_name_patterns: list[str] = []
>  
>      """
>      Class method that initializes class attributes from given command-line
> @@ -67,10 +69,19 @@ def parse_args():
>              action="store_true",
>              help="List all tests that would be executed and exit.",
>          )
> +        parser.add_argument(
> +            "-k",
> +            dest="test_name_patterns",
> +            action="append",
> +            type=str,
> +            help="Only run tests which match the given substring. "
> +            "This argument is passed to unittest.main verbatim.",
> +        )
>          args = parser.parse_args()
>          QemuBaseTest.debug = args.debug
>          QemuBaseTest.keep_scratch |= args.keep_scratch
>          QemuBaseTest.list_tests = args.list_tests
> +        QemuBaseTest.test_name_patterns = args.test_name_patterns
>          return
>  
>      '''
> @@ -313,8 +324,16 @@ def main():
>  
>          tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
>                                     test_output_log = pycotap.LogMode.LogToError)
> -        res = unittest.main(module = None, testRunner = tr, exit = False,
> -                            argv=["__dummy__", path])
> +        argv = ["__dummy__", path] + (
> +            list(
> +                itertools.chain.from_iterable(
> +                    ["-k", x] for x in QemuBaseTest.test_name_patterns
> +                )
> +            )
> +            if QemuBaseTest.test_name_patterns
> +            else []
> +        )
> +        res = unittest.main(module=None, testRunner=tr, exit=False, argv=argv)

unittest.main() supports a whole bunch of CLI args beyond '-k', but none
of them are accessible as we're not forwarding the sys.argv that we have
received. eg we're missing

$ git diff
diff --git a/tests/functional/qemu_test/testcase.py b/tests/functional/qemu_test/testcase.py
index 2a78e735f1..5caf7b13fe 100644
--- a/tests/functional/qemu_test/testcase.py
+++ b/tests/functional/qemu_test/testcase.py
@@ -249,7 +249,7 @@ def main():
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
         res = unittest.main(module = None, testRunner = tr, exit = False,
-                            argv=["__dummy__", path])
+                            argv=[sys.argv[0], path] + sys.argv[1:])
         for (test, message) in res.result.errors + res.result.failures:
 
             if hasattr(test, "log_filename"):

which would unlock

$ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64  PYTHONPATH=`pwd`/python ./tests/functional/test_version.py  -h
usage: test_version.py [-h] [-v] [-q] [--locals] [--durations N] [-f] [-c] [-b] [-k TESTNAMEPATTERNS] [tests ...]

positional arguments:
  tests                a list of any number of test modules, classes and test methods.

options:
  -h, --help           show this help message and exit
  -v, --verbose        Verbose output
  -q, --quiet          Quiet output
  --locals             Show local variables in tracebacks
  --durations N        Show the N slowest test cases (N=0 for all)
  -f, --failfast       Stop on first fail or error
  -c, --catch          Catch Ctrl-C and display results so far
  -b, --buffer         Buffer stdout and stderr during tests
  -k TESTNAMEPATTERNS  Only run tests which match the given substring



One of the goals with the new functional test system was that we stop trying
to (re-)invent a custom test runner harness, as was the case with Avocado,
in favour of relying on the pre-existing python infrastructure to the
greatest extent possible.

Seeing this, and all the other CLI arg handling added in this series, makes
me fairly uncomfortable, as it is effectively inventing a custom test runner
once again which is exactly what we wanted to get away from.

At the same time, there are some pieces in this series that do things that
unittest.main() can't do on its own.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg

[Daniel P. Berrangé]

On Fri, Jul 25, 2025 at 02:25:46PM +0100, Daniel P. Berrangé wrote:
> On Fri, Jul 25, 2025 at 12:41:25PM +0300, Manos Pitsidianakis wrote:
> > Add a CLI argument that takes fnmatch(3)-style patterns as value and can
> > be specified many times. Only tests that match the pattern will be
> > executed. This argument is passed to unittest.main which takes the same
> > argument.
> > 
> > Acked-by: Thomas Huth <thuth@redhat.com>
> > Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
> > ---
> >  tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++--
> >  1 file changed, 21 insertions(+), 2 deletions(-)



> One of the goals with the new functional test system was that we stop trying
> to (re-)invent a custom test runner harness, as was the case with Avocado,
> in favour of relying on the pre-existing python infrastructure to the
> greatest extent possible.
> 
> Seeing this, and all the other CLI arg handling added in this series, makes
> me fairly uncomfortable, as it is effectively inventing a custom test runner
> once again which is exactly what we wanted to get away from.
> 
> At the same time, there are some pieces in this series that do things that
> unittest.main() can't do on its own.

So considering the broader picture, we already have a load of tunables
on the test execution that we control exclusively via envirnoment
variables, as that gives us independance of the test runner, which
owns sys.argv processing.

So in terms of this series, IMHO, we should just add support for
QEMU_TEST_DEBUG=1 and QEMU_TEST_KEEP_SCRATCH=1 as two new tunables,
and not touch sys.argv at all.


With that, the only thing we're missing is a way to enumerate the
test cases, but IMHO that's the job of the test runner and thus
out of scope for QEMU to solve.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg

[Thomas Huth]

On 25/07/2025 16.48, Daniel P. Berrangé wrote:
> On Fri, Jul 25, 2025 at 02:25:46PM +0100, Daniel P. Berrangé wrote:
>> On Fri, Jul 25, 2025 at 12:41:25PM +0300, Manos Pitsidianakis wrote:
>>> Add a CLI argument that takes fnmatch(3)-style patterns as value and can
>>> be specified many times. Only tests that match the pattern will be
>>> executed. This argument is passed to unittest.main which takes the same
>>> argument.
>>>
>>> Acked-by: Thomas Huth <thuth@redhat.com>
>>> Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
>>> ---
>>>   tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++--
>>>   1 file changed, 21 insertions(+), 2 deletions(-)
> 
> 
> 
>> One of the goals with the new functional test system was that we stop trying
>> to (re-)invent a custom test runner harness, as was the case with Avocado,
>> in favour of relying on the pre-existing python infrastructure to the
>> greatest extent possible.
>>
>> Seeing this, and all the other CLI arg handling added in this series, makes
>> me fairly uncomfortable, as it is effectively inventing a custom test runner
>> once again which is exactly what we wanted to get away from.
>>
>> At the same time, there are some pieces in this series that do things that
>> unittest.main() can't do on its own.
> 
> So considering the broader picture, we already have a load of tunables
> on the test execution that we control exclusively via envirnoment
> variables, as that gives us independance of the test runner, which
> owns sys.argv processing.
> 
> So in terms of this series, IMHO, we should just add support for
> QEMU_TEST_DEBUG=1 and QEMU_TEST_KEEP_SCRATCH=1 as two new tunables,
> and not touch sys.argv at all.

I basically agree, but I wonder if we could maybe also have both? First 
parse the sys.argv for our parameters, then pass the unknown ones to 
unittest.main() ?

> With that, the only thing we're missing is a way to enumerate the
> test cases, but IMHO that's the job of the test runner and thus
> out of scope for QEMU to solve.

With the patch applied that passes the argv to unittest.main(), there also 
seems to be a "discover" mode:

  ~/devel/qemu/tests/functional/test_vnc.py -h | grep usage:
usage: test_vnc.py [-h] [-v] [-q] [--locals] [--durations N] [-f] [-c] [-b]
usage: test_vnc.py discover [-h] [-v] [-q] [--locals] [--durations N] [-f]

... however, when I try to run it, it does not work as expected (I get a 
"ModuleNotFoundError: No module named 'discover'" error).

  Thomas

Re: [PATCH v3 4/4] tests/functional: add -k TEST_NAME_PATTERN CLI arg

[Daniel P. Berrangé]

On Sat, Jul 26, 2025 at 08:54:51AM +0200, Thomas Huth wrote:
> On 25/07/2025 16.48, Daniel P. Berrangé wrote:
> > On Fri, Jul 25, 2025 at 02:25:46PM +0100, Daniel P. Berrangé wrote:
> > > On Fri, Jul 25, 2025 at 12:41:25PM +0300, Manos Pitsidianakis wrote:
> > > > Add a CLI argument that takes fnmatch(3)-style patterns as value and can
> > > > be specified many times. Only tests that match the pattern will be
> > > > executed. This argument is passed to unittest.main which takes the same
> > > > argument.
> > > > 
> > > > Acked-by: Thomas Huth <thuth@redhat.com>
> > > > Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
> > > > ---
> > > >   tests/functional/qemu_test/testcase.py | 23 +++++++++++++++++++++--
> > > >   1 file changed, 21 insertions(+), 2 deletions(-)
> > 
> > 
> > 
> > > One of the goals with the new functional test system was that we stop trying
> > > to (re-)invent a custom test runner harness, as was the case with Avocado,
> > > in favour of relying on the pre-existing python infrastructure to the
> > > greatest extent possible.
> > > 
> > > Seeing this, and all the other CLI arg handling added in this series, makes
> > > me fairly uncomfortable, as it is effectively inventing a custom test runner
> > > once again which is exactly what we wanted to get away from.
> > > 
> > > At the same time, there are some pieces in this series that do things that
> > > unittest.main() can't do on its own.
> > 
> > So considering the broader picture, we already have a load of tunables
> > on the test execution that we control exclusively via envirnoment
> > variables, as that gives us independance of the test runner, which
> > owns sys.argv processing.
> > 
> > So in terms of this series, IMHO, we should just add support for
> > QEMU_TEST_DEBUG=1 and QEMU_TEST_KEEP_SCRATCH=1 as two new tunables,
> > and not touch sys.argv at all.
> 
> I basically agree, but I wonder if we could maybe also have both? First
> parse the sys.argv for our parameters, then pass the unknown ones to
> unittest.main() ?

Any argv that we process are subject to clashing with that defined by
the test harness and it is desirable to avoid making assumptions about
what test harness args exist.

To avoid that they would all have to be long options named with a
fixed prefix --qemu-XXXX. Also that sould be something applied to
all env vars, not just the couple of params in this series.

> > With that, the only thing we're missing is a way to enumerate the
> > test cases, but IMHO that's the job of the test runner and thus
> > out of scope for QEMU to solve.
> 
> With the patch applied that passes the argv to unittest.main(), there also
> seems to be a "discover" mode:
> 
>  ~/devel/qemu/tests/functional/test_vnc.py -h | grep usage:
> usage: test_vnc.py [-h] [-v] [-q] [--locals] [--durations N] [-f] [-c] [-b]
> usage: test_vnc.py discover [-h] [-v] [-q] [--locals] [--durations N] [-f]
> 
> ... however, when I try to run it, it does not work as expected (I get a
> "ModuleNotFoundError: No module named 'discover'" error).

Hmm, not sure that the 'discover' option makes sense when we're asking to
run a single test file.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|