[PATCH 0/3] tests/functional: add more CLI args

[PATCH 0/3] tests/functional: add more CLI args

[Manos Pitsidianakis]

This series adds extra CLI args for functional tests, useful for
developers that run test files directly.

It depends on a previous patch that adds a --debug CLI arg, and is
encoded as a b4 change-id dependency, so it should be fetched and
applied automatically when using b4.

Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
---
Manos Pitsidianakis (3):
      tests/functional: add --keep-scratch CLI arg
      tests/functional: add --list-tests CLI arg
      tests/functional: add -k TEST_NAME_PATTERN CLI arg

 tests/functional/qemu_test/testcase.py | 45 +++++++++++++++++++++++++++++-----
 1 file changed, 39 insertions(+), 6 deletions(-)
---
base-commit: 3656e761bcdd207b7759cdcd608212d2a6f9c12d
change-id: 20250718-functional_tests_args-12cdb5e56d84
prerequisite-change-id: 20250716-functional_tests_debug_arg-aa0a5f6b9375:v2
prerequisite-patch-id: 4ccc8f39ffb382d31c8e6450c43a5f8d177af044

--
γαῖα πυρί μιχθήτω

[PATCH 1/3] tests/functional: add --keep-scratch CLI arg

[Manos Pitsidianakis]

Add CLI arg to keep scratch files after test execution, equivalent to
setting QEMU_TEST_KEEP_SCRATCH env var.

Suggested-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
---
 tests/functional/qemu_test/testcase.py | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/tests/functional/qemu_test/testcase.py b/tests/functional/qemu_test/testcase.py
index 3ecaaeffd4df2945fb4c44b4ddef6911527099b9..24f0b452964012532530a92018b2c4367b39fb77 100644
--- a/tests/functional/qemu_test/testcase.py
+++ b/tests/functional/qemu_test/testcase.py
@@ -43,6 +43,13 @@ def parse_args(test_name: str) -> argparse.Namespace:
         help="Also print test and console logs on stdout. This will make the"
         " TAP output invalid and is meant for debugging only.",
     )
+    parser.add_argument(
+        "--keep-scratch",
+        action="store_true",
+        help="Do not purge any scratch files created during the tests. "
+        "This is equivalent to setting QEMU_TEST_KEEP_SCRATCH=1 in the "
+        "environment.",
+    )
     return parser.parse_args()
 
 
@@ -214,6 +221,10 @@ def setUp(self):
         path = os.path.basename(sys.argv[0])[:-3]
         args = parse_args(path)
         self.stdout_handler = None
+        self.keep_scratch = (
+                "QEMU_TEST_KEEP_SCRATCH" in os.environ
+                or args.keep_scratch
+        )
         if args.debug:
             self.stdout_handler = logging.StreamHandler(sys.stdout)
             self.stdout_handler.setLevel(logging.DEBUG)
@@ -255,8 +266,10 @@ def setUp(self):
             self.skipTest('One or more assets is not available')
 
     def tearDown(self):
-        if "QEMU_TEST_KEEP_SCRATCH" not in os.environ:
+        if not self.keep_scratch:
             shutil.rmtree(self.workdir)
+        else:
+            self.log.info(f"Kept scratch files in {self.workdir}")
         if self.socketdir is not None:
             shutil.rmtree(self.socketdir.name)
             self.socketdir = None

-- 
2.47.2

[PATCH 2/3] tests/functional: add --list-tests CLI arg

[Manos Pitsidianakis]

Add CLI argument to list tests and exit.

Example output (current dir is build directory under root dir):

  $ export PYTHONPATH=../python:../tests/functional
  $ export QEMU_TEST_QEMU_BINARY="$(pwd)/qemu-system-aarch64"
  $ ./pyvenv/bin/python3 ../tests/functional/test_aarch64_virt.py --list-tests
  test_aarch64_virt_gicv2 (test_aarch64_virt.Aarch64VirtMachine.test_aarch64_virt_gicv2)
  test_aarch64_virt_gicv3 (test_aarch64_virt.Aarch64VirtMachine.test_aarch64_virt_gicv3)
  test_alpine_virt_tcg_gic_max (test_aarch64_virt.Aarch64VirtMachine.test_alpine_virt_tcg_gic_max)

Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
---
 tests/functional/qemu_test/testcase.py | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

diff --git a/tests/functional/qemu_test/testcase.py b/tests/functional/qemu_test/testcase.py
index 24f0b452964012532530a92018b2c4367b39fb77..4455fcc38016c14db04019bbd64eaae9b47363c1 100644
--- a/tests/functional/qemu_test/testcase.py
+++ b/tests/functional/qemu_test/testcase.py
@@ -50,6 +50,11 @@ def parse_args(test_name: str) -> argparse.Namespace:
         "This is equivalent to setting QEMU_TEST_KEEP_SCRATCH=1 in the "
         "environment.",
     )
+    parser.add_argument(
+        "--list-tests",
+        action="store_true",
+        help="List all tests that would be executed and exit.",
+    )
     return parser.parse_args()
 
 
@@ -281,10 +286,13 @@ def tearDown(self):
 
     def main():
         path = os.path.basename(sys.argv[0])[:-3]
-        # If argparse receives --help or an unknown argument, it will raise a
-        # SystemExit which will get caught by the test runner. Parse the
-        # arguments here too to handle that case.
-        parse_args(path)
+        args = parse_args(path)
+        if args.list_tests:
+            loader = unittest.TestLoader()
+            for test_suite in loader.loadTestsFromName(path):
+                for test in test_suite:
+                    print(test)
+            return
 
         cache = os.environ.get("QEMU_TEST_PRECACHE", None)
         if cache is not None:

-- 
2.47.2

[PATCH 3/3] tests/functional: add -k TEST_NAME_PATTERN CLI arg

[Manos Pitsidianakis]

Add a CLI argument that takes fnmatch(3)-style patterns as value and can
be specified many times. Only tests that match the pattern will be
executed. This argument is passed to unittest.main which takes the same
argument.

Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
---
 tests/functional/qemu_test/testcase.py | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/tests/functional/qemu_test/testcase.py b/tests/functional/qemu_test/testcase.py
index 4455fcc38016c14db04019bbd64eaae9b47363c1..b7a660fd729afe39ff8cf7a0be97c2fc2f2f573f 100644
--- a/tests/functional/qemu_test/testcase.py
+++ b/tests/functional/qemu_test/testcase.py
@@ -55,6 +55,14 @@ def parse_args(test_name: str) -> argparse.Namespace:
         action="store_true",
         help="List all tests that would be executed and exit.",
     )
+    parser.add_argument(
+        "-k",
+        dest="test_name_patterns",
+        action="append",
+        type=str,
+        help="Only run tests which match the given substring. "
+        "This argument is passed to unittest.main verbatim.",
+    )
     return parser.parse_args()
 
 
@@ -301,8 +309,12 @@ def main():
 
         tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                    test_output_log = pycotap.LogMode.LogToError)
+        argv = ["__dummy__", path] + (
+            (["-k"] + args.test_name_patterns)
+            if args.test_name_patterns else []
+        )
         res = unittest.main(module = None, testRunner = tr, exit = False,
-                            argv=["__dummy__", path])
+                            argv=argv)
         for (test, message) in res.result.errors + res.result.failures:
 
             if hasattr(test, "log_filename"):

-- 
2.47.2

Re: [PATCH 3/3] tests/functional: add -k TEST_NAME_PATTERN CLI arg

[Paolo Bonzini]

On 7/18/25 11:12, Manos Pitsidianakis wrote:
> Add a CLI argument that takes fnmatch(3)-style patterns as value and can
> be specified many times. Only tests that match the pattern will be
> executed. This argument is passed to unittest.main which takes the same
> argument.
> 
> Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
> ---
>   tests/functional/qemu_test/testcase.py | 14 +++++++++++++-
>   1 file changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/functional/qemu_test/testcase.py b/tests/functional/qemu_test/testcase.py
> index 4455fcc38016c14db04019bbd64eaae9b47363c1..b7a660fd729afe39ff8cf7a0be97c2fc2f2f573f 100644
> --- a/tests/functional/qemu_test/testcase.py
> +++ b/tests/functional/qemu_test/testcase.py
> @@ -55,6 +55,14 @@ def parse_args(test_name: str) -> argparse.Namespace:
>           action="store_true",
>           help="List all tests that would be executed and exit.",
>       )
> +    parser.add_argument(
> +        "-k",
> +        dest="test_name_patterns",
> +        action="append",
> +        type=str,
> +        help="Only run tests which match the given substring. "
> +        "This argument is passed to unittest.main verbatim.",
> +    )
>       return parser.parse_args()
>   
>   
> @@ -301,8 +309,12 @@ def main():
>   
>           tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
>                                      test_output_log = pycotap.LogMode.LogToError)
> +        argv = ["__dummy__", path] + (
> +            (["-k"] + args.test_name_patterns)
> +            if args.test_name_patterns else []
> +        )

This does not work for >1 occurrences of -k. Maybe something like:

argv = list(itertools.chain(
     ["__dummy__", "path"],
     *[["-k", x] for x in args.test_name_patterns]))

Paolo

>           res = unittest.main(module = None, testRunner = tr, exit = False,
> -                            argv=["__dummy__", path])
> +                            argv=argv)
>           for (test, message) in res.result.errors + res.result.failures:
>   
>               if hasattr(test, "log_filename"):
> 

Re: [PATCH 3/3] tests/functional: add -k TEST_NAME_PATTERN CLI arg

[Manos Pitsidianakis]

On Fri, Jul 18, 2025 at 12:37 PM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 7/18/25 11:12, Manos Pitsidianakis wrote:
> > Add a CLI argument that takes fnmatch(3)-style patterns as value and can
> > be specified many times. Only tests that match the pattern will be
> > executed. This argument is passed to unittest.main which takes the same
> > argument.
> >
> > Signed-off-by: Manos Pitsidianakis <manos.pitsidianakis@linaro.org>
> > ---
> >   tests/functional/qemu_test/testcase.py | 14 +++++++++++++-
> >   1 file changed, 13 insertions(+), 1 deletion(-)
> >
> > diff --git a/tests/functional/qemu_test/testcase.py b/tests/functional/qemu_test/testcase.py
> > index 4455fcc38016c14db04019bbd64eaae9b47363c1..b7a660fd729afe39ff8cf7a0be97c2fc2f2f573f 100644
> > --- a/tests/functional/qemu_test/testcase.py
> > +++ b/tests/functional/qemu_test/testcase.py
> > @@ -55,6 +55,14 @@ def parse_args(test_name: str) -> argparse.Namespace:
> >           action="store_true",
> >           help="List all tests that would be executed and exit.",
> >       )
> > +    parser.add_argument(
> > +        "-k",
> > +        dest="test_name_patterns",
> > +        action="append",
> > +        type=str,
> > +        help="Only run tests which match the given substring. "
> > +        "This argument is passed to unittest.main verbatim.",
> > +    )
> >       return parser.parse_args()
> >
> >
> > @@ -301,8 +309,12 @@ def main():
> >
> >           tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
> >                                      test_output_log = pycotap.LogMode.LogToError)
> > +        argv = ["__dummy__", path] + (
> > +            (["-k"] + args.test_name_patterns)
> > +            if args.test_name_patterns else []
> > +        )
>
> This does not work for >1 occurrences of -k. Maybe something like:
>
> argv = list(itertools.chain(
>      ["__dummy__", "path"],
>      *[["-k", x] for x in args.test_name_patterns]))


Oops, good catch!

Thanks

> Paolo
>
> >           res = unittest.main(module = None, testRunner = tr, exit = False,
> > -                            argv=["__dummy__", path])
> > +                            argv=argv)
> >           for (test, message) in res.result.errors + res.result.failures:
> >
> >               if hasattr(test, "log_filename"):
> >
>