[RFC PATCH 0/2] tests/qtest: Run migration subtests via meson

[RFC PATCH 0/2] tests/qtest: Run migration subtests via meson

[Fabiano Rosas]

We've had some issues with the migration-test failing in CI and giving
no clue as to which migration sub-test has failed. I think it would be
an improvement to have each subtest listed individually.

With this we can see which migration test has failed, can set timeouts
individually and can run 'make check-migration' to run only the
migration tests.

It does however increase the amount of lines emitted with make check,
so let me know if this is an issue.

I had to include some hacking due to meson, but I think it is simple
enough and worth the benefits.

CI run: https://gitlab.com/farosas/qemu/-/pipelines/919324510

Fabiano Rosas (2):
  tests/qtest: Add a script to gather migration tests list
  tests/qtest: Pass migration tests individually to meson

 .gitlab-ci.d/windows.yml                |  2 +-
 tests/qtest/gen_migration_tests_list.py | 12 ++++++++
 tests/qtest/meson.build                 | 40 +++++++++++++++++++------
 3 files changed, 44 insertions(+), 10 deletions(-)
 create mode 100644 tests/qtest/gen_migration_tests_list.py

-- 
2.35.3

[RFC PATCH 1/2] tests/qtest: Add a script to gather migration tests list

[Fabiano Rosas]

This is a workaround for limitations with meson.

We'd like to have each migration (sub)test registered with meson's
test() function, but since they are all inside the single
migration-test.c there's no way for meson to see the tests. We need an
external way to generate the list of tests and pass it to meson.

We cannot call 'migration-test -l' because we'd need to build the
migration-test binary first and while that is possible, there's no
subsequent way to get the generated list into a meson variable for
consumption. None of meson's routines support retrieving an arbitrary
list of strings from a command at build (vs. configure) time.

We also cannot use generators to have 'migration-test' write to a file
because that would happen at build time and meson does not support
reading from a file in the build directory.

So the only approach left is to either have the list of tests
committed into the source code or to grep the source code for the
list. Committing the file would be harder to maintain and the test
registration in migration-test.c is pretty static, so this patch uses
the grep approach. But using python because it is more portable.

Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
 tests/qtest/gen_migration_tests_list.py | 12 ++++++++++++
 1 file changed, 12 insertions(+)
 create mode 100644 tests/qtest/gen_migration_tests_list.py

diff --git a/tests/qtest/gen_migration_tests_list.py b/tests/qtest/gen_migration_tests_list.py
new file mode 100644
index 0000000000..5127f976fa
--- /dev/null
+++ b/tests/qtest/gen_migration_tests_list.py
@@ -0,0 +1,12 @@
+#!/usr/bin/env python3
+
+import re
+import sys
+
+file_path = sys.argv[1]
+
+with open(file_path, 'r') as f:
+    for line in f.readlines():
+        match = re.search('\"(/migration/.*)\"', line)
+        if match:
+            print(match.groups()[0])
-- 
2.35.3

[RFC PATCH 2/2] tests/qtest: Pass migration tests individually to meson

[Fabiano Rosas]

Having glib tests (qtest) all defined in a single test file makes it
hard to know which test has failed when running CI. Create a new
'migration' test suite and move the migration tests individually to
meson.

For now, use the global migration-test timeout value, but we could set
a per-subtest timeout in the future.

Sample output:

$ ../configure --target-list=x86_64-softmmu,aarch64-softmmu ...
$ make check-migration
...
36/469 qemu:migration / /x86_64/migration/precopy/unix/plain                       OK    34.25s   1 subtests passed
37/469 qemu:migration / /x86_64/migration/multifd/tcp/tls/x509/default-host        OK     7.33s   1 subtests passed
39/469 qemu:migration / /x86_64/migration/multifd/tcp/tls/x509/allow-anon-client   OK     7.32s   1 subtests passed
40/469 qemu:migration / /aarch64/migration/postcopy/compress/plain                 SKIP   0.04s
41/469 qemu:migration / /aarch64/migration/postcopy/recovery/compress/plain        SKIP   0.04s
42/469 qemu:migration / /aarch64/migration/bad_dest                                OK     0.65s   1 subtests passed

Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
 .gitlab-ci.d/windows.yml |  2 +-
 tests/qtest/meson.build  | 40 +++++++++++++++++++++++++++++++---------
 2 files changed, 32 insertions(+), 10 deletions(-)

diff --git a/.gitlab-ci.d/windows.yml b/.gitlab-ci.d/windows.yml
index f889a468b5..04bc74a20e 100644
--- a/.gitlab-ci.d/windows.yml
+++ b/.gitlab-ci.d/windows.yml
@@ -84,7 +84,7 @@ msys2-64bit:
   - ..\msys64\usr\bin\bash -lc 'make'
   # qTests don't run successfully with "--without-default-devices",
   # so let's exclude the qtests from CI for now.
-  - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS=\"--no-suite qtest\" || { cat meson-logs/testlog.txt; exit 1; } ;'
+  - ..\msys64\usr\bin\bash -lc 'make check MTESTARGS=\"--no-suite qtest migration\" || { cat meson-logs/testlog.txt; exit 1; } ;'
 
 msys2-32bit:
   extends: .shared_msys2_builder
diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index 5fa6833ad7..43c140921c 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -369,14 +369,36 @@ foreach dir : target_dirs
         test: executable(test, src, dependencies: deps)
       }
     endif
-    test('qtest-@0@/@1@'.format(target_base, test),
-         qtest_executables[test],
-         depends: [test_deps, qtest_emulator, emulator_modules],
-         env: qtest_env,
-         args: ['--tap', '-k'],
-         protocol: 'tap',
-         timeout: slow_qtests.get(test, 30),
-         priority: slow_qtests.get(test, 30),
-         suite: ['qtest', 'qtest-' + target_base])
+
+    migtest = 'migration-test'
+    if test == migtest
+      migtests = run_command(python, files('gen_migration_tests_list.py'),
+                             meson.current_source_dir() / 'migration-test.c',
+                             check: true)
+
+      foreach item : migtests.stdout().strip().split('\n')
+        testname = '/@0@@1@'.format(target_base, item)
+
+        test(testname,
+             qtest_executables['migration-test'],
+             depends: [test_deps, qtest_emulator, emulator_modules],
+             env: qtest_env,
+             args: ['-k', '-p', testname],
+             protocol: 'tap',
+             timeout: slow_qtests.get(migtest),
+             priority: slow_qtests.get(migtest),
+             suite: ['migration'])
+      endforeach
+    else
+      test('qtest-@0@/@1@'.format(target_base, test),
+           qtest_executables[test],
+           depends: [test_deps, qtest_emulator, emulator_modules],
+           env: qtest_env,
+           args: ['--tap', '-k'],
+           protocol: 'tap',
+           timeout: slow_qtests.get(test, 30),
+           priority: slow_qtests.get(test, 30),
+           suite: ['qtest', 'qtest-' + target_base])
+      endif
   endforeach
 endforeach
-- 
2.35.3

Re: [RFC PATCH 0/2] tests/qtest: Run migration subtests via meson

[Daniel P. Berrangé]

On Mon, Jul 03, 2023 at 11:46:58AM -0300, Fabiano Rosas wrote:
> We've had some issues with the migration-test failing in CI and giving
> no clue as to which migration sub-test has failed. I think it would be
> an improvement to have each subtest listed individually.

If the migration-test (or any test) actually fails, the meson test
log should show us exactly which test case failed already. We had
some jobs which didn't publish the log as an artifact, but I think
we finally have those all fixed.

If the migration-test hangs, however, then we get stuck until the
GitLab CI job timeout and get no useful info. For this scenario
we need to enable timeouts at the meson level, which are currently
all disabled. I've got a series here to address this:

  https://lists.gnu.org/archive/html/qemu-devel/2023-06/msg00305.html

It needs 

> 
> With this we can see which migration test has failed, can set timeouts
> individually and can run 'make check-migration' to run only the
> migration tests.

IMHO focusing just on migration-test is undesirable, as while this
has been one of the least reliable ones, other tests do also exhibit
hard to debug failures for much the same reasons. IOW, we need
improved debugging across the board rather than specialcased solutions.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|