[PATCH v1 0/5] migration & CI: Add a CI job for migration compat testing

[PATCH v1 0/5] migration & CI: Add a CI job for migration compat testing

[Fabiano Rosas]

Here's the second half of adding a migration compatibility test to CI.

We've already added support for running the full set of migration
tests with two QEMU binaries since commit 5050ad2a380
("tests/qtest/migration: Support more than one QEMU binary"), now
what's left is adding it to the CI.

I included patches that solve the problem of testing older QEMUs with
new test code. The old QEMU might lack features, bug fixes, etc. that
the tests expect to be present. After this series we can specify a
minimal QEMU version that a specific test supports.

I have NOT solved the issue of a QEMU command line change breaking
older QEMUs because that is outside of the migration realm an we're
about to release 8.2 anyway, so let's kick that problem to the
future. Hopefully it won't happen again soon.

Let me know what you think.

CI run: https://gitlab.com/farosas/qemu/-/pipelines/1098975378

Fabiano Rosas (5):
  tests/qtest: Add a helper to query the QEMU version
  tests/qtest/migration: Add infrastructure to skip tests on older QEMUs
  tests/qtest/migration: Adapt tests to use older QEMUs
  ci: Add a migration compatibility test job
  [NOT FOR MERGE] compat hack until 8.2 is out

 .gitlab-ci.d/buildtest.yml      | 43 +++++++++++++++++++++++
 tests/qtest/libqtest.c          | 25 ++++++++++++-
 tests/qtest/libqtest.h          | 10 ++++++
 tests/qtest/migration-helpers.c | 11 ++++++
 tests/qtest/migration-helpers.h |  1 +
 tests/qtest/migration-test.c    | 62 +++++++++++++++++++++++++++++----
 6 files changed, 144 insertions(+), 8 deletions(-)

-- 
2.35.3

[PATCH v1 1/5] tests/qtest: Add a helper to query the QEMU version

[Fabiano Rosas]

Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
 tests/qtest/libqtest.c | 24 ++++++++++++++++++++++++
 tests/qtest/libqtest.h | 10 ++++++++++
 2 files changed, 34 insertions(+)

diff --git a/tests/qtest/libqtest.c b/tests/qtest/libqtest.c
index f33a210861..7cee68a834 100644
--- a/tests/qtest/libqtest.c
+++ b/tests/qtest/libqtest.c
@@ -337,6 +337,30 @@ void qtest_remove_abrt_handler(void *data)
     }
 }
 
+void qtest_query_version(QTestState *who, int *major, int *minor, int *micro)
+{
+    QDict *rsp, *version;
+
+    rsp = qtest_qmp_assert_success_ref(who, "{ 'execute': 'query-version' }");
+    g_assert(rsp);
+
+    version = qdict_get_qdict(rsp, "qemu");
+
+    if (major) {
+        *major = qdict_get_int(version, "major");
+    }
+
+    if (minor) {
+        *minor = qdict_get_int(version, "minor");
+    }
+
+    if (micro) {
+        *micro = qdict_get_int(version, "micro");
+    }
+
+    qobject_unref(rsp);
+}
+
 static const char *qtest_qemu_binary(const char *var)
 {
     const char *qemu_bin;
diff --git a/tests/qtest/libqtest.h b/tests/qtest/libqtest.h
index 6e3d3525bf..db7a780d26 100644
--- a/tests/qtest/libqtest.h
+++ b/tests/qtest/libqtest.h
@@ -1085,4 +1085,14 @@ bool have_qemu_img(void);
  */
 bool mkimg(const char *file, const char *fmt, unsigned size_mb);
 
+/**
+ * qtest_query_version:
+ * @who: QTestState instance to operate on
+ * @major: Pointer to where to store the major version number
+ * @minor: Pointer to where to store the minor version number
+ * @micro: Pointer to where to store the micro version number
+ *
+ */
+void qtest_query_version(QTestState *who, int *major, int *minor, int *micro);
+
 #endif
-- 
2.35.3

[PATCH v1 2/5] tests/qtest/migration: Add infrastructure to skip tests on older QEMUs

[Fabiano Rosas]

We can run the migration tests with two different QEMU binaries to
test migration compatibility between QEMU versions. This means we'll
be running the tests with an older QEMU in either source or
destination.

We need to avoid trying to test functionality that is unknown to the
older QEMU. This could mean new features, bug fixes, error message
changes, QEMU command line changes, migration API changes, etc.

Add a 'since' argument to the tests that inform when the functionality
that is being test has been added to QEMU so we can skip the test on
older versions.

Also add a version comparison function so we can adapt test code
depending on the QEMU binary version being used.

Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
 tests/qtest/migration-helpers.c | 11 +++++++++++
 tests/qtest/migration-helpers.h |  1 +
 tests/qtest/migration-test.c    | 28 ++++++++++++++++++++++++++++
 3 files changed, 40 insertions(+)

diff --git a/tests/qtest/migration-helpers.c b/tests/qtest/migration-helpers.c
index 24fb7b3525..d21f5cd8c0 100644
--- a/tests/qtest/migration-helpers.c
+++ b/tests/qtest/migration-helpers.c
@@ -292,3 +292,14 @@ char *resolve_machine_version(const char *alias, const char *var1,
 
     return find_common_machine_version(machine_name, var1, var2);
 }
+
+int migration_vercmp(QTestState *who, const char *tgt_version)
+{
+    int major, minor, micro;
+    g_autofree char *version = NULL;
+
+    qtest_query_version(who, &major, &minor, &micro);
+    version = g_strdup_printf("%d.%d.%d", major, minor, micro);
+
+    return strcmp(version, tgt_version);
+}
diff --git a/tests/qtest/migration-helpers.h b/tests/qtest/migration-helpers.h
index e31dc85cc7..7b4f8e851e 100644
--- a/tests/qtest/migration-helpers.h
+++ b/tests/qtest/migration-helpers.h
@@ -47,4 +47,5 @@ char *find_common_machine_version(const char *mtype, const char *var1,
                                   const char *var2);
 char *resolve_machine_version(const char *alias, const char *var1,
                               const char *var2);
+int migration_vercmp(QTestState *who, const char *tgt_version);
 #endif /* MIGRATION_HELPERS_H */
diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
index 0fbaa6a90f..b3ce288a73 100644
--- a/tests/qtest/migration-test.c
+++ b/tests/qtest/migration-test.c
@@ -637,6 +637,12 @@ typedef struct {
     bool use_dirty_ring;
     const char *opts_source;
     const char *opts_target;
+    /*
+     * If a test checks against new functionality that might not be
+     * present in older QEMUs this needs to be set so we can skip
+     * running it when doing compatibility testing.
+     */
+    const char *since;
 } MigrateStart;
 
 /*
@@ -850,6 +856,17 @@ static int test_migrate_start(QTestState **from, QTestState **to,
         qtest_qmp_set_event_callback(*from,
                                      migrate_watch_for_stop,
                                      &got_src_stop);
+
+        if (args->since && migration_vercmp(*from, args->since) < 0) {
+            g_autofree char *msg = NULL;
+
+            msg = g_strdup_printf("Test requires at least QEMU version %s",
+                                  args->since);
+            g_test_skip(msg);
+            qtest_quit(*from);
+
+            return -1;
+        }
     }
 
     cmd_target = g_strdup_printf("-accel kvm%s -accel tcg "
@@ -872,6 +889,17 @@ static int test_migrate_start(QTestState **from, QTestState **to,
                                  migrate_watch_for_resume,
                                  &got_dst_resume);
 
+    if (args->since && migration_vercmp(*to, args->since) < 0) {
+        g_autofree char *msg = NULL;
+
+        msg = g_strdup_printf("Test requires at least QEMU version %s",
+                              args->since);
+        g_test_skip(msg);
+        qtest_quit(*to);
+
+        return -1;
+    }
+
     /*
      * Remove shmem file immediately to avoid memory leak in test failed case.
      * It's valid because QEMU has already opened this file
-- 
2.35.3

[PATCH v1 3/5] tests/qtest/migration: Adapt tests to use older QEMUs

[Fabiano Rosas]

Add the 'since' annotations to recently added tests and adapt the
postcopy test to use the older "uri" API when needed.

Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
 tests/qtest/migration-test.c | 34 +++++++++++++++++++++++++++-------
 1 file changed, 27 insertions(+), 7 deletions(-)

diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c
index b3ce288a73..f16e71d821 100644
--- a/tests/qtest/migration-test.c
+++ b/tests/qtest/migration-test.c
@@ -1337,14 +1337,21 @@ static int migrate_postcopy_prepare(QTestState **from_ptr,
     migrate_ensure_non_converge(from);
 
     migrate_prepare_for_dirty_mem(from);
-    qtest_qmp_assert_success(to, "{ 'execute': 'migrate-incoming',"
-                             "  'arguments': { "
-                             "      'channels': [ { 'channel-type': 'main',"
-                             "      'addr': { 'transport': 'socket',"
-                             "                'type': 'inet',"
-                             "                'host': '127.0.0.1',"
-                             "                'port': '0' } } ] } }");
 
+    /* New syntax was introduced in 8.2 */
+    if (migration_vercmp(to, "8.2") < 0) {
+        qtest_qmp_assert_success(to, "{ 'execute': 'migrate-incoming',"
+                                 "  'arguments': { "
+                                 "      'uri': 'tcp:127.0.0.1:0' } }");
+    } else {
+        qtest_qmp_assert_success(to, "{ 'execute': 'migrate-incoming',"
+                                 "  'arguments': { "
+                                 "      'channels': [ { 'channel-type': 'main',"
+                                 "      'addr': { 'transport': 'socket',"
+                                 "                'type': 'inet',"
+                                 "                'host': '127.0.0.1',"
+                                 "                'port': '0' } } ] } }");
+    }
     /* Wait for the first serial output from the source */
     wait_for_serial("src_serial");
 
@@ -1602,6 +1609,9 @@ static void test_postcopy_recovery_double_fail(void)
 {
     MigrateCommon args = {
         .postcopy_recovery_test_fail = true,
+        .start = {
+            .since = "8.2",
+        },
     };
 
     test_postcopy_recovery_common(&args);
@@ -1664,6 +1674,7 @@ static void test_analyze_script(void)
 {
     MigrateStart args = {
         .opts_source = "-uuid 11111111-1111-1111-1111-111111111111",
+        .since = "8.2",
     };
     QTestState *from, *to;
     g_autofree char *uri = NULL;
@@ -2089,6 +2100,9 @@ static void test_precopy_file(void)
     MigrateCommon args = {
         .connect_uri = uri,
         .listen_uri = "defer",
+        .start = {
+            .since = "8.2"
+        },
     };
 
     test_file_common(&args, true);
@@ -2133,6 +2147,9 @@ static void test_precopy_file_offset(void)
         .connect_uri = uri,
         .listen_uri = "defer",
         .finish_hook = file_offset_finish_hook,
+        .start = {
+            .since = "8.2"
+        },
     };
 
     test_file_common(&args, false);
@@ -2147,6 +2164,9 @@ static void test_precopy_file_offset_bad(void)
         .connect_uri = uri,
         .listen_uri = "defer",
         .result = MIG_TEST_QMP_ERROR,
+        .start = {
+            .since = "8.2"
+        },
     };
 
     test_file_common(&args, false);
-- 
2.35.3

[PATCH v1 4/5] ci: Add a migration compatibility test job

[Fabiano Rosas]

The migration tests have support for being passed two QEMU binaries to
test migration compatibility.

Add a CI job that builds the lastest release of QEMU and another job
that uses that version plus an already present build of the current
version and run the migration tests with the two, both as source and
destination. I.e.:

 old QEMU (n-1) -> current QEMU (development tree)
 current QEMU (development tree) -> old QEMU (n-1)

The purpose of this CI job is to ensure the code we're about to merge
will not cause a migration compatibility problem when migrating the
next release (which will contain that code) to/from the previous
release.

I'm leaving the jobs as manual for now because using an older QEMU in
tests could hit bugs that were already fixed in the current
development tree and we need to handle those case-by-case.

Note: for user forks, the version tags need to be pushed to gitlab
otherwise it won't be able to checkout a different version.

Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
 .gitlab-ci.d/buildtest.yml | 43 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 43 insertions(+)

diff --git a/.gitlab-ci.d/buildtest.yml b/.gitlab-ci.d/buildtest.yml
index 91663946de..9a24eed666 100644
--- a/.gitlab-ci.d/buildtest.yml
+++ b/.gitlab-ci.d/buildtest.yml
@@ -167,6 +167,49 @@ build-system-centos:
       x86_64-softmmu rx-softmmu sh4-softmmu nios2-softmmu
     MAKE_CHECK_ARGS: check-build
 
+build-previous-qemu:
+  extends: .native_build_job_template
+  artifacts:
+    when: on_success
+    expire_in: 2 days
+    paths:
+      - build-previous
+    exclude:
+      - build-previous/**/*.p
+      - build-previous/**/*.a.p
+      - build-previous/**/*.fa.p
+      - build-previous/**/*.c.o
+      - build-previous/**/*.c.o.d
+      - build-previous/**/*.fa
+  needs:
+    job: amd64-opensuse-leap-container
+  variables:
+    QEMU_JOB_OPTIONAL: 1
+    IMAGE: opensuse-leap
+    TARGETS: x86_64-softmmu
+  before_script:
+    - export QEMU_PREV_VERSION="$(sed 's/\([0-9.]*\)\.[0-9]*/v\1.0/' VERSION)"
+    - git checkout $QEMU_PREV_VERSION
+  after_script:
+    - mv build build-previous
+
+check-migration-compat:
+  extends: .common_test_job_template
+  needs:
+    - job: build-previous-qemu
+    - job: build-system-opensuse
+  allow_failure: true
+  variables:
+    QEMU_JOB_OPTIONAL: 1
+    IMAGE: opensuse-leap
+    MAKE_CHECK_ARGS: check-build
+  script:
+    - cd build
+    - QTEST_QEMU_BINARY_SRC=../build-previous/qemu-system-x86_64
+          QTEST_QEMU_BINARY=./qemu-system-x86_64 ./tests/qtest/migration-test
+    - QTEST_QEMU_BINARY_DST=../build-previous/qemu-system-x86_64
+          QTEST_QEMU_BINARY=./qemu-system-x86_64 ./tests/qtest/migration-test
+
 check-system-centos:
   extends: .native_test_job_template
   needs:
-- 
2.35.3

[PATCH v1 5/5] [NOT FOR MERGE] compat hack until 8.2 is out

[Fabiano Rosas]

In the 8.2 development cycle we added a new '-audio' command line
option to QEMU which breaks the migration compatibility tests because
older QEMUs don't know the option.

Since we're just testing n vs. n-1 migration, it's easier if we just
wait until 8.2 is released and it will become our n-1 and the problem
will go away on its own.

Signed-off-by: Fabiano Rosas <farosas@suse.de>
---
 tests/qtest/libqtest.c | 1 -
 1 file changed, 1 deletion(-)

diff --git a/tests/qtest/libqtest.c b/tests/qtest/libqtest.c
index 7cee68a834..031316390e 100644
--- a/tests/qtest/libqtest.c
+++ b/tests/qtest/libqtest.c
@@ -497,7 +497,6 @@ static QTestState *qtest_init_internal(const char *qemu_bin,
                          "-chardev socket,path=%s,id=char0 "
                          "-mon chardev=char0,mode=control "
                          "-display none "
-                         "-audio none "
                          "%s"
                          " -accel qtest",
                          socket_path,
-- 
2.35.3

Re: [PATCH v1 2/5] tests/qtest/migration: Add infrastructure to skip tests on older QEMUs

[Fabiano Rosas]

Fabiano Rosas <farosas@suse.de> writes:

> We can run the migration tests with two different QEMU binaries to
> test migration compatibility between QEMU versions. This means we'll
> be running the tests with an older QEMU in either source or
> destination.
>
> We need to avoid trying to test functionality that is unknown to the
> older QEMU. This could mean new features, bug fixes, error message
> changes, QEMU command line changes, migration API changes, etc.
>
> Add a 'since' argument to the tests that inform when the functionality
> that is being test has been added to QEMU so we can skip the test on
> older versions.
>
> Also add a version comparison function so we can adapt test code
> depending on the QEMU binary version being used.
>
> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> ---
>  tests/qtest/migration-helpers.c | 11 +++++++++++
>  tests/qtest/migration-helpers.h |  1 +
>  tests/qtest/migration-test.c    | 28 ++++++++++++++++++++++++++++
>  3 files changed, 40 insertions(+)
>
> diff --git a/tests/qtest/migration-helpers.c b/tests/qtest/migration-helpers.c
> index 24fb7b3525..d21f5cd8c0 100644
> --- a/tests/qtest/migration-helpers.c
> +++ b/tests/qtest/migration-helpers.c
> @@ -292,3 +292,14 @@ char *resolve_machine_version(const char *alias, const char *var1,
>  
>      return find_common_machine_version(machine_name, var1, var2);
>  }
> +
> +int migration_vercmp(QTestState *who, const char *tgt_version)
> +{
> +    int major, minor, micro;
> +    g_autofree char *version = NULL;
> +
> +    qtest_query_version(who, &major, &minor, &micro);
> +    version = g_strdup_printf("%d.%d.%d", major, minor, micro);

I just noticed this is not right. I need to increment the minor when
there's a micro to account for the versions in between releases. The
whole point of this series is to test a X.Y.0 release vs. a X.Y.Z
development branch.

> +
> +    return strcmp(version, tgt_version);
> +}

Re: [PATCH v1 1/5] tests/qtest: Add a helper to query the QEMU version

[Thomas Huth]

On 07/12/2023 16.58, Fabiano Rosas wrote:
> Signed-off-by: Fabiano Rosas <farosas@suse.de>
> ---
>   tests/qtest/libqtest.c | 24 ++++++++++++++++++++++++
>   tests/qtest/libqtest.h | 10 ++++++++++
>   2 files changed, 34 insertions(+)
> 
> diff --git a/tests/qtest/libqtest.c b/tests/qtest/libqtest.c
> index f33a210861..7cee68a834 100644
> --- a/tests/qtest/libqtest.c
> +++ b/tests/qtest/libqtest.c
> @@ -337,6 +337,30 @@ void qtest_remove_abrt_handler(void *data)
>       }
>   }
>   
> +void qtest_query_version(QTestState *who, int *major, int *minor, int *micro)

I'd prefer "qts" instead of "who" ... but that's bikeshedding, so:

Reviewed-by: Thomas Huth <thuth@redhat.com>

Re: [PATCH v1 2/5] tests/qtest/migration: Add infrastructure to skip tests on older QEMUs

[Thomas Huth]

On 08/12/2023 16.02, Fabiano Rosas wrote:
> Fabiano Rosas <farosas@suse.de> writes:
> 
>> We can run the migration tests with two different QEMU binaries to
>> test migration compatibility between QEMU versions. This means we'll
>> be running the tests with an older QEMU in either source or
>> destination.
>>
>> We need to avoid trying to test functionality that is unknown to the
>> older QEMU. This could mean new features, bug fixes, error message
>> changes, QEMU command line changes, migration API changes, etc.
>>
>> Add a 'since' argument to the tests that inform when the functionality
>> that is being test has been added to QEMU so we can skip the test on
>> older versions.
>>
>> Also add a version comparison function so we can adapt test code
>> depending on the QEMU binary version being used.
>>
>> Signed-off-by: Fabiano Rosas <farosas@suse.de>
>> ---
>>   tests/qtest/migration-helpers.c | 11 +++++++++++
>>   tests/qtest/migration-helpers.h |  1 +
>>   tests/qtest/migration-test.c    | 28 ++++++++++++++++++++++++++++
>>   3 files changed, 40 insertions(+)
>>
>> diff --git a/tests/qtest/migration-helpers.c b/tests/qtest/migration-helpers.c
>> index 24fb7b3525..d21f5cd8c0 100644
>> --- a/tests/qtest/migration-helpers.c
>> +++ b/tests/qtest/migration-helpers.c
>> @@ -292,3 +292,14 @@ char *resolve_machine_version(const char *alias, const char *var1,
>>   
>>       return find_common_machine_version(machine_name, var1, var2);
>>   }
>> +
>> +int migration_vercmp(QTestState *who, const char *tgt_version)
>> +{
>> +    int major, minor, micro;
>> +    g_autofree char *version = NULL;
>> +
>> +    qtest_query_version(who, &major, &minor, &micro);
>> +    version = g_strdup_printf("%d.%d.%d", major, minor, micro);
> 
> I just noticed this is not right. I need to increment the minor when
> there's a micro to account for the versions in between releases. The
> whole point of this series is to test a X.Y.0 release vs. a X.Y.Z
> development branch.

Also this looks like it cannot deal with two-digit minor versions ... we 
still have machine types for QEMU 2.12.0 around ... do we care here?

  Thomas

Re: [PATCH v1 1/5] tests/qtest: Add a helper to query the QEMU version

[Fabiano Rosas]

Thomas Huth <thuth@redhat.com> writes:

> On 07/12/2023 16.58, Fabiano Rosas wrote:
>> Signed-off-by: Fabiano Rosas <farosas@suse.de>
>> ---
>>   tests/qtest/libqtest.c | 24 ++++++++++++++++++++++++
>>   tests/qtest/libqtest.h | 10 ++++++++++
>>   2 files changed, 34 insertions(+)
>> 
>> diff --git a/tests/qtest/libqtest.c b/tests/qtest/libqtest.c
>> index f33a210861..7cee68a834 100644
>> --- a/tests/qtest/libqtest.c
>> +++ b/tests/qtest/libqtest.c
>> @@ -337,6 +337,30 @@ void qtest_remove_abrt_handler(void *data)
>>       }
>>   }
>>   
>> +void qtest_query_version(QTestState *who, int *major, int *minor, int *micro)
>
> I'd prefer "qts" instead of "who" ... but that's bikeshedding, so:

I'll change it. I need to send another version of this series anyway.

Re: [PATCH v1 2/5] tests/qtest/migration: Add infrastructure to skip tests on older QEMUs

[Fabiano Rosas]

Thomas Huth <thuth@redhat.com> writes:

> On 08/12/2023 16.02, Fabiano Rosas wrote:
>> Fabiano Rosas <farosas@suse.de> writes:
>> 
>>> We can run the migration tests with two different QEMU binaries to
>>> test migration compatibility between QEMU versions. This means we'll
>>> be running the tests with an older QEMU in either source or
>>> destination.
>>>
>>> We need to avoid trying to test functionality that is unknown to the
>>> older QEMU. This could mean new features, bug fixes, error message
>>> changes, QEMU command line changes, migration API changes, etc.
>>>
>>> Add a 'since' argument to the tests that inform when the functionality
>>> that is being test has been added to QEMU so we can skip the test on
>>> older versions.
>>>
>>> Also add a version comparison function so we can adapt test code
>>> depending on the QEMU binary version being used.
>>>
>>> Signed-off-by: Fabiano Rosas <farosas@suse.de>
>>> ---
>>>   tests/qtest/migration-helpers.c | 11 +++++++++++
>>>   tests/qtest/migration-helpers.h |  1 +
>>>   tests/qtest/migration-test.c    | 28 ++++++++++++++++++++++++++++
>>>   3 files changed, 40 insertions(+)
>>>
>>> diff --git a/tests/qtest/migration-helpers.c b/tests/qtest/migration-helpers.c
>>> index 24fb7b3525..d21f5cd8c0 100644
>>> --- a/tests/qtest/migration-helpers.c
>>> +++ b/tests/qtest/migration-helpers.c
>>> @@ -292,3 +292,14 @@ char *resolve_machine_version(const char *alias, const char *var1,
>>>   
>>>       return find_common_machine_version(machine_name, var1, var2);
>>>   }
>>> +
>>> +int migration_vercmp(QTestState *who, const char *tgt_version)
>>> +{
>>> +    int major, minor, micro;
>>> +    g_autofree char *version = NULL;
>>> +
>>> +    qtest_query_version(who, &major, &minor, &micro);
>>> +    version = g_strdup_printf("%d.%d.%d", major, minor, micro);
>> 
>> I just noticed this is not right. I need to increment the minor when
>> there's a micro to account for the versions in between releases. The
>> whole point of this series is to test a X.Y.0 release vs. a X.Y.Z
>> development branch.
>
> Also this looks like it cannot deal with two-digit minor versions ... we 
> still have machine types for QEMU 2.12.0 around ... do we care here?
>

I particularly don't. I suspect any issues this series would have caught
would have already been spotted in the wild. This is all about catching
compatibility issues during the development cycle, so not very useful
for older QEMUs and distro versions.