Re: [PATCH v3 3/4] ci: Add a migration compatibility test job

[Fabiano Rosas <farosas@suse.de>]

Peter Xu <peterx@redhat.com> writes:

> On Fri, Jan 05, 2024 at 03:04:48PM -0300, Fabiano Rosas wrote:
>> The migration tests have support for being passed two QEMU binaries to
>> test migration compatibility.
>> 
>> Add a CI job that builds the lastest release of QEMU and another job
>> that uses that version plus an already present build of the current
>> version and run the migration tests with the two, both as source and
>> destination. I.e.:
>> 
>>  old QEMU (n-1) -> current QEMU (development tree)
>>  current QEMU (development tree) -> old QEMU (n-1)
>> 
>> The purpose of this CI job is to ensure the code we're about to merge
>> will not cause a migration compatibility problem when migrating the
>> next release (which will contain that code) to/from the previous
>> release.
>> 
>> I'm leaving the jobs as manual for now because using an older QEMU in
>> tests could hit bugs that were already fixed in the current
>> development tree and we need to handle those case-by-case.
>
> Can we opt-out those broken tests using either your "since:" thing or
> anything similar?

If it's something migration related, then yes. But there might be other
types of breakages that have nothing to do with migration. Our tests are
not resilent enough (nor they should) to detect when QEMU aborted for
other reasons. Think about the -audio issue: the old QEMU would just say
"there's no -audio option, abort" and that's a test failure of course.

> I hope we can start to run something by default in the CI in 9.0 to cover
> n-1 -> n, even if starting with a subset of tests.  Is it possible?

We could maybe have it enabled with "allow_failure" set. The important
thing here is that we don't want to get reports of "flaky test". These
tests are kind of flaky by definition, there's no way to backport a fix
to the older QEMU, so there's always the chance that this test will be
broken for a whole release cycle. We should act fast in adding the
"since" annotation or other workaround, but that depends on our
availability and the type of bug that we hit.