Re: [PATCH v3 4/4] [NOT FOR MERGE] tests/qtest/migration: Adapt tests to use older QEMUs

[Fabiano Rosas <farosas@suse.de>]

Peter Xu <peterx@redhat.com> writes:

> On Tue, Jan 09, 2024 at 11:46:32AM -0300, Fabiano Rosas wrote:
>> Hm, it would be better to avoid the extra maintenance task at the start
>> of every release, no? It also blocks us from doing n-2 even
>> experimentally.
>
> See my other reply, on whether we can use "n-1" for migration-test.  If
> that can work for us, then IIUC we can avoid either "since:" or any
> relevant flag, neither do we need to unmask tests after each releases.  All
> old tests should always "just work" with a new qemu binary.

Hmm.. There are some assumptions here:

1) New code will always be compatible with old tests. E.g. some
   patchseries changed code and changed a test to match the new
   code. Then we'd need a flag like 'since' anyway to mark that the new
   QEMU cannot be used with the old test.

   (if new QEMU is not compatible with old tests without any good
   reason, then that's just a regression I think)

2) There would not be issues when fixing bugs/refactoring
   tests. E.g. old tests had a bug that is now fixed, but since we're
   not using the new tests, the bug is always there until next
   release. This could block the entire test suite, specially with
   concurrency bugs which can start triggering due to changes in timing.

3) New code that can only be reached via new tests cannot cause
   regressions. E.g. new code is added but is kept under a machine
   property or migration capability. That code will only show the
   regression after the new test enables that cap/property. At that
   point it's too late because it was already released.

In general I like the simplicity of your approach, but it would be
annoying to change this series only to find out we still need some sort
of flag later. Even worse, #3 would miss the point of this kind of
testing entirely.

#1 could be mitigated by a "no changes to tests rule". We'd start
requiring that new tests be written and an existing test is never
altered. For #2 and #3 I don't have a solution.