Re: [PATCH v3 4/4] [NOT FOR MERGE] tests/qtest/migration: Adapt tests to use older QEMUs

[Fabiano Rosas <farosas@suse.de>]

Peter Xu <peterx@redhat.com> writes:

> On Mon, Jan 08, 2024 at 12:37:46PM -0300, Fabiano Rosas wrote:
>> Peter Xu <peterx@redhat.com> writes:
>> 
>> > On Fri, Jan 05, 2024 at 03:04:49PM -0300, Fabiano Rosas wrote:
>> >> [This patch is not necessary anymore after 8.2 has been released]
>> >> 
>> >> Add the 'since' annotations to recently added tests and adapt the
>> >> postcopy test to use the older "uri" API when needed.
>> >> 
>> >> Signed-off-by: Fabiano Rosas <farosas@suse.de>
>> >
>> > You marked this as not-for-merge.  Would something like this still be
>> > useful in the future?  IIUC it's a matter of whether we'd still want to
>> > test those old binaries.
>> >
>> 
>> Technically yes, but I fail to see what benefit testing old binaries
>> would bring us. I'm thinking maybe it could be useful for bisecting
>> compatibility issues, but I can't think of a scenario where we'd like to
>> change the older QEMU instead of the newer.
>> 
>> I'm of course open to suggestions if you or anyone else has an use case
>> that you'd like to keep viable.
>> 
>> So far, my idea is that once a new QEMU is released, all the "since:"
>> annotations become obsolete. We could even remove them. This series is
>> just infrastructure to make our life easier if a change is ever
>> introduced that is incompatible with the n-1 QEMU. IMO we cannot have
>> compatibility testing if a random change might break a test and make it
>> more difficult to run the remaining tests. So we'd use 'since' or the
>> vercmp function to skip/adapt the offending tests until the next QEMU is
>> released.
>> 
>> I'm basing myself on this loosely worded support statement from our
>> docs:
>> 
>>   "In general QEMU tries to maintain forward migration compatibility
>>   (i.e. migrating from QEMU n->n+1) and there are users who benefit from
>>   backward compatibility as well."
>
> I think we could still have users migrating from e.g. 8.0 -> 9.0 as long as
> with the same machine type, especially when upgrading upper level stack
> (e.g. an openstack cluster upgrade), where IIUC can jump a few qemu major
> versions.  That does sound like a common use case, and I suspect the doc
> was only taking one example on why compatibility needs to be maintained,
> rather than emphasizing "+1 only".

Oh, I would expect people to be migrating in all sorts of ways. But we
need to think in terms of what upstream QEMU supports so we can guide
the development. And hopefully have a test for everything we actually
support and everyone that touches migration code having the same view on
this.

I can barely think about n->n+1 to be honest, that's why I was writing
this compatibility test even before Juan asked for it.

You raise a good point about a cloud provider or distro jumping major
versions. That's a tricky situation. Because then their support
statement would potentially cover something that's completely different
from what we're testing upstream.

> However then the question is whether those old binaries needs to be
> convered.
>
> Then I noticed that taking all these "since: XXX" and cmdline changes along
> with migration-test may be yet another burden even if we want to cover old
> binaries for whatever reason.  I am now more convinced myself that we
> should try to get rid of as much burden as we can for migration, because we
> already have enough, and it's not ideal to keep growing that unnecessarily.
>
> One good thing with CI in this case (I still don't have enough knowledge on
> CI, so I am hoping some CI people can review that patch, though) is that if
> we can always guarantee n-1 -> n works for the test cases we enabled, it
> most probably means when n boosts again to n+1, we keep making sure n ->
> n+1 works perfectly, then n-1 -> n+1 should not fail either, considering
> that we're testing the stream protocol matching each other.  There might be
> outliers (especially if not described with VMSDs) but should be corner
> cases.

I agree that the transitivity should be preserved. If we could override
the QEMU_PREV_VERSION variable in the CI script, that would be an easy
way of running a sanity check every once in a while.

> So I tend to agree with you on that we drop this patch, keep it simple
> until we're much more clear what we can get from that.
>
> But then if so - do we need "since" at all to be expressed in versions?

I agree that we don't need "since" semantics.

> Basically we keep qtest always be valid only on the latest qemu binary as
> before (which actually works the same as Linux v.s. kselftests, which makes
> sense), there's one exception now with "n-1" due to the CI we plan to add.
> Dropping this patch means we don't yet plan to support n-2.  Then maybe
> instead of a "since" we only need a boolean showing "whether one test needs
> to be covered by a cross-binary test"?  Then we set it in incompatible
> binaries (skip all cross-binary tests directly, rather than relying on any
> qemu versions, no compare needed), and can also drop that when a new
> release starts.

Hm, it would be better to avoid the extra maintenance task at the start
of every release, no? It also blocks us from doing n-2 even
experimentally.