Re: [PATCH v2 04/29] tests/acceptance: Use 'version-min' tag to verify QEMU binary version

["Philippe Mathieu-Daudé" <philmd@redhat.com>]

Cc'ing Markus

On 1/30/20 6:08 PM, Cornelia Huck wrote:
> On Thu, 30 Jan 2020 14:28:19 +0100
> Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> 
>> On 1/30/20 1:26 PM, Cornelia Huck wrote:
>>> On Wed, 29 Jan 2020 22:23:20 +0100
>>> Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
>>>    
>>>> Introduce the 'version-min' tag.
>>>> Tests can set it to the minimum version of QEMU they require.
>>>> If QEMU is older, the tests will be cancelled (skipped):
>>>>
>>>>     $ python -m avocado --show=app run tests/acceptance/x86_cpu_model_versions.py
>>>>      (01/11) X86CPUModelAliases.test_4_0_alias_compatibility: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
>>>>      (02/11) X86CPUModelAliases.test_4_1_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
>>>>      (03/11) X86CPUModelAliases.test_none_alias: CANCEL: Test expects version '4.1' but QEMU binary is '3.1.1' (0.10 s)
>>>>      ...
>>>>
>>>> Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>>>> ---
>>>>    tests/acceptance/avocado_qemu/__init__.py | 18 ++++++++++++++++++
>>>>    1 file changed, 18 insertions(+)
>>>
>>> Hm... this might cause headaches if someone tries to backport tests on
>>> distro versions; but we probably won't be able to test for the feature
>>> in many cases anyway?
>>>    
>>
>> Are you saying a distro version might not match upstream version when a
>> feature is introduced? Isn't it the point of versioning? If you are
>> thinking of some case you point me at it so I can improve this patch?
> 
> Something like the following:
> 
> - a feature together with a test is introduced in QEMU version n
> - a distro is using m (where m < n), but wants the new feature and
>    backports it
> 
> If that distro now wants to include the acceptance test as well (to run
> in their internal testing), they need to adapt the version check as
> well, or the test simply won't run. Depending on how this is reported,
> it might not be easily noticed. If the test depends on another feature
> (but the feature does not), it might get a bit more complicated.
> 
> It's probably not a big problem, but I wanted to point it out.

OK now I understand better.

I wrote that patch having QAPI versioned features in mind, which are 
versioned/documented.
For example:

   ##
   # @qom-list-types:
   #
   ...
   #
   # Since: 1.1

Now it is true that we add a new feature to a device, we do not document 
that in QAPI, and I don't see how we can test it via QMP.