Re: [RFC PATCH-for-5.2] gitlab-ci: Do not automatically run Avocado integration tests anymore

["Philippe Mathieu-Daudé" <philmd@redhat.com>]

On 11/27/20 6:47 PM, Thomas Huth wrote:
> On 27/11/2020 18.41, Philippe Mathieu-Daudé wrote:
>> We lately realized that the Avocado framework was not designed
>> to be regularly run on CI environments. Therefore, as of 5.2
>> we deprecate the gitlab-ci jobs using Avocado. To not disrupt
>> current users, it is possible to keep the current behavior by
>> setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable
>> (see [*]).
>> From now on, using these jobs (or adding new tests to them)
>> is strongly discouraged.
>>
>> Tests based on Avocado will be ported to new job schemes during
>> the next releases, with better documentation and templates.
> 
> Why should we disable the jobs by default as long as there is no replacement
> available yet?

Why keep it enabled if it is failing randomly, if images hardcoded
in tests are being removed from public servers, etc...?


They are not disabled, they are still runnable manually or setting
QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE...

We realized by default Avocado runs all tests on the CI jobs,
triggering failures and complains. Developer stopped to contribute/
review integration tests because of that. We want developers be
able to contribute tests to the repository fearlessly.

If we don't change anything, we'll keep having CI failures due
to Avocado design issues (artifacts removed from remote servers,
etc...).

I haven't seen any attempt on this list to improve the current
fragile situation, but better suggestions are certainly welcome.

Thanks,

Phil.