Re: [PATCH v4 1/6] tests/requirements.txt: bump up avocado-framework version to 101.0

[Thomas Huth <thuth@redhat.com>]

On 21/04/2023 06.19, Cleber Rosa wrote:
> 
> On 4/18/23 03:10, Thomas Huth wrote:
>> On 17/04/2023 15.43, Alex Bennée wrote:
>>> From: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>>
>>> Avocado version 101.0 has a fix to re-compute the checksum
>>> of an asset file if the algorithm used in the *-CHECKSUM
>>> file isn't the same as the one being passed to it by the
>>> avocado user (i.e. the avocado_qemu python module).
>>> In the earlier avocado versions this fix wasn't there due
>>> to which if the checksum wouldn't match the earlier
>>> checksum (calculated by a different algorithm), the avocado
>>> code would start downloading a fresh image from the internet
>>> URL thus making the test-cases take longer to execute.
>>>
>>> Bump up the avocado-framework version to 101.0.
>>>
>>> Signed-off-by: Kautuk Consul <kconsul@linux.vnet.ibm.com>
>>> Tested-by: Hariharan T S <hariharan.ts@linux.vnet.ibm.com>
>>> Message-Id: <20230327115030.3418323-2-kconsul@linux.vnet.ibm.com>
>>> Message-Id: <20230330101141.30199-10-alex.bennee@linaro.org>
>>>
>>> ---
>>> v2
>>>    - limit --max-parallel-tasks $(JOBS_OPTION:-j%=%)
>>> ---
>>>   tests/Makefile.include | 18 +++++++++++-------
>>>   tests/requirements.txt |  2 +-
>>>   2 files changed, 12 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index 9422ddaece..a4de0ad5a2 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -138,14 +138,18 @@ get-vm-image-fedora-31-%: check-venv
>>>   # download all vm images, according to defined targets
>>>   get-vm-images: check-venv $(patsubst %,get-vm-image-fedora-31-%, 
>>> $(FEDORA_31_DOWNLOAD))
>>>   +JOBS_OPTION=$(lastword -j1 $(filter-out -j, $(filter -j%,$(MAKEFLAGS))))
>>> +
>>>   check-avocado: check-venv $(TESTS_RESULTS_DIR) get-vm-images
>>> -    $(call quiet-command, \
>>> -            $(TESTS_PYTHON) -m avocado \
>>> -            --show=$(AVOCADO_SHOW) run 
>>> --job-results-dir=$(TESTS_RESULTS_DIR) \
>>> -            $(if $(AVOCADO_TAGS),, --filter-by-tags-include-empty \
>>> -            --filter-by-tags-include-empty-key) \
>>> -            $(AVOCADO_CMDLINE_TAGS) \
>>> -            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS), \
>>> +    $(call quiet-command,                             \
>>> +            $(TESTS_PYTHON) -m avocado \
>>> +            --show=$(AVOCADO_SHOW) run 
>>> --job-results-dir=$(TESTS_RESULTS_DIR)     \
>>> +            $(if $(AVOCADO_TAGS),,                         \
>>> +            --filter-by-tags-include-empty                 \
>>> +            --filter-by-tags-include-empty-key)             \
>>> +        --max-parallel-tasks $(JOBS_OPTION:-j%=%)             \
>>> +            $(AVOCADO_CMDLINE_TAGS)                         \
>>> +            $(if $(GITLAB_CI),,--failfast) $(AVOCADO_TESTS),             \
>>
>> We might need to revisit this --failfast logic, too. If I've got that 
>> right, failfast is now the default with the new system? So we might want 
>> to disable it by default again if GITLAB_CI is not set?
>>
>>  Thomas
>>
> Hi Thomas,
> 
> I must be missing something, because under Avocado 101.0, I'm getting the 
> following behavior without the --failfast flag:
> 
>    $ avocado run --max-parallel-tasks=1 -- /bin/true /bin/false /bin/true
>    JOB ID     : 646f476f01b8d5599a57530606de543f2d9a5366
>    JOB LOG    : /root/avocado/job-results/job-2023-04-21T04.15-646f476/job.log
>     (1/3) /bin/true: STARTED
>     (1/3) /bin/true: PASS (0.01 s)
>     (2/3) /bin/false: STARTED
>     (2/3) /bin/false: FAIL (0.01 s)
>     (3/3) /bin/true: STARTED
>     (3/3) /bin/true: PASS (0.01 s)
>    RESULTS    : PASS 2 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
> CANCEL 0
>    JOB TIME   : 1.01 s
> 
>    Test summary:
>    /bin/false: FAIL
> 
> And this with --failfast:
> 
>    $ avocado run --failfast --max-parallel-tasks=1 -- /bin/true /bin/false 
> /bin/true
>    JOB ID     : ae4894607a42194a7382efa545eccaccf7495fa3
>    JOB LOG    : /root/avocado/job-results/job-2023-04-21T04.17-ae48946/job.log
>     (1/3) /bin/true: STARTED
>     (1/3) /bin/true: PASS (0.01 s)
>     (2/3) /bin/false: STARTED
>     (2/3) /bin/false: FAIL (0.01 s)
>    Interrupting job (failfast).
>    RESULTS    : PASS 1 | ERROR 0 | FAIL 1 | SKIP 1 | WARN 0 | INTERRUPT 0 | 
> CANCEL 0
>    JOB TIME   : 0.75 s
> 
>    Test summary:
>    /bin/false: FAIL
> 
> Maybe it's something in the Makefile I'm missing, or something specific to 
> these tests, but that would be a long shot.

Ah, thanks, I think I simply got it mixed up while trying to read the above 
Makefile logic - never mind my question, it should be fine.

  Thomas