Re: [PATCH 1/1] tests/avocado: update firmware to enable sbsa-ref/max

["Philippe Mathieu-Daudé" <philmd@linaro.org>]

On 7/6/23 17:29, Marcin Juszkiewicz wrote:
> W dniu 7.06.2023 o 16:33, Philippe Mathieu-Daudé pisze:
>> On 30/5/23 17:22, Marcin Juszkiewicz wrote:
>>> Update prebuilt firmware images to have TF-A with FEAT_FGT support
>>> enabled. This allowed us to enable test for "max" cpu in sbsa-ref
>>> machine.
>>>
>>> Signed-off-by: Marcin Juszkiewicz <marcin.juszkiewicz@linaro.org>
>>> ---
>>>   tests/avocado/machine_aarch64_sbsaref.py | 22 +++++++++++-----------
>>>   1 file changed, 11 insertions(+), 11 deletions(-)
>>>
>>> diff --git a/tests/avocado/machine_aarch64_sbsaref.py 
>>> b/tests/avocado/machine_aarch64_sbsaref.py
>>> index 0a79fa7ab6..35f8042416 100644
>>> --- a/tests/avocado/machine_aarch64_sbsaref.py
>>> +++ b/tests/avocado/machine_aarch64_sbsaref.py
>>> @@ -29,23 +29,23 @@ def fetch_firmware(self):
>>>           """
>>>           Flash volumes generated using:
>>> -        - Fedora GNU Toolchain version 12.2.1 20220819 (Red Hat 
>>> Cross 12.2.1-2)
>>> +        - Fedora GNU Toolchain version 13.1.1 20230511 (Red Hat 
>>> 13.1.1-2)
>>>           - Trusted Firmware-A
>>> - https://github.com/ARM-software/arm-trusted-firmware/tree/5fdb2e54
>>> + https://github.com/ARM-software/arm-trusted-firmware/tree/c0d8ee38
>>>           - Tianocore EDK II
>>> -          https://github.com/tianocore/edk2/tree/494127613b
>>> -          https://github.com/tianocore/edk2-non-osi/tree/41876073
>>> -          https://github.com/tianocore/edk2-platforms/tree/8efa4f42
>>> +          https://github.com/tianocore/edk2/tree/0f9283429dd4
>>> +          https://github.com/tianocore/edk2-non-osi/tree/f0bb00937ad6
>>> +          https://github.com/tianocore/edk2-platforms/tree/7880b92e2a04
>>
>> Thanks for updating this comment!
> 
> Having a way to reproduce is crucial for CI.
> 
>>> -    @skip("requires TF-A update to handle FEAT_FGT")
>>> +    @skipUnless(os.getenv("AVOCADO_TIMEOUT_EXPECTED"), "Test might 
>>> timeout")
>>
>> Can it still timeout?
> 
> All Linux based tests in this file have that @skipUnless as they take 
> some time:
> 
> test_sbsaref_edk2_firmware: PASS (2.72 s)
> test_sbsaref_alpine_linux_cortex_a57: PASS (23.71 s)
> test_sbsaref_alpine_linux_neoverse_n1: PASS (23.53 s)
> test_sbsaref_alpine_linux_max: PASS (28.16 s)

I suppose this was due to a bug we had with Avocado consuming QEMU's
console. I don't remember recent complains. Alex, do you know if this
was fixed?
We define the class timeout to 180s, so all tests inherit it. In your
run all tests take <30sec, it should be fine to run them on CI.
Adding ~1min30s extra on the job running these Avocado tests seem
reasonable to me, but I have Cc'ed Thomas who took care to reduce
testing time.

Regards,

Phil.