Re: Functional tests precache behaviour

[Pierrick Bouvier <pierrick.bouvier@linaro.org>]

On 4/30/25 9:02 AM, Daniel P. Berrangé wrote:
> On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote:
>> On 4/30/25 8:00 AM, Thomas Huth wrote:
>>> On 30/04/2025 16.34, Pierrick Bouvier wrote:
>>>> Hi folks,
>>>>
>>>> $ ninja -C build precache-functional
>>>> 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https://
>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>>>> gzimg/armv7.img.gz: HTTP error 503
>>>> 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https://
>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>>>> gzimg/armv7.img.gz: HTTP error 503
>>>> 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https://
>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>>>> gzimg/armv7.img.gz: HTTP error 503
>>>> 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/
>>>> pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz:
>>>> Download retries exceeded: skipping asset precache
>>>> $ echo $?
>>>> 0
>>>>
>>>> Since we silently skip the asset precaching, how can we identify that an
>>>> asset is not available anymore (temporarily or not)?
>>>> Should we rely on test itself failing when trying to download again this asset?
>>>
>>> The current logic fails hard for 404 errors, so if the asset is completely
>>> gone, we should notice it. For other error codes, we assume that it is only
>>> a temporary server problem that will hopefully be fixed on the server side
>>> sooner or later.
>>>
>>
>> Sounds good.
>> Should we replicate this semantic when running the test itself?
>> It would be more useful to skip it because an asset is missing instead of
>> reporting an error, except if it's a 404 error.
> 
> The tests already gracefully skip if one or more required assets
> are not available. See the 'setUp' method of QemuBaseTest
> 
>          if not self.assets_available():
>              self.skipTest('One or more assets is not available')
> 
> 
> In the 404 case, the pre-cache step should fail and thus we shouldn't
> even get to running the test.
> 

This is not the behaviour I observe (error, with server returning 503) 
[1], thus my original email.

Maybe something is missing in the associated test, or in our test 
infrastructure?

Nothing funky in the command line used, you can reproduce it with:
$ rm -rf ~/.cache/qemu build/
$ ./configure
$ ./build/pyvenv/bin/meson test -C build --setup thorough --suite 
func-quick --suite func-thorough -t 5 --print-errorlogs func-ppc-ppc_40p

[1] 
https://github.com/pbo-linaro/qemu-ci/actions/runs/14747788692/job/41398348905

Traceback (most recent call last):
   File 
"/home/runner/work/qemu-ci/qemu-ci/tests/functional/test_ppc_40p.py", 
line 68, in test_openbios_and_netbsd
     drive_path = self.ASSET_NETBSD71.fetch()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File 
"/home/runner/work/qemu-ci/qemu-ci/tests/functional/qemu_test/asset.py", 
line 175, in fetch
     raise AssetError(self, "Download retries exceeded", transient=True)
qemu_test.asset.AssetError: 
https://archive.netbsd.org/pub/NetBSD-archive/NetBSD-7.1.2/iso/NetBSD-7.1.2-prep.iso: 
Download retries exceeded

> 
> With regards,
> Daniel