Functional tests precache behaviour

Functional tests precache behaviour

[Pierrick Bouvier]

Hi folks,

$ ninja -C build precache-functional
2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download 
https://archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: 
HTTP error 503
2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download 
https://archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: 
HTTP error 503
2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download 
https://archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: 
HTTP error 503
2025-04-30 07:23:25,871 - qemu-test - ERROR - 
https://archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: 
Download retries exceeded: skipping asset precache
$ echo $?
0

Since we silently skip the asset precaching, how can we identify that an 
asset is not available anymore (temporarily or not)?
Should we rely on test itself failing when trying to download again this 
asset?

Regards,
Pierrick

Re: Functional tests precache behaviour

[Thomas Huth]

On 30/04/2025 16.34, Pierrick Bouvier wrote:
> Hi folks,
> 
> $ ninja -C build precache-functional
> 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https:// 
> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ 
> gzimg/armv7.img.gz: HTTP error 503
> 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https:// 
> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ 
> gzimg/armv7.img.gz: HTTP error 503
> 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https:// 
> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ 
> gzimg/armv7.img.gz: HTTP error 503
> 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/ 
> pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: 
> Download retries exceeded: skipping asset precache
> $ echo $?
> 0
> 
> Since we silently skip the asset precaching, how can we identify that an 
> asset is not available anymore (temporarily or not)?
> Should we rely on test itself failing when trying to download again this asset?

The current logic fails hard for 404 errors, so if the asset is completely 
gone, we should notice it. For other error codes, we assume that it is only 
a temporary server problem that will hopefully be fixed on the server side 
sooner or later.

  Thomas

Re: Functional tests precache behaviour

[Pierrick Bouvier]

On 4/30/25 8:00 AM, Thomas Huth wrote:
> On 30/04/2025 16.34, Pierrick Bouvier wrote:
>> Hi folks,
>>
>> $ ninja -C build precache-functional
>> 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https://
>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>> gzimg/armv7.img.gz: HTTP error 503
>> 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https://
>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>> gzimg/armv7.img.gz: HTTP error 503
>> 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https://
>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>> gzimg/armv7.img.gz: HTTP error 503
>> 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/
>> pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz:
>> Download retries exceeded: skipping asset precache
>> $ echo $?
>> 0
>>
>> Since we silently skip the asset precaching, how can we identify that an
>> asset is not available anymore (temporarily or not)?
>> Should we rely on test itself failing when trying to download again this asset?
> 
> The current logic fails hard for 404 errors, so if the asset is completely
> gone, we should notice it. For other error codes, we assume that it is only
> a temporary server problem that will hopefully be fixed on the server side
> sooner or later.
>

Sounds good.
Should we replicate this semantic when running the test itself?
It would be more useful to skip it because an asset is missing instead 
of reporting an error, except if it's a 404 error.

>    Thomas
> 

Re: Functional tests precache behaviour

[Daniel P. Berrangé]

On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote:
> On 4/30/25 8:00 AM, Thomas Huth wrote:
> > On 30/04/2025 16.34, Pierrick Bouvier wrote:
> > > Hi folks,
> > > 
> > > $ ninja -C build precache-functional
> > > 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https://
> > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > gzimg/armv7.img.gz: HTTP error 503
> > > 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https://
> > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > gzimg/armv7.img.gz: HTTP error 503
> > > 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https://
> > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > gzimg/armv7.img.gz: HTTP error 503
> > > 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/
> > > pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz:
> > > Download retries exceeded: skipping asset precache
> > > $ echo $?
> > > 0
> > > 
> > > Since we silently skip the asset precaching, how can we identify that an
> > > asset is not available anymore (temporarily or not)?
> > > Should we rely on test itself failing when trying to download again this asset?
> > 
> > The current logic fails hard for 404 errors, so if the asset is completely
> > gone, we should notice it. For other error codes, we assume that it is only
> > a temporary server problem that will hopefully be fixed on the server side
> > sooner or later.
> > 
> 
> Sounds good.
> Should we replicate this semantic when running the test itself?
> It would be more useful to skip it because an asset is missing instead of
> reporting an error, except if it's a 404 error.

The tests already gracefully skip if one or more required assets
are not available. See the 'setUp' method of QemuBaseTest

        if not self.assets_available():
            self.skipTest('One or more assets is not available')


In the 404 case, the pre-cache step should fail and thus we shouldn't
even get to running the test.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Functional tests precache behaviour

[Pierrick Bouvier]

On 4/30/25 9:02 AM, Daniel P. Berrangé wrote:
> On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote:
>> On 4/30/25 8:00 AM, Thomas Huth wrote:
>>> On 30/04/2025 16.34, Pierrick Bouvier wrote:
>>>> Hi folks,
>>>>
>>>> $ ninja -C build precache-functional
>>>> 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https://
>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>>>> gzimg/armv7.img.gz: HTTP error 503
>>>> 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https://
>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>>>> gzimg/armv7.img.gz: HTTP error 503
>>>> 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https://
>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>>>> gzimg/armv7.img.gz: HTTP error 503
>>>> 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/
>>>> pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz:
>>>> Download retries exceeded: skipping asset precache
>>>> $ echo $?
>>>> 0
>>>>
>>>> Since we silently skip the asset precaching, how can we identify that an
>>>> asset is not available anymore (temporarily or not)?
>>>> Should we rely on test itself failing when trying to download again this asset?
>>>
>>> The current logic fails hard for 404 errors, so if the asset is completely
>>> gone, we should notice it. For other error codes, we assume that it is only
>>> a temporary server problem that will hopefully be fixed on the server side
>>> sooner or later.
>>>
>>
>> Sounds good.
>> Should we replicate this semantic when running the test itself?
>> It would be more useful to skip it because an asset is missing instead of
>> reporting an error, except if it's a 404 error.
> 
> The tests already gracefully skip if one or more required assets
> are not available. See the 'setUp' method of QemuBaseTest
> 
>          if not self.assets_available():
>              self.skipTest('One or more assets is not available')
> 
> 
> In the 404 case, the pre-cache step should fail and thus we shouldn't
> even get to running the test.
> 

This is not the behaviour I observe (error, with server returning 503) 
[1], thus my original email.

Maybe something is missing in the associated test, or in our test 
infrastructure?

Nothing funky in the command line used, you can reproduce it with:
$ rm -rf ~/.cache/qemu build/
$ ./configure
$ ./build/pyvenv/bin/meson test -C build --setup thorough --suite 
func-quick --suite func-thorough -t 5 --print-errorlogs func-ppc-ppc_40p

[1] 
https://github.com/pbo-linaro/qemu-ci/actions/runs/14747788692/job/41398348905

Traceback (most recent call last):
   File 
"/home/runner/work/qemu-ci/qemu-ci/tests/functional/test_ppc_40p.py", 
line 68, in test_openbios_and_netbsd
     drive_path = self.ASSET_NETBSD71.fetch()
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   File 
"/home/runner/work/qemu-ci/qemu-ci/tests/functional/qemu_test/asset.py", 
line 175, in fetch
     raise AssetError(self, "Download retries exceeded", transient=True)
qemu_test.asset.AssetError: 
https://archive.netbsd.org/pub/NetBSD-archive/NetBSD-7.1.2/iso/NetBSD-7.1.2-prep.iso: 
Download retries exceeded

> 
> With regards,
> Daniel

Re: Functional tests precache behaviour

[Pierrick Bouvier]

On 4/30/25 8:00 AM, Thomas Huth wrote:
> On 30/04/2025 16.34, Pierrick Bouvier wrote:
>> Hi folks,
>>
>> $ ninja -C build precache-functional
>> 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https://
>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>> gzimg/armv7.img.gz: HTTP error 503
>> 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https://
>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>> gzimg/armv7.img.gz: HTTP error 503
>> 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https://
>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>> gzimg/armv7.img.gz: HTTP error 503
>> 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/
>> pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz:
>> Download retries exceeded: skipping asset precache
>> $ echo $?
>> 0
>>
>> Since we silently skip the asset precaching, how can we identify that an
>> asset is not available anymore (temporarily or not)?
>> Should we rely on test itself failing when trying to download again this asset?
> 
> The current logic fails hard for 404 errors, so if the asset is completely
> gone, we should notice it. For other error codes, we assume that it is only
> a temporary server problem that will hopefully be fixed on the server side
> sooner or later.
> 
>    Thomas
> 

By the way,
thanks for all your effort to get rid of avocado tests, and converting 
them to functional. It's infinitely better, especially the caching aspect.

Re: Functional tests precache behaviour

[Daniel P. Berrangé]

On Wed, Apr 30, 2025 at 09:21:41AM -0700, Pierrick Bouvier wrote:
> On 4/30/25 9:02 AM, Daniel P. Berrangé wrote:
> > On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote:
> > > On 4/30/25 8:00 AM, Thomas Huth wrote:
> > > > On 30/04/2025 16.34, Pierrick Bouvier wrote:
> > > > > Hi folks,
> > > > > 
> > > > > $ ninja -C build precache-functional
> > > > > 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https://
> > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > > > gzimg/armv7.img.gz: HTTP error 503
> > > > > 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https://
> > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > > > gzimg/armv7.img.gz: HTTP error 503
> > > > > 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https://
> > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > > > gzimg/armv7.img.gz: HTTP error 503
> > > > > 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/
> > > > > pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz:
> > > > > Download retries exceeded: skipping asset precache
> > > > > $ echo $?
> > > > > 0
> > > > > 
> > > > > Since we silently skip the asset precaching, how can we identify that an
> > > > > asset is not available anymore (temporarily or not)?
> > > > > Should we rely on test itself failing when trying to download again this asset?
> > > > 
> > > > The current logic fails hard for 404 errors, so if the asset is completely
> > > > gone, we should notice it. For other error codes, we assume that it is only
> > > > a temporary server problem that will hopefully be fixed on the server side
> > > > sooner or later.
> > > > 
> > > 
> > > Sounds good.
> > > Should we replicate this semantic when running the test itself?
> > > It would be more useful to skip it because an asset is missing instead of
> > > reporting an error, except if it's a 404 error.
> > 
> > The tests already gracefully skip if one or more required assets
> > are not available. See the 'setUp' method of QemuBaseTest
> > 
> >          if not self.assets_available():
> >              self.skipTest('One or more assets is not available')
> > 
> > 
> > In the 404 case, the pre-cache step should fail and thus we shouldn't
> > even get to running the test.
> > 
> 
> This is not the behaviour I observe (error, with server returning 503) [1],
> thus my original email.
> 
> Maybe something is missing in the associated test, or in our test
> infrastructure?
> 
> Nothing funky in the command line used, you can reproduce it with:
> $ rm -rf ~/.cache/qemu build/
> $ ./configure
> $ ./build/pyvenv/bin/meson test -C build --setup thorough --suite func-quick
> --suite func-thorough -t 5 --print-errorlogs func-ppc-ppc_40p

Oh, you're running meson test directly.

The behaviour I describe is wrt the official way of running tests via
'make check' or 'make check-functional'.

When you use 'make', we set 'QEMU_TEST_NO_DOWNLOAD=1' when the tests
themselves are run, so only the 'make precache-functional' will be
permitted to try downloading.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Functional tests precache behaviour

[Pierrick Bouvier]

On 4/30/25 9:29 AM, Daniel P. Berrangé wrote:
> On Wed, Apr 30, 2025 at 09:21:41AM -0700, Pierrick Bouvier wrote:
>> On 4/30/25 9:02 AM, Daniel P. Berrangé wrote:
>>> On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote:
>>>> On 4/30/25 8:00 AM, Thomas Huth wrote:
>>>>> On 30/04/2025 16.34, Pierrick Bouvier wrote:
>>>>>> Hi folks,
>>>>>>
>>>>>> $ ninja -C build precache-functional
>>>>>> 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https://
>>>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>>>>>> gzimg/armv7.img.gz: HTTP error 503
>>>>>> 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https://
>>>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>>>>>> gzimg/armv7.img.gz: HTTP error 503
>>>>>> 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https://
>>>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>>>>>> gzimg/armv7.img.gz: HTTP error 503
>>>>>> 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/
>>>>>> pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz:
>>>>>> Download retries exceeded: skipping asset precache
>>>>>> $ echo $?
>>>>>> 0
>>>>>>
>>>>>> Since we silently skip the asset precaching, how can we identify that an
>>>>>> asset is not available anymore (temporarily or not)?
>>>>>> Should we rely on test itself failing when trying to download again this asset?
>>>>>
>>>>> The current logic fails hard for 404 errors, so if the asset is completely
>>>>> gone, we should notice it. For other error codes, we assume that it is only
>>>>> a temporary server problem that will hopefully be fixed on the server side
>>>>> sooner or later.
>>>>>
>>>>
>>>> Sounds good.
>>>> Should we replicate this semantic when running the test itself?
>>>> It would be more useful to skip it because an asset is missing instead of
>>>> reporting an error, except if it's a 404 error.
>>>
>>> The tests already gracefully skip if one or more required assets
>>> are not available. See the 'setUp' method of QemuBaseTest
>>>
>>>           if not self.assets_available():
>>>               self.skipTest('One or more assets is not available')
>>>
>>>
>>> In the 404 case, the pre-cache step should fail and thus we shouldn't
>>> even get to running the test.
>>>
>>
>> This is not the behaviour I observe (error, with server returning 503) [1],
>> thus my original email.
>>
>> Maybe something is missing in the associated test, or in our test
>> infrastructure?
>>

Or... in my command :)

>> Nothing funky in the command line used, you can reproduce it with:
>> $ rm -rf ~/.cache/qemu build/
>> $ ./configure
>> $ ./build/pyvenv/bin/meson test -C build --setup thorough --suite func-quick
>> --suite func-thorough -t 5 --print-errorlogs func-ppc-ppc_40p
> 
> Oh, you're running meson test directly.
> 
> The behaviour I describe is wrt the official way of running tests via
> 'make check' or 'make check-functional'.
> 
> When you use 'make', we set 'QEMU_TEST_NO_DOWNLOAD=1' when the tests
> themselves are run, so only the 'make precache-functional' will be
> permitted to try downloading.
>

Oh thanks, that's what I was missing!

I'm running meson because the Makefile wrapper does not allow to pass 
any additional parameters, or running specific test.

> With regards,
> Daniel

Re: Functional tests precache behaviour

[Daniel P. Berrangé]

On Wed, Apr 30, 2025 at 09:34:10AM -0700, Pierrick Bouvier wrote:
> On 4/30/25 9:29 AM, Daniel P. Berrangé wrote:
> > On Wed, Apr 30, 2025 at 09:21:41AM -0700, Pierrick Bouvier wrote:
> > > On 4/30/25 9:02 AM, Daniel P. Berrangé wrote:
> > > > On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote:
> > > > > On 4/30/25 8:00 AM, Thomas Huth wrote:
> > > > > > On 30/04/2025 16.34, Pierrick Bouvier wrote:
> > > > > > > Hi folks,
> > > > > > > 
> > > > > > > $ ninja -C build precache-functional
> > > > > > > 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https://
> > > > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > > > > > gzimg/armv7.img.gz: HTTP error 503
> > > > > > > 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https://
> > > > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > > > > > gzimg/armv7.img.gz: HTTP error 503
> > > > > > > 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https://
> > > > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > > > > > gzimg/armv7.img.gz: HTTP error 503
> > > > > > > 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/
> > > > > > > pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz:
> > > > > > > Download retries exceeded: skipping asset precache
> > > > > > > $ echo $?
> > > > > > > 0
> > > > > > > 
> > > > > > > Since we silently skip the asset precaching, how can we identify that an
> > > > > > > asset is not available anymore (temporarily or not)?
> > > > > > > Should we rely on test itself failing when trying to download again this asset?
> > > > > > 
> > > > > > The current logic fails hard for 404 errors, so if the asset is completely
> > > > > > gone, we should notice it. For other error codes, we assume that it is only
> > > > > > a temporary server problem that will hopefully be fixed on the server side
> > > > > > sooner or later.
> > > > > > 
> > > > > 
> > > > > Sounds good.
> > > > > Should we replicate this semantic when running the test itself?
> > > > > It would be more useful to skip it because an asset is missing instead of
> > > > > reporting an error, except if it's a 404 error.
> > > > 
> > > > The tests already gracefully skip if one or more required assets
> > > > are not available. See the 'setUp' method of QemuBaseTest
> > > > 
> > > >           if not self.assets_available():
> > > >               self.skipTest('One or more assets is not available')
> > > > 
> > > > 
> > > > In the 404 case, the pre-cache step should fail and thus we shouldn't
> > > > even get to running the test.
> > > > 
> > > 
> > > This is not the behaviour I observe (error, with server returning 503) [1],
> > > thus my original email.
> > > 
> > > Maybe something is missing in the associated test, or in our test
> > > infrastructure?
> > > 
> 
> Or... in my command :)
> 
> > > Nothing funky in the command line used, you can reproduce it with:
> > > $ rm -rf ~/.cache/qemu build/
> > > $ ./configure
> > > $ ./build/pyvenv/bin/meson test -C build --setup thorough --suite func-quick
> > > --suite func-thorough -t 5 --print-errorlogs func-ppc-ppc_40p
> > 
> > Oh, you're running meson test directly.
> > 
> > The behaviour I describe is wrt the official way of running tests via
> > 'make check' or 'make check-functional'.
> > 
> > When you use 'make', we set 'QEMU_TEST_NO_DOWNLOAD=1' when the tests
> > themselves are run, so only the 'make precache-functional' will be
> > permitted to try downloading.
> > 
> 
> Oh thanks, that's what I was missing!
> 
> I'm running meson because the Makefile wrapper does not allow to pass any
> additional parameters, or running specific test.

FWIW, if you want to run a specific test, personally don't use meson
or make, as you can just invoke the file directly:

 $ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64 \
   PYTHONPATH=./python \
   ./tests/functional/test_x86_cpu_model_versions.py 

This was the key feature I wanted when we replaced avocado, as debugging
tests without a harness getting in the way is much simpler

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Functional tests precache behaviour

[Pierrick Bouvier]

On 4/30/25 9:39 AM, Daniel P. Berrangé wrote:
> On Wed, Apr 30, 2025 at 09:34:10AM -0700, Pierrick Bouvier wrote:
>> On 4/30/25 9:29 AM, Daniel P. Berrangé wrote:
>>> On Wed, Apr 30, 2025 at 09:21:41AM -0700, Pierrick Bouvier wrote:
>>>> On 4/30/25 9:02 AM, Daniel P. Berrangé wrote:
>>>>> On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote:
>>>>>> On 4/30/25 8:00 AM, Thomas Huth wrote:
>>>>>>> On 30/04/2025 16.34, Pierrick Bouvier wrote:
>>>>>>>> Hi folks,
>>>>>>>>
>>>>>>>> $ ninja -C build precache-functional
>>>>>>>> 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https://
>>>>>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>>>>>>>> gzimg/armv7.img.gz: HTTP error 503
>>>>>>>> 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https://
>>>>>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>>>>>>>> gzimg/armv7.img.gz: HTTP error 503
>>>>>>>> 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https://
>>>>>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
>>>>>>>> gzimg/armv7.img.gz: HTTP error 503
>>>>>>>> 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/
>>>>>>>> pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz:
>>>>>>>> Download retries exceeded: skipping asset precache
>>>>>>>> $ echo $?
>>>>>>>> 0
>>>>>>>>
>>>>>>>> Since we silently skip the asset precaching, how can we identify that an
>>>>>>>> asset is not available anymore (temporarily or not)?
>>>>>>>> Should we rely on test itself failing when trying to download again this asset?
>>>>>>>
>>>>>>> The current logic fails hard for 404 errors, so if the asset is completely
>>>>>>> gone, we should notice it. For other error codes, we assume that it is only
>>>>>>> a temporary server problem that will hopefully be fixed on the server side
>>>>>>> sooner or later.
>>>>>>>
>>>>>>
>>>>>> Sounds good.
>>>>>> Should we replicate this semantic when running the test itself?
>>>>>> It would be more useful to skip it because an asset is missing instead of
>>>>>> reporting an error, except if it's a 404 error.
>>>>>
>>>>> The tests already gracefully skip if one or more required assets
>>>>> are not available. See the 'setUp' method of QemuBaseTest
>>>>>
>>>>>            if not self.assets_available():
>>>>>                self.skipTest('One or more assets is not available')
>>>>>
>>>>>
>>>>> In the 404 case, the pre-cache step should fail and thus we shouldn't
>>>>> even get to running the test.
>>>>>
>>>>
>>>> This is not the behaviour I observe (error, with server returning 503) [1],
>>>> thus my original email.
>>>>
>>>> Maybe something is missing in the associated test, or in our test
>>>> infrastructure?
>>>>
>>
>> Or... in my command :)
>>
>>>> Nothing funky in the command line used, you can reproduce it with:
>>>> $ rm -rf ~/.cache/qemu build/
>>>> $ ./configure
>>>> $ ./build/pyvenv/bin/meson test -C build --setup thorough --suite func-quick
>>>> --suite func-thorough -t 5 --print-errorlogs func-ppc-ppc_40p
>>>
>>> Oh, you're running meson test directly.
>>>
>>> The behaviour I describe is wrt the official way of running tests via
>>> 'make check' or 'make check-functional'.
>>>
>>> When you use 'make', we set 'QEMU_TEST_NO_DOWNLOAD=1' when the tests
>>> themselves are run, so only the 'make precache-functional' will be
>>> permitted to try downloading.
>>>
>>
>> Oh thanks, that's what I was missing!
>>
>> I'm running meson because the Makefile wrapper does not allow to pass any
>> additional parameters, or running specific test.
> 
> FWIW, if you want to run a specific test, personally don't use meson
> or make, as you can just invoke the file directly:
> 
>   $ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64 \
>     PYTHONPATH=./python \
>     ./tests/functional/test_x86_cpu_model_versions.py
>
> This was the key feature I wanted when we replaced avocado, as debugging
> tests without a harness getting in the way is much simpler
>

Sounds good, thanks Daniel.

I usually find meson easier to run, since it will build the code also, 
and allow to list all tests easily. I just think the behaviour of 
missing assets relying on precache-functional is a bit awkward (it would 
be more intuitive to gracefully skip the test without needing a special 
env var), but since I'm running an undocumented workflow, I won't insist.

Thanks for all the information.

> With regards,
> Daniel

Regards,
Pierrick

Re: Functional tests precache behaviour

[Peter Maydell]

On Wed, 30 Apr 2025 at 17:41, Daniel P. Berrangé <berrange@redhat.com> wrote:
> FWIW, if you want to run a specific test, personally don't use meson
> or make, as you can just invoke the file directly:
>
>  $ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64 \
>    PYTHONPATH=./python \
>    ./tests/functional/test_x86_cpu_model_versions.py

The rune in docs/devel says you also need to:
 * put tests/functional on the PYTHONPATH too
 * run from the build tree, not the source tree
 * run using the python binary in pyvenv/

So you can do this, but it's pretty clunky; I have to
look up the runes every time. It would be nice if there
was a wrapper to do this for you.

(Also it doesn't work if the thing you're trying to test
is "does this test pass within the meson test timeout" :-))

IIRC there is also a rune for "run a single test within make/meson",
but I forget what it is and docs/devel/testing/functional.rst
doesn't mention it.

thanks
-- PMM

Re: Functional tests precache behaviour

[Pierrick Bouvier]

On 5/1/25 10:56 AM, Peter Maydell wrote:
> On Wed, 30 Apr 2025 at 17:41, Daniel P. Berrangé <berrange@redhat.com> wrote:
>> FWIW, if you want to run a specific test, personally don't use meson
>> or make, as you can just invoke the file directly:
>>
>>   $ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64 \
>>     PYTHONPATH=./python \
>>     ./tests/functional/test_x86_cpu_model_versions.py
> 
> The rune in docs/devel says you also need to:
>   * put tests/functional on the PYTHONPATH too
>   * run from the build tree, not the source tree
>   * run using the python binary in pyvenv/
> 
> So you can do this, but it's pretty clunky; I have to
> look up the runes every time. It would be nice if there
> was a wrapper to do this for you.
> 

I think that meson test command is pretty easy and "standard" (once 
learned, you can apply this to any other project using meson test 
infrastructure), so maybe it's the wrapper we could be interested to 
promote.

If we go this way, two things would be interesting to change:
- enable setup thorough by default, so all tests are visible by default 
(instead of having to dive into tests/meson.build, and understand which 
setup does what). make check-functional can still restrict the setup, 
it's not a problem.
- abort gracefully (without needing an extra env var) when an asset is 
missing and can't be downloaded, with a different error than 404.

With those two changes, "meson test" does all we need, including the 
build, in an intuitive and standard way.

> (Also it doesn't work if the thing you're trying to test
> is "does this test pass within the meson test timeout" :-))
> 
> IIRC there is also a rune for "run a single test within make/meson",
> but I forget what it is and docs/devel/testing/functional.rst
> doesn't mention it.
>

I agree with you that even though it's possible, it's something you need 
to check in the documentation everytime, which is not convenient.

> thanks
> -- PMM

Re: Functional tests precache behaviour

[Pierrick Bouvier]

On 5/1/25 2:26 PM, Pierrick Bouvier wrote:
> On 5/1/25 10:56 AM, Peter Maydell wrote:
>> On Wed, 30 Apr 2025 at 17:41, Daniel P. Berrangé <berrange@redhat.com> wrote:
>>> FWIW, if you want to run a specific test, personally don't use meson
>>> or make, as you can just invoke the file directly:
>>>
>>>    $ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64 \
>>>      PYTHONPATH=./python \
>>>      ./tests/functional/test_x86_cpu_model_versions.py
>>
>> The rune in docs/devel says you also need to:
>>    * put tests/functional on the PYTHONPATH too
>>    * run from the build tree, not the source tree
>>    * run using the python binary in pyvenv/
>>
>> So you can do this, but it's pretty clunky; I have to
>> look up the runes every time. It would be nice if there
>> was a wrapper to do this for you.
>>
> 
> I think that meson test command is pretty easy and "standard" (once
> learned, you can apply this to any other project using meson test
> infrastructure), so maybe it's the wrapper we could be interested to
> promote.
> 
> If we go this way, two things would be interesting to change:
> - enable setup thorough by default, so all tests are visible by default
> (instead of having to dive into tests/meson.build, and understand which
> setup does what). make check-functional can still restrict the setup,
> it's not a problem.

Sent a series for this:
https://lore.kernel.org/qemu-devel/20250503201806.3045723-1-pierrick.bouvier@linaro.org

> - abort gracefully (without needing an extra env var) when an asset is
> missing and can't be downloaded, with a different error than 404.
> 

Not easy to do, as asset.fetch() is not aware of current test, so it 
can't called skipTest, and it would be cumbersome to have to pass the 
self TestCase as a parameter.

> With those two changes, "meson test" does all we need, including the
> build, in an intuitive and standard way.
> 
>> (Also it doesn't work if the thing you're trying to test
>> is "does this test pass within the meson test timeout" :-))
>>
>> IIRC there is also a rune for "run a single test within make/meson",
>> but I forget what it is and docs/devel/testing/functional.rst
>> doesn't mention it.
>>
> 
> I agree with you that even though it's possible, it's something you need
> to check in the documentation everytime, which is not convenient.
> 
>> thanks
>> -- PMM
>