Re: Functional tests precache behaviour

["Daniel P. Berrangé" <berrange@redhat.com>]

On Wed, Apr 30, 2025 at 09:34:10AM -0700, Pierrick Bouvier wrote:
> On 4/30/25 9:29 AM, Daniel P. Berrangé wrote:
> > On Wed, Apr 30, 2025 at 09:21:41AM -0700, Pierrick Bouvier wrote:
> > > On 4/30/25 9:02 AM, Daniel P. Berrangé wrote:
> > > > On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote:
> > > > > On 4/30/25 8:00 AM, Thomas Huth wrote:
> > > > > > On 30/04/2025 16.34, Pierrick Bouvier wrote:
> > > > > > > Hi folks,
> > > > > > > 
> > > > > > > $ ninja -C build precache-functional
> > > > > > > 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https://
> > > > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > > > > > gzimg/armv7.img.gz: HTTP error 503
> > > > > > > 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https://
> > > > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > > > > > gzimg/armv7.img.gz: HTTP error 503
> > > > > > > 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https://
> > > > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > > > > > gzimg/armv7.img.gz: HTTP error 503
> > > > > > > 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/
> > > > > > > pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz:
> > > > > > > Download retries exceeded: skipping asset precache
> > > > > > > $ echo $?
> > > > > > > 0
> > > > > > > 
> > > > > > > Since we silently skip the asset precaching, how can we identify that an
> > > > > > > asset is not available anymore (temporarily or not)?
> > > > > > > Should we rely on test itself failing when trying to download again this asset?
> > > > > > 
> > > > > > The current logic fails hard for 404 errors, so if the asset is completely
> > > > > > gone, we should notice it. For other error codes, we assume that it is only
> > > > > > a temporary server problem that will hopefully be fixed on the server side
> > > > > > sooner or later.
> > > > > > 
> > > > > 
> > > > > Sounds good.
> > > > > Should we replicate this semantic when running the test itself?
> > > > > It would be more useful to skip it because an asset is missing instead of
> > > > > reporting an error, except if it's a 404 error.
> > > > 
> > > > The tests already gracefully skip if one or more required assets
> > > > are not available. See the 'setUp' method of QemuBaseTest
> > > > 
> > > >           if not self.assets_available():
> > > >               self.skipTest('One or more assets is not available')
> > > > 
> > > > 
> > > > In the 404 case, the pre-cache step should fail and thus we shouldn't
> > > > even get to running the test.
> > > > 
> > > 
> > > This is not the behaviour I observe (error, with server returning 503) [1],
> > > thus my original email.
> > > 
> > > Maybe something is missing in the associated test, or in our test
> > > infrastructure?
> > > 
> 
> Or... in my command :)
> 
> > > Nothing funky in the command line used, you can reproduce it with:
> > > $ rm -rf ~/.cache/qemu build/
> > > $ ./configure
> > > $ ./build/pyvenv/bin/meson test -C build --setup thorough --suite func-quick
> > > --suite func-thorough -t 5 --print-errorlogs func-ppc-ppc_40p
> > 
> > Oh, you're running meson test directly.
> > 
> > The behaviour I describe is wrt the official way of running tests via
> > 'make check' or 'make check-functional'.
> > 
> > When you use 'make', we set 'QEMU_TEST_NO_DOWNLOAD=1' when the tests
> > themselves are run, so only the 'make precache-functional' will be
> > permitted to try downloading.
> > 
> 
> Oh thanks, that's what I was missing!
> 
> I'm running meson because the Makefile wrapper does not allow to pass any
> additional parameters, or running specific test.

FWIW, if you want to run a specific test, personally don't use meson
or make, as you can just invoke the file directly:

 $ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64 \
   PYTHONPATH=./python \
   ./tests/functional/test_x86_cpu_model_versions.py 

This was the key feature I wanted when we replaced avocado, as debugging
tests without a harness getting in the way is much simpler

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|