Re: Functional tests precache behaviour

["Daniel P. Berrangé" <berrange@redhat.com>]

On Wed, Apr 30, 2025 at 09:21:41AM -0700, Pierrick Bouvier wrote:
> On 4/30/25 9:02 AM, Daniel P. Berrangé wrote:
> > On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote:
> > > On 4/30/25 8:00 AM, Thomas Huth wrote:
> > > > On 30/04/2025 16.34, Pierrick Bouvier wrote:
> > > > > Hi folks,
> > > > > 
> > > > > $ ninja -C build precache-functional
> > > > > 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https://
> > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > > > gzimg/armv7.img.gz: HTTP error 503
> > > > > 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https://
> > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > > > gzimg/armv7.img.gz: HTTP error 503
> > > > > 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https://
> > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/
> > > > > gzimg/armv7.img.gz: HTTP error 503
> > > > > 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/
> > > > > pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz:
> > > > > Download retries exceeded: skipping asset precache
> > > > > $ echo $?
> > > > > 0
> > > > > 
> > > > > Since we silently skip the asset precaching, how can we identify that an
> > > > > asset is not available anymore (temporarily or not)?
> > > > > Should we rely on test itself failing when trying to download again this asset?
> > > > 
> > > > The current logic fails hard for 404 errors, so if the asset is completely
> > > > gone, we should notice it. For other error codes, we assume that it is only
> > > > a temporary server problem that will hopefully be fixed on the server side
> > > > sooner or later.
> > > > 
> > > 
> > > Sounds good.
> > > Should we replicate this semantic when running the test itself?
> > > It would be more useful to skip it because an asset is missing instead of
> > > reporting an error, except if it's a 404 error.
> > 
> > The tests already gracefully skip if one or more required assets
> > are not available. See the 'setUp' method of QemuBaseTest
> > 
> >          if not self.assets_available():
> >              self.skipTest('One or more assets is not available')
> > 
> > 
> > In the 404 case, the pre-cache step should fail and thus we shouldn't
> > even get to running the test.
> > 
> 
> This is not the behaviour I observe (error, with server returning 503) [1],
> thus my original email.
> 
> Maybe something is missing in the associated test, or in our test
> infrastructure?
> 
> Nothing funky in the command line used, you can reproduce it with:
> $ rm -rf ~/.cache/qemu build/
> $ ./configure
> $ ./build/pyvenv/bin/meson test -C build --setup thorough --suite func-quick
> --suite func-thorough -t 5 --print-errorlogs func-ppc-ppc_40p

Oh, you're running meson test directly.

The behaviour I describe is wrt the official way of running tests via
'make check' or 'make check-functional'.

When you use 'make', we set 'QEMU_TEST_NO_DOWNLOAD=1' when the tests
themselves are run, so only the 'make precache-functional' will be
permitted to try downloading.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|