* Functional tests precache behaviour @ 2025-04-30 14:34 Pierrick Bouvier 2025-04-30 15:00 ` Thomas Huth 0 siblings, 1 reply; 13+ messages in thread From: Pierrick Bouvier @ 2025-04-30 14:34 UTC (permalink / raw) To: Thomas Huth, Daniel P. Berrangé, qemu-devel@nongnu.org Hi folks, $ ninja -C build precache-functional 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https://archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: HTTP error 503 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https://archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: HTTP error 503 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https://archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: HTTP error 503 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: Download retries exceeded: skipping asset precache $ echo $? 0 Since we silently skip the asset precaching, how can we identify that an asset is not available anymore (temporarily or not)? Should we rely on test itself failing when trying to download again this asset? Regards, Pierrick ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Functional tests precache behaviour 2025-04-30 14:34 Functional tests precache behaviour Pierrick Bouvier @ 2025-04-30 15:00 ` Thomas Huth 2025-04-30 15:48 ` Pierrick Bouvier 2025-04-30 16:22 ` Pierrick Bouvier 0 siblings, 2 replies; 13+ messages in thread From: Thomas Huth @ 2025-04-30 15:00 UTC (permalink / raw) To: Pierrick Bouvier, Daniel P. Berrangé, qemu-devel@nongnu.org On 30/04/2025 16.34, Pierrick Bouvier wrote: > Hi folks, > > $ ninja -C build precache-functional > 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https:// > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ > gzimg/armv7.img.gz: HTTP error 503 > 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https:// > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ > gzimg/armv7.img.gz: HTTP error 503 > 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https:// > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ > gzimg/armv7.img.gz: HTTP error 503 > 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/ > pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: > Download retries exceeded: skipping asset precache > $ echo $? > 0 > > Since we silently skip the asset precaching, how can we identify that an > asset is not available anymore (temporarily or not)? > Should we rely on test itself failing when trying to download again this asset? The current logic fails hard for 404 errors, so if the asset is completely gone, we should notice it. For other error codes, we assume that it is only a temporary server problem that will hopefully be fixed on the server side sooner or later. Thomas ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Functional tests precache behaviour 2025-04-30 15:00 ` Thomas Huth @ 2025-04-30 15:48 ` Pierrick Bouvier 2025-04-30 16:02 ` Daniel P. Berrangé 2025-04-30 16:22 ` Pierrick Bouvier 1 sibling, 1 reply; 13+ messages in thread From: Pierrick Bouvier @ 2025-04-30 15:48 UTC (permalink / raw) To: Thomas Huth, Daniel P. Berrangé, qemu-devel@nongnu.org On 4/30/25 8:00 AM, Thomas Huth wrote: > On 30/04/2025 16.34, Pierrick Bouvier wrote: >> Hi folks, >> >> $ ninja -C build precache-functional >> 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https:// >> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >> gzimg/armv7.img.gz: HTTP error 503 >> 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https:// >> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >> gzimg/armv7.img.gz: HTTP error 503 >> 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https:// >> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >> gzimg/armv7.img.gz: HTTP error 503 >> 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/ >> pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: >> Download retries exceeded: skipping asset precache >> $ echo $? >> 0 >> >> Since we silently skip the asset precaching, how can we identify that an >> asset is not available anymore (temporarily or not)? >> Should we rely on test itself failing when trying to download again this asset? > > The current logic fails hard for 404 errors, so if the asset is completely > gone, we should notice it. For other error codes, we assume that it is only > a temporary server problem that will hopefully be fixed on the server side > sooner or later. > Sounds good. Should we replicate this semantic when running the test itself? It would be more useful to skip it because an asset is missing instead of reporting an error, except if it's a 404 error. > Thomas > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Functional tests precache behaviour 2025-04-30 15:48 ` Pierrick Bouvier @ 2025-04-30 16:02 ` Daniel P. Berrangé 2025-04-30 16:21 ` Pierrick Bouvier 0 siblings, 1 reply; 13+ messages in thread From: Daniel P. Berrangé @ 2025-04-30 16:02 UTC (permalink / raw) To: Pierrick Bouvier; +Cc: Thomas Huth, qemu-devel@nongnu.org On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote: > On 4/30/25 8:00 AM, Thomas Huth wrote: > > On 30/04/2025 16.34, Pierrick Bouvier wrote: > > > Hi folks, > > > > > > $ ninja -C build precache-functional > > > 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https:// > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ > > > gzimg/armv7.img.gz: HTTP error 503 > > > 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https:// > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ > > > gzimg/armv7.img.gz: HTTP error 503 > > > 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https:// > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ > > > gzimg/armv7.img.gz: HTTP error 503 > > > 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/ > > > pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: > > > Download retries exceeded: skipping asset precache > > > $ echo $? > > > 0 > > > > > > Since we silently skip the asset precaching, how can we identify that an > > > asset is not available anymore (temporarily or not)? > > > Should we rely on test itself failing when trying to download again this asset? > > > > The current logic fails hard for 404 errors, so if the asset is completely > > gone, we should notice it. For other error codes, we assume that it is only > > a temporary server problem that will hopefully be fixed on the server side > > sooner or later. > > > > Sounds good. > Should we replicate this semantic when running the test itself? > It would be more useful to skip it because an asset is missing instead of > reporting an error, except if it's a 404 error. The tests already gracefully skip if one or more required assets are not available. See the 'setUp' method of QemuBaseTest if not self.assets_available(): self.skipTest('One or more assets is not available') In the 404 case, the pre-cache step should fail and thus we shouldn't even get to running the test. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Functional tests precache behaviour 2025-04-30 16:02 ` Daniel P. Berrangé @ 2025-04-30 16:21 ` Pierrick Bouvier 2025-04-30 16:29 ` Daniel P. Berrangé 0 siblings, 1 reply; 13+ messages in thread From: Pierrick Bouvier @ 2025-04-30 16:21 UTC (permalink / raw) To: Daniel P. Berrangé; +Cc: Thomas Huth, qemu-devel@nongnu.org On 4/30/25 9:02 AM, Daniel P. Berrangé wrote: > On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote: >> On 4/30/25 8:00 AM, Thomas Huth wrote: >>> On 30/04/2025 16.34, Pierrick Bouvier wrote: >>>> Hi folks, >>>> >>>> $ ninja -C build precache-functional >>>> 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https:// >>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >>>> gzimg/armv7.img.gz: HTTP error 503 >>>> 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https:// >>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >>>> gzimg/armv7.img.gz: HTTP error 503 >>>> 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https:// >>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >>>> gzimg/armv7.img.gz: HTTP error 503 >>>> 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/ >>>> pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: >>>> Download retries exceeded: skipping asset precache >>>> $ echo $? >>>> 0 >>>> >>>> Since we silently skip the asset precaching, how can we identify that an >>>> asset is not available anymore (temporarily or not)? >>>> Should we rely on test itself failing when trying to download again this asset? >>> >>> The current logic fails hard for 404 errors, so if the asset is completely >>> gone, we should notice it. For other error codes, we assume that it is only >>> a temporary server problem that will hopefully be fixed on the server side >>> sooner or later. >>> >> >> Sounds good. >> Should we replicate this semantic when running the test itself? >> It would be more useful to skip it because an asset is missing instead of >> reporting an error, except if it's a 404 error. > > The tests already gracefully skip if one or more required assets > are not available. See the 'setUp' method of QemuBaseTest > > if not self.assets_available(): > self.skipTest('One or more assets is not available') > > > In the 404 case, the pre-cache step should fail and thus we shouldn't > even get to running the test. > This is not the behaviour I observe (error, with server returning 503) [1], thus my original email. Maybe something is missing in the associated test, or in our test infrastructure? Nothing funky in the command line used, you can reproduce it with: $ rm -rf ~/.cache/qemu build/ $ ./configure $ ./build/pyvenv/bin/meson test -C build --setup thorough --suite func-quick --suite func-thorough -t 5 --print-errorlogs func-ppc-ppc_40p [1] https://github.com/pbo-linaro/qemu-ci/actions/runs/14747788692/job/41398348905 Traceback (most recent call last): File "/home/runner/work/qemu-ci/qemu-ci/tests/functional/test_ppc_40p.py", line 68, in test_openbios_and_netbsd drive_path = self.ASSET_NETBSD71.fetch() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/runner/work/qemu-ci/qemu-ci/tests/functional/qemu_test/asset.py", line 175, in fetch raise AssetError(self, "Download retries exceeded", transient=True) qemu_test.asset.AssetError: https://archive.netbsd.org/pub/NetBSD-archive/NetBSD-7.1.2/iso/NetBSD-7.1.2-prep.iso: Download retries exceeded > > With regards, > Daniel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Functional tests precache behaviour 2025-04-30 16:21 ` Pierrick Bouvier @ 2025-04-30 16:29 ` Daniel P. Berrangé 2025-04-30 16:34 ` Pierrick Bouvier 0 siblings, 1 reply; 13+ messages in thread From: Daniel P. Berrangé @ 2025-04-30 16:29 UTC (permalink / raw) To: Pierrick Bouvier; +Cc: Thomas Huth, qemu-devel@nongnu.org On Wed, Apr 30, 2025 at 09:21:41AM -0700, Pierrick Bouvier wrote: > On 4/30/25 9:02 AM, Daniel P. Berrangé wrote: > > On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote: > > > On 4/30/25 8:00 AM, Thomas Huth wrote: > > > > On 30/04/2025 16.34, Pierrick Bouvier wrote: > > > > > Hi folks, > > > > > > > > > > $ ninja -C build precache-functional > > > > > 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https:// > > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ > > > > > gzimg/armv7.img.gz: HTTP error 503 > > > > > 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https:// > > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ > > > > > gzimg/armv7.img.gz: HTTP error 503 > > > > > 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https:// > > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ > > > > > gzimg/armv7.img.gz: HTTP error 503 > > > > > 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/ > > > > > pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: > > > > > Download retries exceeded: skipping asset precache > > > > > $ echo $? > > > > > 0 > > > > > > > > > > Since we silently skip the asset precaching, how can we identify that an > > > > > asset is not available anymore (temporarily or not)? > > > > > Should we rely on test itself failing when trying to download again this asset? > > > > > > > > The current logic fails hard for 404 errors, so if the asset is completely > > > > gone, we should notice it. For other error codes, we assume that it is only > > > > a temporary server problem that will hopefully be fixed on the server side > > > > sooner or later. > > > > > > > > > > Sounds good. > > > Should we replicate this semantic when running the test itself? > > > It would be more useful to skip it because an asset is missing instead of > > > reporting an error, except if it's a 404 error. > > > > The tests already gracefully skip if one or more required assets > > are not available. See the 'setUp' method of QemuBaseTest > > > > if not self.assets_available(): > > self.skipTest('One or more assets is not available') > > > > > > In the 404 case, the pre-cache step should fail and thus we shouldn't > > even get to running the test. > > > > This is not the behaviour I observe (error, with server returning 503) [1], > thus my original email. > > Maybe something is missing in the associated test, or in our test > infrastructure? > > Nothing funky in the command line used, you can reproduce it with: > $ rm -rf ~/.cache/qemu build/ > $ ./configure > $ ./build/pyvenv/bin/meson test -C build --setup thorough --suite func-quick > --suite func-thorough -t 5 --print-errorlogs func-ppc-ppc_40p Oh, you're running meson test directly. The behaviour I describe is wrt the official way of running tests via 'make check' or 'make check-functional'. When you use 'make', we set 'QEMU_TEST_NO_DOWNLOAD=1' when the tests themselves are run, so only the 'make precache-functional' will be permitted to try downloading. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Functional tests precache behaviour 2025-04-30 16:29 ` Daniel P. Berrangé @ 2025-04-30 16:34 ` Pierrick Bouvier 2025-04-30 16:39 ` Daniel P. Berrangé 0 siblings, 1 reply; 13+ messages in thread From: Pierrick Bouvier @ 2025-04-30 16:34 UTC (permalink / raw) To: Daniel P. Berrangé; +Cc: Thomas Huth, qemu-devel@nongnu.org On 4/30/25 9:29 AM, Daniel P. Berrangé wrote: > On Wed, Apr 30, 2025 at 09:21:41AM -0700, Pierrick Bouvier wrote: >> On 4/30/25 9:02 AM, Daniel P. Berrangé wrote: >>> On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote: >>>> On 4/30/25 8:00 AM, Thomas Huth wrote: >>>>> On 30/04/2025 16.34, Pierrick Bouvier wrote: >>>>>> Hi folks, >>>>>> >>>>>> $ ninja -C build precache-functional >>>>>> 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https:// >>>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >>>>>> gzimg/armv7.img.gz: HTTP error 503 >>>>>> 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https:// >>>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >>>>>> gzimg/armv7.img.gz: HTTP error 503 >>>>>> 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https:// >>>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >>>>>> gzimg/armv7.img.gz: HTTP error 503 >>>>>> 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/ >>>>>> pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: >>>>>> Download retries exceeded: skipping asset precache >>>>>> $ echo $? >>>>>> 0 >>>>>> >>>>>> Since we silently skip the asset precaching, how can we identify that an >>>>>> asset is not available anymore (temporarily or not)? >>>>>> Should we rely on test itself failing when trying to download again this asset? >>>>> >>>>> The current logic fails hard for 404 errors, so if the asset is completely >>>>> gone, we should notice it. For other error codes, we assume that it is only >>>>> a temporary server problem that will hopefully be fixed on the server side >>>>> sooner or later. >>>>> >>>> >>>> Sounds good. >>>> Should we replicate this semantic when running the test itself? >>>> It would be more useful to skip it because an asset is missing instead of >>>> reporting an error, except if it's a 404 error. >>> >>> The tests already gracefully skip if one or more required assets >>> are not available. See the 'setUp' method of QemuBaseTest >>> >>> if not self.assets_available(): >>> self.skipTest('One or more assets is not available') >>> >>> >>> In the 404 case, the pre-cache step should fail and thus we shouldn't >>> even get to running the test. >>> >> >> This is not the behaviour I observe (error, with server returning 503) [1], >> thus my original email. >> >> Maybe something is missing in the associated test, or in our test >> infrastructure? >> Or... in my command :) >> Nothing funky in the command line used, you can reproduce it with: >> $ rm -rf ~/.cache/qemu build/ >> $ ./configure >> $ ./build/pyvenv/bin/meson test -C build --setup thorough --suite func-quick >> --suite func-thorough -t 5 --print-errorlogs func-ppc-ppc_40p > > Oh, you're running meson test directly. > > The behaviour I describe is wrt the official way of running tests via > 'make check' or 'make check-functional'. > > When you use 'make', we set 'QEMU_TEST_NO_DOWNLOAD=1' when the tests > themselves are run, so only the 'make precache-functional' will be > permitted to try downloading. > Oh thanks, that's what I was missing! I'm running meson because the Makefile wrapper does not allow to pass any additional parameters, or running specific test. > With regards, > Daniel ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Functional tests precache behaviour 2025-04-30 16:34 ` Pierrick Bouvier @ 2025-04-30 16:39 ` Daniel P. Berrangé 2025-04-30 16:46 ` Pierrick Bouvier 2025-05-01 17:56 ` Peter Maydell 0 siblings, 2 replies; 13+ messages in thread From: Daniel P. Berrangé @ 2025-04-30 16:39 UTC (permalink / raw) To: Pierrick Bouvier; +Cc: Thomas Huth, qemu-devel@nongnu.org On Wed, Apr 30, 2025 at 09:34:10AM -0700, Pierrick Bouvier wrote: > On 4/30/25 9:29 AM, Daniel P. Berrangé wrote: > > On Wed, Apr 30, 2025 at 09:21:41AM -0700, Pierrick Bouvier wrote: > > > On 4/30/25 9:02 AM, Daniel P. Berrangé wrote: > > > > On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote: > > > > > On 4/30/25 8:00 AM, Thomas Huth wrote: > > > > > > On 30/04/2025 16.34, Pierrick Bouvier wrote: > > > > > > > Hi folks, > > > > > > > > > > > > > > $ ninja -C build precache-functional > > > > > > > 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https:// > > > > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ > > > > > > > gzimg/armv7.img.gz: HTTP error 503 > > > > > > > 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https:// > > > > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ > > > > > > > gzimg/armv7.img.gz: HTTP error 503 > > > > > > > 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https:// > > > > > > > archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ > > > > > > > gzimg/armv7.img.gz: HTTP error 503 > > > > > > > 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/ > > > > > > > pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: > > > > > > > Download retries exceeded: skipping asset precache > > > > > > > $ echo $? > > > > > > > 0 > > > > > > > > > > > > > > Since we silently skip the asset precaching, how can we identify that an > > > > > > > asset is not available anymore (temporarily or not)? > > > > > > > Should we rely on test itself failing when trying to download again this asset? > > > > > > > > > > > > The current logic fails hard for 404 errors, so if the asset is completely > > > > > > gone, we should notice it. For other error codes, we assume that it is only > > > > > > a temporary server problem that will hopefully be fixed on the server side > > > > > > sooner or later. > > > > > > > > > > > > > > > > Sounds good. > > > > > Should we replicate this semantic when running the test itself? > > > > > It would be more useful to skip it because an asset is missing instead of > > > > > reporting an error, except if it's a 404 error. > > > > > > > > The tests already gracefully skip if one or more required assets > > > > are not available. See the 'setUp' method of QemuBaseTest > > > > > > > > if not self.assets_available(): > > > > self.skipTest('One or more assets is not available') > > > > > > > > > > > > In the 404 case, the pre-cache step should fail and thus we shouldn't > > > > even get to running the test. > > > > > > > > > > This is not the behaviour I observe (error, with server returning 503) [1], > > > thus my original email. > > > > > > Maybe something is missing in the associated test, or in our test > > > infrastructure? > > > > > Or... in my command :) > > > > Nothing funky in the command line used, you can reproduce it with: > > > $ rm -rf ~/.cache/qemu build/ > > > $ ./configure > > > $ ./build/pyvenv/bin/meson test -C build --setup thorough --suite func-quick > > > --suite func-thorough -t 5 --print-errorlogs func-ppc-ppc_40p > > > > Oh, you're running meson test directly. > > > > The behaviour I describe is wrt the official way of running tests via > > 'make check' or 'make check-functional'. > > > > When you use 'make', we set 'QEMU_TEST_NO_DOWNLOAD=1' when the tests > > themselves are run, so only the 'make precache-functional' will be > > permitted to try downloading. > > > > Oh thanks, that's what I was missing! > > I'm running meson because the Makefile wrapper does not allow to pass any > additional parameters, or running specific test. FWIW, if you want to run a specific test, personally don't use meson or make, as you can just invoke the file directly: $ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64 \ PYTHONPATH=./python \ ./tests/functional/test_x86_cpu_model_versions.py This was the key feature I wanted when we replaced avocado, as debugging tests without a harness getting in the way is much simpler With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Functional tests precache behaviour 2025-04-30 16:39 ` Daniel P. Berrangé @ 2025-04-30 16:46 ` Pierrick Bouvier 2025-05-01 17:56 ` Peter Maydell 1 sibling, 0 replies; 13+ messages in thread From: Pierrick Bouvier @ 2025-04-30 16:46 UTC (permalink / raw) To: Daniel P. Berrangé; +Cc: Thomas Huth, qemu-devel@nongnu.org On 4/30/25 9:39 AM, Daniel P. Berrangé wrote: > On Wed, Apr 30, 2025 at 09:34:10AM -0700, Pierrick Bouvier wrote: >> On 4/30/25 9:29 AM, Daniel P. Berrangé wrote: >>> On Wed, Apr 30, 2025 at 09:21:41AM -0700, Pierrick Bouvier wrote: >>>> On 4/30/25 9:02 AM, Daniel P. Berrangé wrote: >>>>> On Wed, Apr 30, 2025 at 08:48:59AM -0700, Pierrick Bouvier wrote: >>>>>> On 4/30/25 8:00 AM, Thomas Huth wrote: >>>>>>> On 30/04/2025 16.34, Pierrick Bouvier wrote: >>>>>>>> Hi folks, >>>>>>>> >>>>>>>> $ ninja -C build precache-functional >>>>>>>> 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https:// >>>>>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >>>>>>>> gzimg/armv7.img.gz: HTTP error 503 >>>>>>>> 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https:// >>>>>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >>>>>>>> gzimg/armv7.img.gz: HTTP error 503 >>>>>>>> 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https:// >>>>>>>> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >>>>>>>> gzimg/armv7.img.gz: HTTP error 503 >>>>>>>> 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/ >>>>>>>> pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: >>>>>>>> Download retries exceeded: skipping asset precache >>>>>>>> $ echo $? >>>>>>>> 0 >>>>>>>> >>>>>>>> Since we silently skip the asset precaching, how can we identify that an >>>>>>>> asset is not available anymore (temporarily or not)? >>>>>>>> Should we rely on test itself failing when trying to download again this asset? >>>>>>> >>>>>>> The current logic fails hard for 404 errors, so if the asset is completely >>>>>>> gone, we should notice it. For other error codes, we assume that it is only >>>>>>> a temporary server problem that will hopefully be fixed on the server side >>>>>>> sooner or later. >>>>>>> >>>>>> >>>>>> Sounds good. >>>>>> Should we replicate this semantic when running the test itself? >>>>>> It would be more useful to skip it because an asset is missing instead of >>>>>> reporting an error, except if it's a 404 error. >>>>> >>>>> The tests already gracefully skip if one or more required assets >>>>> are not available. See the 'setUp' method of QemuBaseTest >>>>> >>>>> if not self.assets_available(): >>>>> self.skipTest('One or more assets is not available') >>>>> >>>>> >>>>> In the 404 case, the pre-cache step should fail and thus we shouldn't >>>>> even get to running the test. >>>>> >>>> >>>> This is not the behaviour I observe (error, with server returning 503) [1], >>>> thus my original email. >>>> >>>> Maybe something is missing in the associated test, or in our test >>>> infrastructure? >>>> >> >> Or... in my command :) >> >>>> Nothing funky in the command line used, you can reproduce it with: >>>> $ rm -rf ~/.cache/qemu build/ >>>> $ ./configure >>>> $ ./build/pyvenv/bin/meson test -C build --setup thorough --suite func-quick >>>> --suite func-thorough -t 5 --print-errorlogs func-ppc-ppc_40p >>> >>> Oh, you're running meson test directly. >>> >>> The behaviour I describe is wrt the official way of running tests via >>> 'make check' or 'make check-functional'. >>> >>> When you use 'make', we set 'QEMU_TEST_NO_DOWNLOAD=1' when the tests >>> themselves are run, so only the 'make precache-functional' will be >>> permitted to try downloading. >>> >> >> Oh thanks, that's what I was missing! >> >> I'm running meson because the Makefile wrapper does not allow to pass any >> additional parameters, or running specific test. > > FWIW, if you want to run a specific test, personally don't use meson > or make, as you can just invoke the file directly: > > $ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64 \ > PYTHONPATH=./python \ > ./tests/functional/test_x86_cpu_model_versions.py > > This was the key feature I wanted when we replaced avocado, as debugging > tests without a harness getting in the way is much simpler > Sounds good, thanks Daniel. I usually find meson easier to run, since it will build the code also, and allow to list all tests easily. I just think the behaviour of missing assets relying on precache-functional is a bit awkward (it would be more intuitive to gracefully skip the test without needing a special env var), but since I'm running an undocumented workflow, I won't insist. Thanks for all the information. > With regards, > Daniel Regards, Pierrick ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Functional tests precache behaviour 2025-04-30 16:39 ` Daniel P. Berrangé 2025-04-30 16:46 ` Pierrick Bouvier @ 2025-05-01 17:56 ` Peter Maydell 2025-05-01 21:26 ` Pierrick Bouvier 1 sibling, 1 reply; 13+ messages in thread From: Peter Maydell @ 2025-05-01 17:56 UTC (permalink / raw) To: Daniel P. Berrangé Cc: Pierrick Bouvier, Thomas Huth, qemu-devel@nongnu.org On Wed, 30 Apr 2025 at 17:41, Daniel P. Berrangé <berrange@redhat.com> wrote: > FWIW, if you want to run a specific test, personally don't use meson > or make, as you can just invoke the file directly: > > $ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64 \ > PYTHONPATH=./python \ > ./tests/functional/test_x86_cpu_model_versions.py The rune in docs/devel says you also need to: * put tests/functional on the PYTHONPATH too * run from the build tree, not the source tree * run using the python binary in pyvenv/ So you can do this, but it's pretty clunky; I have to look up the runes every time. It would be nice if there was a wrapper to do this for you. (Also it doesn't work if the thing you're trying to test is "does this test pass within the meson test timeout" :-)) IIRC there is also a rune for "run a single test within make/meson", but I forget what it is and docs/devel/testing/functional.rst doesn't mention it. thanks -- PMM ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Functional tests precache behaviour 2025-05-01 17:56 ` Peter Maydell @ 2025-05-01 21:26 ` Pierrick Bouvier 2025-05-03 20:21 ` Pierrick Bouvier 0 siblings, 1 reply; 13+ messages in thread From: Pierrick Bouvier @ 2025-05-01 21:26 UTC (permalink / raw) To: Peter Maydell, Daniel P. Berrangé; +Cc: Thomas Huth, qemu-devel@nongnu.org On 5/1/25 10:56 AM, Peter Maydell wrote: > On Wed, 30 Apr 2025 at 17:41, Daniel P. Berrangé <berrange@redhat.com> wrote: >> FWIW, if you want to run a specific test, personally don't use meson >> or make, as you can just invoke the file directly: >> >> $ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64 \ >> PYTHONPATH=./python \ >> ./tests/functional/test_x86_cpu_model_versions.py > > The rune in docs/devel says you also need to: > * put tests/functional on the PYTHONPATH too > * run from the build tree, not the source tree > * run using the python binary in pyvenv/ > > So you can do this, but it's pretty clunky; I have to > look up the runes every time. It would be nice if there > was a wrapper to do this for you. > I think that meson test command is pretty easy and "standard" (once learned, you can apply this to any other project using meson test infrastructure), so maybe it's the wrapper we could be interested to promote. If we go this way, two things would be interesting to change: - enable setup thorough by default, so all tests are visible by default (instead of having to dive into tests/meson.build, and understand which setup does what). make check-functional can still restrict the setup, it's not a problem. - abort gracefully (without needing an extra env var) when an asset is missing and can't be downloaded, with a different error than 404. With those two changes, "meson test" does all we need, including the build, in an intuitive and standard way. > (Also it doesn't work if the thing you're trying to test > is "does this test pass within the meson test timeout" :-)) > > IIRC there is also a rune for "run a single test within make/meson", > but I forget what it is and docs/devel/testing/functional.rst > doesn't mention it. > I agree with you that even though it's possible, it's something you need to check in the documentation everytime, which is not convenient. > thanks > -- PMM ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Functional tests precache behaviour 2025-05-01 21:26 ` Pierrick Bouvier @ 2025-05-03 20:21 ` Pierrick Bouvier 0 siblings, 0 replies; 13+ messages in thread From: Pierrick Bouvier @ 2025-05-03 20:21 UTC (permalink / raw) To: Peter Maydell, Daniel P. Berrangé; +Cc: Thomas Huth, qemu-devel@nongnu.org On 5/1/25 2:26 PM, Pierrick Bouvier wrote: > On 5/1/25 10:56 AM, Peter Maydell wrote: >> On Wed, 30 Apr 2025 at 17:41, Daniel P. Berrangé <berrange@redhat.com> wrote: >>> FWIW, if you want to run a specific test, personally don't use meson >>> or make, as you can just invoke the file directly: >>> >>> $ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64 \ >>> PYTHONPATH=./python \ >>> ./tests/functional/test_x86_cpu_model_versions.py >> >> The rune in docs/devel says you also need to: >> * put tests/functional on the PYTHONPATH too >> * run from the build tree, not the source tree >> * run using the python binary in pyvenv/ >> >> So you can do this, but it's pretty clunky; I have to >> look up the runes every time. It would be nice if there >> was a wrapper to do this for you. >> > > I think that meson test command is pretty easy and "standard" (once > learned, you can apply this to any other project using meson test > infrastructure), so maybe it's the wrapper we could be interested to > promote. > > If we go this way, two things would be interesting to change: > - enable setup thorough by default, so all tests are visible by default > (instead of having to dive into tests/meson.build, and understand which > setup does what). make check-functional can still restrict the setup, > it's not a problem. Sent a series for this: https://lore.kernel.org/qemu-devel/20250503201806.3045723-1-pierrick.bouvier@linaro.org > - abort gracefully (without needing an extra env var) when an asset is > missing and can't be downloaded, with a different error than 404. > Not easy to do, as asset.fetch() is not aware of current test, so it can't called skipTest, and it would be cumbersome to have to pass the self TestCase as a parameter. > With those two changes, "meson test" does all we need, including the > build, in an intuitive and standard way. > >> (Also it doesn't work if the thing you're trying to test >> is "does this test pass within the meson test timeout" :-)) >> >> IIRC there is also a rune for "run a single test within make/meson", >> but I forget what it is and docs/devel/testing/functional.rst >> doesn't mention it. >> > > I agree with you that even though it's possible, it's something you need > to check in the documentation everytime, which is not convenient. > >> thanks >> -- PMM > ^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: Functional tests precache behaviour 2025-04-30 15:00 ` Thomas Huth 2025-04-30 15:48 ` Pierrick Bouvier @ 2025-04-30 16:22 ` Pierrick Bouvier 1 sibling, 0 replies; 13+ messages in thread From: Pierrick Bouvier @ 2025-04-30 16:22 UTC (permalink / raw) To: Thomas Huth, Daniel P. Berrangé, qemu-devel@nongnu.org On 4/30/25 8:00 AM, Thomas Huth wrote: > On 30/04/2025 16.34, Pierrick Bouvier wrote: >> Hi folks, >> >> $ ninja -C build precache-functional >> 2025-04-30 07:23:20,382 - qemu-test - ERROR - Unable to download https:// >> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >> gzimg/armv7.img.gz: HTTP error 503 >> 2025-04-30 07:23:23,131 - qemu-test - ERROR - Unable to download https:// >> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >> gzimg/armv7.img.gz: HTTP error 503 >> 2025-04-30 07:23:25,870 - qemu-test - ERROR - Unable to download https:// >> archive.netbsd.org/pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/ >> gzimg/armv7.img.gz: HTTP error 503 >> 2025-04-30 07:23:25,871 - qemu-test - ERROR - https://archive.netbsd.org/ >> pub/NetBSD-archive/NetBSD-9.0/evbarm-earmv7hf/binary/gzimg/armv7.img.gz: >> Download retries exceeded: skipping asset precache >> $ echo $? >> 0 >> >> Since we silently skip the asset precaching, how can we identify that an >> asset is not available anymore (temporarily or not)? >> Should we rely on test itself failing when trying to download again this asset? > > The current logic fails hard for 404 errors, so if the asset is completely > gone, we should notice it. For other error codes, we assume that it is only > a temporary server problem that will hopefully be fixed on the server side > sooner or later. > > Thomas > By the way, thanks for all your effort to get rid of avocado tests, and converting them to functional. It's infinitely better, especially the caching aspect. ^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2025-05-03 20:21 UTC | newest] Thread overview: 13+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2025-04-30 14:34 Functional tests precache behaviour Pierrick Bouvier 2025-04-30 15:00 ` Thomas Huth 2025-04-30 15:48 ` Pierrick Bouvier 2025-04-30 16:02 ` Daniel P. Berrangé 2025-04-30 16:21 ` Pierrick Bouvier 2025-04-30 16:29 ` Daniel P. Berrangé 2025-04-30 16:34 ` Pierrick Bouvier 2025-04-30 16:39 ` Daniel P. Berrangé 2025-04-30 16:46 ` Pierrick Bouvier 2025-05-01 17:56 ` Peter Maydell 2025-05-01 21:26 ` Pierrick Bouvier 2025-05-03 20:21 ` Pierrick Bouvier 2025-04-30 16:22 ` Pierrick Bouvier
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).