Re: Functional tests precache behaviour

[Pierrick Bouvier <pierrick.bouvier@linaro.org>]

On 5/1/25 10:56 AM, Peter Maydell wrote:
> On Wed, 30 Apr 2025 at 17:41, Daniel P. Berrangé <berrange@redhat.com> wrote:
>> FWIW, if you want to run a specific test, personally don't use meson
>> or make, as you can just invoke the file directly:
>>
>>   $ QEMU_TEST_QEMU_BINARY=./build/qemu-system-x86_64 \
>>     PYTHONPATH=./python \
>>     ./tests/functional/test_x86_cpu_model_versions.py
> 
> The rune in docs/devel says you also need to:
>   * put tests/functional on the PYTHONPATH too
>   * run from the build tree, not the source tree
>   * run using the python binary in pyvenv/
> 
> So you can do this, but it's pretty clunky; I have to
> look up the runes every time. It would be nice if there
> was a wrapper to do this for you.
> 

I think that meson test command is pretty easy and "standard" (once 
learned, you can apply this to any other project using meson test 
infrastructure), so maybe it's the wrapper we could be interested to 
promote.

If we go this way, two things would be interesting to change:
- enable setup thorough by default, so all tests are visible by default 
(instead of having to dive into tests/meson.build, and understand which 
setup does what). make check-functional can still restrict the setup, 
it's not a problem.
- abort gracefully (without needing an extra env var) when an asset is 
missing and can't be downloaded, with a different error than 404.

With those two changes, "meson test" does all we need, including the 
build, in an intuitive and standard way.

> (Also it doesn't work if the thing you're trying to test
> is "does this test pass within the meson test timeout" :-))
> 
> IIRC there is also a rune for "run a single test within make/meson",
> but I forget what it is and docs/devel/testing/functional.rst
> doesn't mention it.
>

I agree with you that even though it's possible, it's something you need 
to check in the documentation everytime, which is not convenient.

> thanks
> -- PMM