Re: [RFC PATCH 4/8] tests/pytest: add pytest to the meson build system

[Thomas Huth <thuth@redhat.com>]

On 12/07/2024 12.26, Daniel P. Berrangé wrote:
> On Fri, Jul 12, 2024 at 12:14:45PM +0200, Thomas Huth wrote:
>> On 12/07/2024 11.01, Daniel P. Berrangé wrote:
>>> On Thu, Jul 11, 2024 at 01:55:42PM +0200, Thomas Huth wrote:
>>>> From: Ani Sinha <ani@anisinha.ca>
>>>>
>>>> Integrate the pytest framework with the meson build system. This
>>>> will make meson run all the pytests under the pytest directory.
>>>
>>> Lets add a note about the compelling benefit of this new approach
>>>
>>>     With this change, each functional test becomes subject
>>>     to an individual execution timeout, defaulting to 60
>>>     seconds, but overridable per-test.
>>
>> The avocado runner uses timeouts, too, so it's not really an additional
>> benefit that we get here.
> 
> At the meson level though, we can't put an overall cap on
> the execution time, as there's only 1 huge test visible,
> and thus the meson timeout multiplier also won't get
> honoured IIUC.
> 
>>
>>> For CI purposes we'll need to add 'python3-pytest' to
>>> tests/lcitool/projects/qemu.yml, and re-generate the
>>> the dockerfiles. Some of the other non-gitlab CI
>>> integrations probably need manual additions of pytest
>>> packages.
>>
>> I'm currently rather looking into getting rid of pytest and to use pycotap
>> instead: Using the TAP protocol for running the tests, you get a much nicer
>> output from the meson test runner, which can then count the subtests and
>> properly report SKIPs for tests that have not been run.
>>
>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>> index d39d5dd6a4..68151717d7 100644
>>>> --- a/tests/Makefile.include
>>>> +++ b/tests/Makefile.include
>>>> @@ -3,12 +3,14 @@
>>>>    .PHONY: check-help
>>>>    check-help:
>>>>    	@echo "Regression testing targets:"
>>>> -	@echo " $(MAKE) check                  Run block, qapi-schema, unit, softfloat, qtest and decodetree tests"
>>>> +	@echo " $(MAKE) check                  Run block, qapi-schema, unit, softfloat, qtest, pytest and decodetree tests"
>>>>    	@echo " $(MAKE) bench                  Run speed tests"
>>>>    	@echo
>>>>    	@echo "Individual test suites:"
>>>>    	@echo " $(MAKE) check-qtest-TARGET     Run qtest tests for given target"
>>>>    	@echo " $(MAKE) check-qtest            Run qtest tests"
>>>> +	@echo " $(MAKE) check-pytest           Run pytest tests"
>>>> +	@echo " $(MAKE) check-pytest-TARGET    Run pytest for a given target"
>>>
>>> Or name it after the type of test rather than harness ?
>>>
>>>    eg  check-functional / check-functional-TARGET
>>>
>>> For that matter perhaps also for the dir name ?
>>>
>>>      tests/functional/*.py
>>
>> I almost expected that discussion again ... (see
>> https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html ) ...
>> last time we couldn't really agree on such a name and decided to go with the
>> name of the framework...
>>
>> I agree that "pytest" is likely not the best name here, especially if
>> switching to the pycotap test runner instead of using the "pytest" program,
>> but "functional" might trigger the same discussion again as last time ...
>> should it rather be "functional" or "validation" or "integration" etc.?
> 
> IMHO you can just make an executive decision and pick one of those
> three. None of them are terrible, any would be a valid choice.

Ok, I think I'll go with "functional" since that still seems to be the best 
fit to me.

  Thomas