Re: [PATCH v4 35/35] docs/devel/testing: Add documentation for functional tests

[Thomas Huth <thuth@redhat.com>]

On 29/08/2024 12.34, Daniel P. Berrangé wrote:
> On Wed, Aug 21, 2024 at 10:27:36AM +0200, Thomas Huth wrote:
>> Document the new functional testing framework. The text is originally
>> based on the Avocado documentation, but heavily modified to match the
>> new framework.
>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>   docs/devel/testing/functional.rst | 269 ++++++++++++++++++++++++++++++
>>   docs/devel/testing/index.rst      |   1 +
>>   docs/devel/testing/main.rst       |  12 ++
>>   3 files changed, 282 insertions(+)
>>   create mode 100644 docs/devel/testing/functional.rst
> 
> Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
> 
>> +The tests should be written in the style of the Python `unittest`_
>> +framework, using stdio for the TAP protocol. The folder
>> +``tests/functional/qemu_test`` provides classes (e.g. the ``QemuBaseTest``
>> +and the ``QemuSystemTest`` classes) and utility functions that help
>> +to get your test into the right shape.
> 
> One gotcha when using TAP protocol is that you can't just spew
> debug info to stdout/stderr. Each line of debug info needs to
> be prefixed with '#' so it is interpreted as diagnostic output.

Actually, that's the great thing about pycotap (in comparison to other 
Python TAP implementations that I've seen), it helps you to get this right: 
By instantiating the TAPTestRunner like this:

     tr = pycotap.TAPTestRunner(message_log = pycotap.LogMode.LogToError,
                                test_output_log = pycotap.LogMode.LogToError)

The stdio output gets redirected to stderr. And the meson test runner is 
fine by collecting the error messages from stderr and showing them to the 
user in the right way in case the test failed (and puts them into the log 
file if the test succeeded).

But yes, the doc text is a little bit sparse on this topic, I'll try to 
elaborate in the next iteration.

  Thomas