Re: [PATCH v2 5/6] tracetool: add test suite for tracetool with reference output

[Markus Armbruster <armbru@redhat.com>]

Daniel P. Berrangé <berrange@redhat.com> writes:

> When reviewing tracetool patches it is often very unclear what the
> expected output will be for the generated backends. Compounding
> this is that a default build will only enable the 'log' trace
> backend, so developers won't see generated code for other backends
> without making a special effort. Some backends are also platform
> specific, so can't be enabled in QEMU builds, even though tracetool
> could generate the code.
>
> To address this, introduce a test suite for tracetool which is
> conceptually similar to the qapi-schema test. It is a simple
> python program that runs tracetool and compares the actual output
> to historical reference output kept in git. The test directly
> emits TAP format logs for ease of integration with meson.
>
> This can be run with
>
>   make check-tracetool
>
> to make it easier for developers changing generated output, the
> sample expected content can be auto-recreated
>
>   QEMU_TEST_REGENERATE=1 make check-tracetool

make check-qapi-schema uses QAPI_TEST_UPDATE for this.  Should we use a
single environment variable for this purpose?  I'd be fine with changing
QAPI_TEST_UPDATE.

> and the changes reviewed and added to the commit. This will also
> assist reviewers interpreting the change.
>
> Developers are reminded of this in the test output on failure:
>
>   $ make check-tracetool
>   1/6 qemu:tracetool / dtrace        OK              0.14s
>   2/6 qemu:tracetool / ftrace        FAIL            0.06s   exit status 1
>   ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
>   1..2
>   ok 1 - ftrace.c
>   #
>   not ok 1 - ftrace.h (set QEMU_TEST_REGENERATE=1 to recreate reference output if tracetool generator was intentionally changed)

Neat!  Nice to have in tests/qapi-schema/test-qapi.py, too.
Observation, not a demand.

>   ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
>
>   3/6 qemu:tracetool / log           OK              0.06s
>   4/6 qemu:tracetool / simple        OK              0.06s
>   5/6 qemu:tracetool / syslog        OK              0.06s
>   6/6 qemu:tracetool / ust           OK              0.11s
>
>   Summary of Failures:
>
>   2/6 qemu:tracetool / ftrace FAIL            0.06s   exit status 1
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

[...]

> diff --git a/MAINTAINERS b/MAINTAINERS
> index 28cea34271..c08c51f4c6 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -3556,6 +3556,7 @@ F: scripts/tracetool/
>  F: scripts/qemu-trace-stap*
>  F: docs/tools/qemu-trace-stap.rst
>  F: docs/devel/tracing.rst
> +F: tests/tracetool/
>  T: git https://github.com/stefanha/qemu.git tracing
>  
>  Simpletrace
> diff --git a/docs/devel/testing/main.rst b/docs/devel/testing/main.rst
> index 2b5cb0c148..11f05c0006 100644
> --- a/docs/devel/testing/main.rst
> +++ b/docs/devel/testing/main.rst
> @@ -178,6 +178,34 @@ parser (either fixing a bug or extending/modifying the syntax). To do this:
>  
>    ``qapi-schema += foo.json``
>  
> +.. _tracetool-tests:
> +
> +Tracetool tests
> +~~~~~~~~~~~~~~~
> +
> +The tracetool tests validate the generated source files used for defining
> +probes for various tracing backends and source formats. The test operates
> +by running the tracetool program against a sample trace-events file, and
> +comparing the generated output against known good reference output. The
> +tests can be run with:
> +
> +.. code::
> +
> +  make check-tracetool
> +
> +The reference output is stored in files under tests/tracetool, and when
> +the tracetool backend/format output is intentionally changed, the reference
> +files need to be updated. This can be automated by setting the
> +QEMU_TEST_REGENERATE=1 environment variable:
> +
> +.. code::
> +
> +   QEMU_TEST_REGENERATE=1 make check-tracetool
> +
> +The resulting changes must be reviewed by the author to ensure they match
> +the intended results, before adding the updated reference output to the
> +same commit that alters the generator code.

Section "QAPI schema tests" could use similar advice on how to update
reference output.  Observation, not a demand.

> +
>  check-block
>  ~~~~~~~~~~~
>  

[...]