Re: [KTAP V2 PATCH v3] ktap_v2: add test metadata

Kees Cook
2024-02-15T17:38:09Z

On Thu, Feb 15, 2024 at 05:18:59PM +0000, Rae Moar wrote:
> Add specification for test metadata to the KTAP v2 spec.
>
> KTAP v1 only specifies the output format of very basic test information:
> test result and test name. Any additional test information either gets
> added to general diagnostic data or is not included in the output at all.
>
> The purpose of KTAP metadata is to create a framework to include and
> easily identify additional important test information in KTAP.
>
> KTAP metadata could include any test information that is pertinent for
> user interaction before or after the running of the test. For example,
> the test file path or the test speed.
>
> Since this includes a large variety of information, this specification
> will recognize notable types of KTAP metadata to ensure consistent format
> across test frameworks. See the full list of types in the specification.
>
> Example of KTAP Metadata:
>
> KTAP version 2
> #:ktap_test: main
> #:ktap_arch: uml
> 1..1
> KTAP version 2
> #:ktap_test: suite_1
> #:ktap_subsystem: example
> #:ktap_test_file: lib/test.c
> 1..2
> ok 1 test_1
> #:ktap_test: test_2
> #:ktap_speed: very_slow
> # test_2 has begun
> #:custom_is_flaky: true
> ok 2 test_2
> # suite_1 has passed
> ok 1 suite_1
>
> The changes to the KTAP specification outline the format, location, and
> different types of metadata.
>
> Reviewed-by: David Gow <davidgow@google.com>
> Signed-off-by: Rae Moar <rmoar@google.com>

Thanks for the updates!

Reviewed-by: Kees Cook <keescook@chromium.org>

We'll need to update the test runners to adjust their outputs...

--
Kees Cook