Re: [RFC PATCH] fstests: add basic json output support

[David Disseldorp <ddiss@suse.de>]

Hi Qu,

On Thu, 15 Dec 2022 19:41:13 +0800, Qu Wenruo wrote:

> Although the current result files "check.log" and "check.time" is enough
> for human to read, it's not that easy to parse.

Have you looked at the existing junit XML based report types, available
via "check -R xunit ..."? junit is standardized, parsable and supported
by tools such as:
- https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html
- https://github.com/weiwei/junitparser
- https://ddiss.github.io/online-junit-parser/
- https://plugins.jenkins.io/junit/

> Thus this patch will introduce a json output to "$RESULT_BASE/check.json".
> 
> The example output would look like this:
> 
>   {
>       "section": "(none)",
>       "fstype": "btrfs",
>       "start_time": 1671103264,
>       "arch": "x86_64",
>       "kernel": "6.1.0-rc8-custom+",
>       "results": [
>           {
>               "testcase": "btrfs/001",
>               "status": "pass",
>               "start_time": 1671103264,
>               "end_time": 1671103266
>           },
>           {
>               "testcase": "btrfs/006",
>               "status": "pass",
>               "start_time": 1671103266,
>               "end_time": 1671103268
>           },
>           {
>               "testcase": "btrfs/007",
>               "status": "pass",
>               "start_time": 1671103268,
>               "end_time": 1671103271
>           }
>       ]
>   }
> 
> Which should make later parsing much easier.
> 
> Signed-off-by: Qu Wenruo <wqu@suse.com>
> ---
> Reason for RFC:
> 
> - Not crash safe
>   If one test case caused a crash, the "check.json" file will be an
>   invalid one, missing the closing "] }" string.
> 
> - Is json really a good choice?
>   It may be much easier to convert to a web page, but we will still
>   need to parse and handle the result using another languages anyway,
>   like to determine a regression.

I'm not opposed to adding an extra json report type, but I really think
it should be plumbed into the existing common/report API.

>   Another alternative is .csv, and it can be much easier to handle.
>   (pure "echo >> $output", no need to handle the comma rule).
>   But for .csv, we may waste a lot of columes for things like "arch",
>   "kernel", "section".

My preference for any new output formats, especially if they're intended
for parsing, is that they're based on an existing standard/tool. E.g.
https://testanything.org .

Cheers, David