Re: [RFC PATCH] fstests: add basic json output support

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2022/12/16 00:42, David Disseldorp wrote:
> Hi Qu,
> 
> On Thu, 15 Dec 2022 19:41:13 +0800, Qu Wenruo wrote:
> 
>> Although the current result files "check.log" and "check.time" is enough
>> for human to read, it's not that easy to parse.
> 
> Have you looked at the existing junit XML based report types, available
> via "check -R xunit ..."? junit is standardized, parsable and supported
> by tools such as:
> - https://docs.gitlab.com/ee/ci/testing/unit_test_reports.html
> - https://github.com/weiwei/junitparser
> - https://ddiss.github.io/online-junit-parser/
> - https://plugins.jenkins.io/junit/

Oh, that's way better, the existing reporting facility is really what I 
need.

> 
>> Thus this patch will introduce a json output to "$RESULT_BASE/check.json".
>>
>> The example output would look like this:
>>
>>    {
>>        "section": "(none)",
>>        "fstype": "btrfs",
>>        "start_time": 1671103264,
>>        "arch": "x86_64",
>>        "kernel": "6.1.0-rc8-custom+",
>>        "results": [
>>            {
>>                "testcase": "btrfs/001",
>>                "status": "pass",
>>                "start_time": 1671103264,
>>                "end_time": 1671103266
>>            },
>>            {
>>                "testcase": "btrfs/006",
>>                "status": "pass",
>>                "start_time": 1671103266,
>>                "end_time": 1671103268
>>            },
>>            {
>>                "testcase": "btrfs/007",
>>                "status": "pass",
>>                "start_time": 1671103268,
>>                "end_time": 1671103271
>>            }
>>        ]
>>    }
>>
>> Which should make later parsing much easier.
>>
>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>> ---
>> Reason for RFC:
>>
>> - Not crash safe
>>    If one test case caused a crash, the "check.json" file will be an
>>    invalid one, missing the closing "] }" string.
>>
>> - Is json really a good choice?
>>    It may be much easier to convert to a web page, but we will still
>>    need to parse and handle the result using another languages anyway,
>>    like to determine a regression.
> 
> I'm not opposed to adding an extra json report type, but I really think
> it should be plumbed into the existing common/report API.
> 
>>    Another alternative is .csv, and it can be much easier to handle.
>>    (pure "echo >> $output", no need to handle the comma rule).
>>    But for .csv, we may waste a lot of columes for things like "arch",
>>    "kernel", "section".
> 
> My preference for any new output formats, especially if they're intended
> for parsing, is that they're based on an existing standard/tool. E.g.
> https://testanything.org .

That's a much better unified protocol, I'll definitely look into it.

Thanks for all the feedbacks!
Qu

> 
> Cheers, David