Re: [PATCH 2/8] report: derive an xml schema for the xunit report

Qu Wenruo
2022-12-20T02:18:48Z

On 2022/12/20 08:01, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
>
> The "xunit" report format emits an XML document that more or less
> follows the junit xml schema. However, there are two major exceptions:
>
> 1. fstests does not emit an @errors attribute on the testsuite element
> because we don't have the concept of unanticipated errors such as
> "unchecked throwables".
>
> 2. The system-out/system-err elements sound like they belong under the
> testcase element, though the schema itself imprecisely says "while the
> test was executed". The schema puts them under the top-level testsuite
> element, but we put them under the testcase element.
>
> Define an xml schema for the xunit report format, and update the xml
> headers to link to the schema file. This enables consumers of the
> reports to check mechanically that the incoming document follows the
> format.

One thing is, does the official XMLs use tabs as indents?

We got some lines definitely too long for human to read.
Any way to make them a little better?

But overall, it really defines a good standard for us to follow.
This is definitely a good start.

>
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> ---
[...]
> + <xs:choice minOccurs="0" maxOccurs="2">

For this, I prefer maxOccurs to be at least 3.

We have 3 different possible outputs:

- $seqnum.out.bad
- $seqnum.full
- $seqnum.dmesg

[...]

> + </xs:annotation>
> + <xs:simpleType>
> + <xs:restriction base="pre-string">
> + <xs:whiteSpace value="preserve"/>
> + </xs:restriction>
> + </xs:simpleType>
> + </xs:element>
> + <xs:element name="system-err" minOccurs="0" maxOccurs="1">
> + <xs:annotation>
> + <xs:documentation xml:lang="en">Data that was written to standard error while the test was executed</xs:documentation>

We don't use stderr, but $seqnum.full and $seqnum.dmesg.

Or can we just rename the "system-out" and "system-err" to something
fstests specific? E.g.

- system-output
- system-full
- system-dmesg

Or the system-err/out thing is mostly to keep the compatibility?
If so, I'd prefer some properties to make it explicit which output
represents which fstests specific output.

> + </xs:annotation>
> + <xs:simpleType>
> + <xs:restriction base="pre-string">
> + <xs:whiteSpace value="preserve"/>
> + </xs:restriction>
> + </xs:simpleType>
> + </xs:element>
> + </xs:choice>
> + </xs:sequence>
> + <xs:attribute name="name" type="xs:token" use="required">
> + <xs:annotation>
> + <xs:documentation xml:lang="en">Name of the test method</xs:documentation>

Can we update the description to something more fstests specific, better
with an example?
Like "test case number, e.g. generic/001".

This can apply to most description copied from the JUnit doc.

[...]
> + <xs:attribute name="timestamp" type="ISO8601_DATETIME_PATTERN" use="required">
> + <xs:annotation>
> + <xs:documentation xml:lang="en">when the test was executed. Timezone may not be specified.</xs:documentation>
> + </xs:annotation>

This means the start time, thus all our existing timestamp is not
following the spec already.

Thanks,
Qu