* [PATCH 1/8] check: generate section reports between tests
2022-12-20 0:01 [PATCHSET RFC 0/8] fstests: improve junit xml reporting Darrick J. Wong
@ 2022-12-20 0:01 ` Darrick J. Wong
2022-12-20 1:14 ` Qu Wenruo
2022-12-20 3:16 ` Theodore Ts'o
2022-12-20 0:01 ` [PATCH 2/8] report: derive an xml schema for the xunit report Darrick J. Wong
` (6 subsequent siblings)
7 siblings, 2 replies; 19+ messages in thread
From: Darrick J. Wong @ 2022-12-20 0:01 UTC (permalink / raw)
To: djwong, zlang; +Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs
From: Darrick J. Wong <djwong@kernel.org>
Generate the section report between tests so that the summary report
always reflects the outcome of the most recent test. Two usecases are
envisioned here -- if a cluster-based test runner anticipates that the
testrun could crash the VM, they can set REPORT_DIR to (say) an NFS
mount to preserve the intermediate results. If the VM does indeed
crash, the scheduler can examine the state of the crashed VM and move
the tests to another VM. The second usecase is a reporting agent that
runs in the VM to upload live results to a test dashboard.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
check | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/check b/check
index 1ff0f44af1..70a0b537b1 100755
--- a/check
+++ b/check
@@ -842,6 +842,16 @@ function run_section()
fi
seqres="$REPORT_DIR/$seqnum"
+ # Generate the entire section report with whatever test results
+ # we have so far. Leave the $sect_time parameter empty so that
+ # it's a little more obvious that this test run is incomplete.
+ if $do_report; then
+ local sect_now=`_wallclock`
+ _make_section_report "$section" "${#try[*]}" \
+ "${#bad[*]}" "${#notrun[*]}" \
+ "" &> /dev/null
+ fi
+
mkdir -p $RESULT_DIR
rm -f ${RESULT_DIR}/require_scratch*
rm -f ${RESULT_DIR}/require_test*
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 1/8] check: generate section reports between tests
2022-12-20 0:01 ` [PATCH 1/8] check: generate section reports between tests Darrick J. Wong
@ 2022-12-20 1:14 ` Qu Wenruo
2023-02-14 18:46 ` Darrick J. Wong
2022-12-20 3:16 ` Theodore Ts'o
1 sibling, 1 reply; 19+ messages in thread
From: Qu Wenruo @ 2022-12-20 1:14 UTC (permalink / raw)
To: Darrick J. Wong, zlang; +Cc: linux-xfs, fstests, guan, leah.rumancik
On 2022/12/20 08:01, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
>
> Generate the section report between tests so that the summary report
> always reflects the outcome of the most recent test. Two usecases are
> envisioned here -- if a cluster-based test runner anticipates that the
> testrun could crash the VM, they can set REPORT_DIR to (say) an NFS
> mount to preserve the intermediate results. If the VM does indeed
> crash, the scheduler can examine the state of the crashed VM and move
> the tests to another VM. The second usecase is a reporting agent that
> runs in the VM to upload live results to a test dashboard.
>
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> ---
> check | 10 ++++++++++
> 1 file changed, 10 insertions(+)
>
>
> diff --git a/check b/check
> index 1ff0f44af1..70a0b537b1 100755
> --- a/check
> +++ b/check
> @@ -842,6 +842,16 @@ function run_section()
> fi
> seqres="$REPORT_DIR/$seqnum"
>
> + # Generate the entire section report with whatever test results
> + # we have so far. Leave the $sect_time parameter empty so that
> + # it's a little more obvious that this test run is incomplete.
> + if $do_report; then
> + local sect_now=`_wallclock`
> + _make_section_report "$section" "${#try[*]}" \
> + "${#bad[*]}" "${#notrun[*]}" \
> + "" &> /dev/null
> + fi
> +
In fact that's exactly what I just wanted to do next...
However one problem I can't work around is the timestamp.
Currently our timestamp is the finish time, with this change our
timestamp is the start time.
I'm not sure if there is any strong definition on the timestamp, but
personally speaking, timestamp for the start time makes more sense to me.
But there may be some users relying on the timestamp to be the ending time?
Thanks,
Qu
> mkdir -p $RESULT_DIR
> rm -f ${RESULT_DIR}/require_scratch*
> rm -f ${RESULT_DIR}/require_test*
>
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH 1/8] check: generate section reports between tests
2022-12-20 1:14 ` Qu Wenruo
@ 2023-02-14 18:46 ` Darrick J. Wong
2023-02-15 2:06 ` Qu Wenruo
0 siblings, 1 reply; 19+ messages in thread
From: Darrick J. Wong @ 2023-02-14 18:46 UTC (permalink / raw)
To: Qu Wenruo; +Cc: zlang, linux-xfs, fstests, guan, leah.rumancik
On Tue, Dec 20, 2022 at 09:14:07AM +0800, Qu Wenruo wrote:
>
>
> On 2022/12/20 08:01, Darrick J. Wong wrote:
> > From: Darrick J. Wong <djwong@kernel.org>
> >
> > Generate the section report between tests so that the summary report
> > always reflects the outcome of the most recent test. Two usecases are
> > envisioned here -- if a cluster-based test runner anticipates that the
> > testrun could crash the VM, they can set REPORT_DIR to (say) an NFS
> > mount to preserve the intermediate results. If the VM does indeed
> > crash, the scheduler can examine the state of the crashed VM and move
> > the tests to another VM. The second usecase is a reporting agent that
> > runs in the VM to upload live results to a test dashboard.
> >
> > Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> > ---
> > check | 10 ++++++++++
> > 1 file changed, 10 insertions(+)
> >
> >
> > diff --git a/check b/check
> > index 1ff0f44af1..70a0b537b1 100755
> > --- a/check
> > +++ b/check
> > @@ -842,6 +842,16 @@ function run_section()
> > fi
> > seqres="$REPORT_DIR/$seqnum"
> > + # Generate the entire section report with whatever test results
> > + # we have so far. Leave the $sect_time parameter empty so that
> > + # it's a little more obvious that this test run is incomplete.
> > + if $do_report; then
> > + local sect_now=`_wallclock`
> > + _make_section_report "$section" "${#try[*]}" \
> > + "${#bad[*]}" "${#notrun[*]}" \
> > + "" &> /dev/null
> > + fi
> > +
>
> In fact that's exactly what I just wanted to do next...
>
> However one problem I can't work around is the timestamp.
>
> Currently our timestamp is the finish time, with this change our timestamp
> is the start time.
Er... what do you mean exactly by start and finish time? The most
recent test? The entire run?
From check:
if [ -w /dev/kmsg ]; then
export date_time=`date +"%F %T"`
echo "run fstests $seqnum at $date_time" > /dev/kmsg
...
fi
From common/report:
if [ -z "$date_time" ]; then
date_time=$(date +"%F %T")
fi
...timestamp="${date_time/ /T}">
The "date_time" variable can be set by ./check just prior to starting
each test, but only if /dev/kmsg is a writable file. If it's set, then
the timestamp in the xml file reflects the start time of the most recent
test.
If date_time is not set, then the timestamp in the xml file records the
point in time when the report is generated... which I suppose is a good
enough estimate if we failed to record date_time when the test was
started.
> I'm not sure if there is any strong definition on the timestamp, but
> personally speaking, timestamp for the start time makes more sense to me.
> But there may be some users relying on the timestamp to be the ending time?
The junit xml schema says that the timestamp should be "when the test
was executed". I think ./check ought to be setting date_time regardless
of /dev/kmsg. ATM my test dashboard computes the difference between now
and the timestamp= attribute of the last report received to decide if it
should flag a test VM as possibly stalled.
Though to undercut everything I just wrote, the junit xml schema defines
testsuite and testcase as distinct entities and then refers to "the
test".
Given that fstests doesn't strictly follow that schema anyway, I guess
we can define the timestamp as we want, or add more timestamp
attributes?
<testsuite suite_timestamp="..." timestamp="..." report_timestamp="..."...>
--D
> Thanks,
> Qu
>
> > mkdir -p $RESULT_DIR
> > rm -f ${RESULT_DIR}/require_scratch*
> > rm -f ${RESULT_DIR}/require_test*
> >
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [PATCH 1/8] check: generate section reports between tests
2023-02-14 18:46 ` Darrick J. Wong
@ 2023-02-15 2:06 ` Qu Wenruo
0 siblings, 0 replies; 19+ messages in thread
From: Qu Wenruo @ 2023-02-15 2:06 UTC (permalink / raw)
To: Darrick J. Wong; +Cc: zlang, linux-xfs, fstests, guan, leah.rumancik
On 2023/2/15 02:46, Darrick J. Wong wrote:
> On Tue, Dec 20, 2022 at 09:14:07AM +0800, Qu Wenruo wrote:
>>
>>
>> On 2022/12/20 08:01, Darrick J. Wong wrote:
>>> From: Darrick J. Wong <djwong@kernel.org>
>>>
>>> Generate the section report between tests so that the summary report
>>> always reflects the outcome of the most recent test. Two usecases are
>>> envisioned here -- if a cluster-based test runner anticipates that the
>>> testrun could crash the VM, they can set REPORT_DIR to (say) an NFS
>>> mount to preserve the intermediate results. If the VM does indeed
>>> crash, the scheduler can examine the state of the crashed VM and move
>>> the tests to another VM. The second usecase is a reporting agent that
>>> runs in the VM to upload live results to a test dashboard.
>>>
>>> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
>>> ---
>>> check | 10 ++++++++++
>>> 1 file changed, 10 insertions(+)
>>>
>>>
>>> diff --git a/check b/check
>>> index 1ff0f44af1..70a0b537b1 100755
>>> --- a/check
>>> +++ b/check
>>> @@ -842,6 +842,16 @@ function run_section()
>>> fi
>>> seqres="$REPORT_DIR/$seqnum"
>>> + # Generate the entire section report with whatever test results
>>> + # we have so far. Leave the $sect_time parameter empty so that
>>> + # it's a little more obvious that this test run is incomplete.
>>> + if $do_report; then
>>> + local sect_now=`_wallclock`
>>> + _make_section_report "$section" "${#try[*]}" \
>>> + "${#bad[*]}" "${#notrun[*]}" \
>>> + "" &> /dev/null
>>> + fi
>>> +
>>
>> In fact that's exactly what I just wanted to do next...
>>
>> However one problem I can't work around is the timestamp.
>>
>> Currently our timestamp is the finish time, with this change our timestamp
>> is the start time.
>
> Er... what do you mean exactly by start and finish time? The most
> recent test? The entire run?
Personally speaking, I'd prefer start/finish time of the whole
suite/section at least.
>
> From check:
>
> if [ -w /dev/kmsg ]; then
> export date_time=`date +"%F %T"`
> echo "run fstests $seqnum at $date_time" > /dev/kmsg
> ...
> fi
>
> From common/report:
>
> if [ -z "$date_time" ]; then
> date_time=$(date +"%F %T")
> fi
>
> ...timestamp="${date_time/ /T}">
>
> The "date_time" variable can be set by ./check just prior to starting
> each test, but only if /dev/kmsg is a writable file. If it's set, then
> the timestamp in the xml file reflects the start time of the most recent
> test.
>
> If date_time is not set, then the timestamp in the xml file records the
> point in time when the report is generated... which I suppose is a good
> enough estimate if we failed to record date_time when the test was
> started. >
>> I'm not sure if there is any strong definition on the timestamp, but
>> personally speaking, timestamp for the start time makes more sense to me.
>> But there may be some users relying on the timestamp to be the ending time?
>
> The junit xml schema says that the timestamp should be "when the test
> was executed". I think ./check ought to be setting date_time regardless
> of /dev/kmsg. ATM my test dashboard computes the difference between now
> and the timestamp= attribute of the last report received to decide if it
> should flag a test VM as possibly stalled.
>
> Though to undercut everything I just wrote, the junit xml schema defines
> testsuite and testcase as distinct entities and then refers to "the
> test".
>
> Given that fstests doesn't strictly follow that schema anyway, I guess
> we can define the timestamp as we want, or add more timestamp
> attributes?
>
> <testsuite suite_timestamp="..." timestamp="..." report_timestamp="..."...>
OK, then this looks much better.
Thanks,
Qu
>
> --D
>
>> Thanks,
>> Qu
>>
>>> mkdir -p $RESULT_DIR
>>> rm -f ${RESULT_DIR}/require_scratch*
>>> rm -f ${RESULT_DIR}/require_test*
>>>
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 1/8] check: generate section reports between tests
2022-12-20 0:01 ` [PATCH 1/8] check: generate section reports between tests Darrick J. Wong
2022-12-20 1:14 ` Qu Wenruo
@ 2022-12-20 3:16 ` Theodore Ts'o
2022-12-20 18:18 ` Leah Rumancik
1 sibling, 1 reply; 19+ messages in thread
From: Theodore Ts'o @ 2022-12-20 3:16 UTC (permalink / raw)
To: Darrick J. Wong
Cc: zlang, linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs
On Mon, Dec 19, 2022 at 04:01:09PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
>
> Generate the section report between tests so that the summary report
> always reflects the outcome of the most recent test. Two usecases are
> envisioned here -- if a cluster-based test runner anticipates that the
> testrun could crash the VM, they can set REPORT_DIR to (say) an NFS
> mount to preserve the intermediate results. If the VM does indeed
> crash, the scheduler can examine the state of the crashed VM and move
> the tests to another VM. The second usecase is a reporting agent that
> runs in the VM to upload live results to a test dashboard.
Leah has been working on adding crash recovery for gce-xfstests.
It'll be interesting to see how her work dovetails with your patches.
The basic design we've worked out works by having the test framework
recognize whether the VM had been had been previously been running
tests. We keep track of the last test that was run by hooking into
$LOGGER_PROG. We then use a python script[1] to append to the xunit file
a test result for the test that was running at the time of the crash,
and we set the test result to "error", and then we resume running
tests from where we had left off.
[1] https://github.com/lrumancik/xfstests-bld/blob/ltm-auto-resume-new/test-appliance/files/usr/local/bin/add_error_xunit
To deal with cases where the kernel has deadlocked, when the test VM
is launched by the LTM server, the LTM server will monitor the test
VM, if the LTM server notices that the test VM has failed to make
forward progress within a set time, it will force the test VM to
reboot, at which point the recovery process described above kicks in.
Eventually, we'll have the LTM server examine the serial console of
the test VM, looking for indications of kernel panics and RCU / soft
lockup warnings, so we can more quickly force a reboot when the system
under test is clearly unhappy.
The advantage of this design is that it doesen't require using NFS to
store the results, and in theory we don't even need to use a separate
monitoring VM; we could just use a software and kernel watchdogs to
notice when the tests have stopped making forward progress.
- Ted
P.S. We're not using section reporting since we generally use launch
separate VM's for each "section" so we can speed up the test run time
by sharding across those VM's. And then we have the LTM server merge
the results together into a single test run report.
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 2/8] report: derive an xml schema for the xunit report
2022-12-20 0:01 [PATCHSET RFC 0/8] fstests: improve junit xml reporting Darrick J. Wong
2022-12-20 0:01 ` [PATCH 1/8] check: generate section reports between tests Darrick J. Wong
@ 2022-12-20 0:01 ` Darrick J. Wong
2022-12-20 2:18 ` Qu Wenruo
2022-12-20 0:01 ` [PATCH 3/8] report: capture the time zone in the test report timestamp Darrick J. Wong
` (5 subsequent siblings)
7 siblings, 1 reply; 19+ messages in thread
From: Darrick J. Wong @ 2022-12-20 0:01 UTC (permalink / raw)
To: djwong, zlang; +Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs
From: Darrick J. Wong <djwong@kernel.org>
The "xunit" report format emits an XML document that more or less
follows the junit xml schema. However, there are two major exceptions:
1. fstests does not emit an @errors attribute on the testsuite element
because we don't have the concept of unanticipated errors such as
"unchecked throwables".
2. The system-out/system-err elements sound like they belong under the
testcase element, though the schema itself imprecisely says "while the
test was executed". The schema puts them under the top-level testsuite
element, but we put them under the testcase element.
Define an xml schema for the xunit report format, and update the xml
headers to link to the schema file. This enables consumers of the
reports to check mechanically that the incoming document follows the
format.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 15 +++-
doc/xunit.xsd | 226 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 238 insertions(+), 3 deletions(-)
create mode 100644 doc/xunit.xsd
diff --git a/common/report b/common/report
index 64f9c86648..1d84650270 100644
--- a/common/report
+++ b/common/report
@@ -48,9 +48,18 @@ _xunit_make_section_report()
if [ -z "$date_time" ]; then
date_time=$(date +"%F %T")
fi
- local stats="failures=\"$bad_count\" skipped=\"$notrun_count\" tests=\"$tests_count\" time=\"$sect_time\""
- local hw_info="hostname=\"$HOST\" timestamp=\"${date_time/ /T}\" "
- echo "<testsuite name=\"xfstests\" $stats $hw_info >" >> $REPORT_DIR/result.xml
+
+ local fstests_ns="https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git"
+ cat >> $REPORT_DIR/result.xml << ENDL
+<testsuite
+ xmlns="$fstests_ns"
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+ xsi:schemaLocation="$fstests_ns $fstests_ns/tree/doc/xunit.xsd"
+
+ name="xfstests"
+ failures="$bad_count" skipped="$notrun_count" tests="$tests_count" time="$sect_time"
+ hostname="$HOST" timestamp="${date_time/ /T}">
+ENDL
# Properties
echo -e "\t<properties>" >> $REPORT_DIR/result.xml
diff --git a/doc/xunit.xsd b/doc/xunit.xsd
new file mode 100644
index 0000000000..0aef8a9839
--- /dev/null
+++ b/doc/xunit.xsd
@@ -0,0 +1,226 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<xs:schema
+ xmlns="https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git"
+ xmlns:xs="http://www.w3.org/2001/XMLSchema"
+
+ targetNamespace="https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git"
+ elementFormDefault="qualified"
+ attributeFormDefault="unqualified">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">fstests xunit test result schema, derived from https://github.com/windyroad/JUnit-Schema</xs:documentation>
+ </xs:annotation>
+ <xs:element name="testsuite" type="testsuite"/>
+ <xs:simpleType name="ISO8601_DATETIME_PATTERN">
+ <xs:restriction base="xs:dateTime">
+ <xs:pattern value="[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}"/>
+ </xs:restriction>
+ </xs:simpleType>
+ <xs:element name="testsuites">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Contains an aggregation of testsuite results</xs:documentation>
+ </xs:annotation>
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="testsuite" minOccurs="0" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:complexContent>
+ <xs:extension base="testsuite">
+ <xs:attribute name="package" type="xs:token" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Derived from testsuite/@name in the non-aggregated documents</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="id" type="xs:int" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Starts at '0' for the first testsuite and is incremented by 1 for each following testsuite</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ </xs:extension>
+ </xs:complexContent>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ <xs:complexType name="testsuite">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Contains the results of executing a testsuite</xs:documentation>
+ </xs:annotation>
+ <xs:sequence>
+ <xs:element name="properties">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Properties (e.g., environment settings) set during test execution</xs:documentation>
+ </xs:annotation>
+ <xs:complexType>
+ <xs:sequence>
+ <xs:element name="property" minOccurs="0" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:attribute name="name" use="required">
+ <xs:simpleType>
+ <xs:restriction base="xs:token">
+ <xs:minLength value="1"/>
+ </xs:restriction>
+ </xs:simpleType>
+ </xs:attribute>
+ <xs:attribute name="value" type="xs:string" use="required"/>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ </xs:complexType>
+ </xs:element>
+ <xs:element name="testcase" minOccurs="0" maxOccurs="unbounded">
+ <xs:complexType>
+ <xs:sequence>
+ <xs:choice minOccurs="0" maxOccurs="1">
+ <xs:element name="skipped" minOccurs="0" maxOccurs="1">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Indicates that the test was skipped.</xs:documentation>
+ </xs:annotation>
+ <xs:complexType>
+ <xs:simpleContent>
+ <xs:extension base="pre-string">
+ <xs:attribute name="message" type="xs:string">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The message specifying why the test case was skipped</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ </xs:extension>
+ </xs:simpleContent>
+ </xs:complexType>
+ </xs:element>
+ <xs:element name="error" minOccurs="0" maxOccurs="1">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Indicates that the test errored. An errored test is one that had an unanticipated problem. e.g., an unchecked throwable; or a problem with the implementation of the test. Contains as a text node relevant data for the error, e.g., a stack trace</xs:documentation>
+ </xs:annotation>
+ <xs:complexType>
+ <xs:simpleContent>
+ <xs:extension base="pre-string">
+ <xs:attribute name="message" type="xs:string">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The error message. e.g., if a java exception is thrown, the return value of getMessage()</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="type" type="xs:string" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The type of error that occured. e.g., if a java execption is thrown the full class name of the exception.</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ </xs:extension>
+ </xs:simpleContent>
+ </xs:complexType>
+ </xs:element>
+ <xs:element name="failure">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Indicates that the test failed. A failure is a test which the code has explicitly failed by using the mechanisms for that purpose. e.g., via an assertEquals. Contains as a text node relevant data for the failure, e.g., a stack trace</xs:documentation>
+ </xs:annotation>
+ <xs:complexType>
+ <xs:simpleContent>
+ <xs:extension base="pre-string">
+ <xs:attribute name="message" type="xs:string">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The message specified in the assert</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="type" type="xs:string" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The type of the assert.</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ </xs:extension>
+ </xs:simpleContent>
+ </xs:complexType>
+ </xs:element>
+ </xs:choice>
+ <xs:choice minOccurs="0" maxOccurs="2">
+ <xs:element name="system-out" minOccurs="0" maxOccurs="1">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Data that was written to standard out while the test was executed</xs:documentation>
+ </xs:annotation>
+ <xs:simpleType>
+ <xs:restriction base="pre-string">
+ <xs:whiteSpace value="preserve"/>
+ </xs:restriction>
+ </xs:simpleType>
+ </xs:element>
+ <xs:element name="system-err" minOccurs="0" maxOccurs="1">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Data that was written to standard error while the test was executed</xs:documentation>
+ </xs:annotation>
+ <xs:simpleType>
+ <xs:restriction base="pre-string">
+ <xs:whiteSpace value="preserve"/>
+ </xs:restriction>
+ </xs:simpleType>
+ </xs:element>
+ </xs:choice>
+ </xs:sequence>
+ <xs:attribute name="name" type="xs:token" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Name of the test method</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="classname" type="xs:token" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Full class name for the class the test method is in.</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="time" type="xs:decimal" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Time taken (in seconds) to execute the test</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ </xs:complexType>
+ </xs:element>
+ </xs:sequence>
+ <xs:attribute name="name" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Full class name of the test for non-aggregated testsuite documents. Class name without the package for aggregated testsuites documents</xs:documentation>
+ </xs:annotation>
+ <xs:simpleType>
+ <xs:restriction base="xs:token">
+ <xs:minLength value="1"/>
+ </xs:restriction>
+ </xs:simpleType>
+ </xs:attribute>
+ <xs:attribute name="timestamp" type="ISO8601_DATETIME_PATTERN" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">when the test was executed. Timezone may not be specified.</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="hostname" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Host on which the tests were executed. 'localhost' should be used if the hostname cannot be determined.</xs:documentation>
+ </xs:annotation>
+ <xs:simpleType>
+ <xs:restriction base="xs:token">
+ <xs:minLength value="1"/>
+ </xs:restriction>
+ </xs:simpleType>
+ </xs:attribute>
+ <xs:attribute name="tests" type="xs:int" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The total number of tests in the suite</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="failures" type="xs:int" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The total number of tests in the suite that failed. A failure is a test which the code has explicitly failed by using the mechanisms for that purpose. e.g., via an assertEquals</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="skipped" type="xs:int" use="optional">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">The total number of ignored or skipped tests in the suite.</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ <xs:attribute name="time" type="xs:decimal" use="required">
+ <xs:annotation>
+ <xs:documentation xml:lang="en">Time taken (in seconds) to execute the tests in the suite</xs:documentation>
+ </xs:annotation>
+ </xs:attribute>
+ </xs:complexType>
+ <xs:simpleType name="pre-string">
+ <xs:restriction base="xs:string">
+ <xs:whiteSpace value="preserve"/>
+ </xs:restriction>
+ </xs:simpleType>
+</xs:schema>
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 2/8] report: derive an xml schema for the xunit report
2022-12-20 0:01 ` [PATCH 2/8] report: derive an xml schema for the xunit report Darrick J. Wong
@ 2022-12-20 2:18 ` Qu Wenruo
2023-02-14 18:54 ` Darrick J. Wong
0 siblings, 1 reply; 19+ messages in thread
From: Qu Wenruo @ 2022-12-20 2:18 UTC (permalink / raw)
To: Darrick J. Wong, zlang; +Cc: linux-xfs, fstests, guan, leah.rumancik
On 2022/12/20 08:01, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
>
> The "xunit" report format emits an XML document that more or less
> follows the junit xml schema. However, there are two major exceptions:
>
> 1. fstests does not emit an @errors attribute on the testsuite element
> because we don't have the concept of unanticipated errors such as
> "unchecked throwables".
>
> 2. The system-out/system-err elements sound like they belong under the
> testcase element, though the schema itself imprecisely says "while the
> test was executed". The schema puts them under the top-level testsuite
> element, but we put them under the testcase element.
>
> Define an xml schema for the xunit report format, and update the xml
> headers to link to the schema file. This enables consumers of the
> reports to check mechanically that the incoming document follows the
> format.
One thing is, does the official XMLs use tabs as indents?
We got some lines definitely too long for human to read.
Any way to make them a little better?
But overall, it really defines a good standard for us to follow.
This is definitely a good start.
>
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> ---
[...]
> + <xs:choice minOccurs="0" maxOccurs="2">
For this, I prefer maxOccurs to be at least 3.
We have 3 different possible outputs:
- $seqnum.out.bad
- $seqnum.full
- $seqnum.dmesg
[...]
> + </xs:annotation>
> + <xs:simpleType>
> + <xs:restriction base="pre-string">
> + <xs:whiteSpace value="preserve"/>
> + </xs:restriction>
> + </xs:simpleType>
> + </xs:element>
> + <xs:element name="system-err" minOccurs="0" maxOccurs="1">
> + <xs:annotation>
> + <xs:documentation xml:lang="en">Data that was written to standard error while the test was executed</xs:documentation>
We don't use stderr, but $seqnum.full and $seqnum.dmesg.
Or can we just rename the "system-out" and "system-err" to something
fstests specific? E.g.
- system-output
- system-full
- system-dmesg
Or the system-err/out thing is mostly to keep the compatibility?
If so, I'd prefer some properties to make it explicit which output
represents which fstests specific output.
> + </xs:annotation>
> + <xs:simpleType>
> + <xs:restriction base="pre-string">
> + <xs:whiteSpace value="preserve"/>
> + </xs:restriction>
> + </xs:simpleType>
> + </xs:element>
> + </xs:choice>
> + </xs:sequence>
> + <xs:attribute name="name" type="xs:token" use="required">
> + <xs:annotation>
> + <xs:documentation xml:lang="en">Name of the test method</xs:documentation>
Can we update the description to something more fstests specific, better
with an example?
Like "test case number, e.g. generic/001".
This can apply to most description copied from the JUnit doc.
[...]
> + <xs:attribute name="timestamp" type="ISO8601_DATETIME_PATTERN" use="required">
> + <xs:annotation>
> + <xs:documentation xml:lang="en">when the test was executed. Timezone may not be specified.</xs:documentation>
> + </xs:annotation>
This means the start time, thus all our existing timestamp is not
following the spec already.
Thanks,
Qu
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 2/8] report: derive an xml schema for the xunit report
2022-12-20 2:18 ` Qu Wenruo
@ 2023-02-14 18:54 ` Darrick J. Wong
0 siblings, 0 replies; 19+ messages in thread
From: Darrick J. Wong @ 2023-02-14 18:54 UTC (permalink / raw)
To: Qu Wenruo; +Cc: zlang, linux-xfs, fstests, guan, leah.rumancik
On Tue, Dec 20, 2022 at 10:18:18AM +0800, Qu Wenruo wrote:
>
>
> On 2022/12/20 08:01, Darrick J. Wong wrote:
> > From: Darrick J. Wong <djwong@kernel.org>
> >
> > The "xunit" report format emits an XML document that more or less
> > follows the junit xml schema. However, there are two major exceptions:
> >
> > 1. fstests does not emit an @errors attribute on the testsuite element
> > because we don't have the concept of unanticipated errors such as
> > "unchecked throwables".
> >
> > 2. The system-out/system-err elements sound like they belong under the
> > testcase element, though the schema itself imprecisely says "while the
> > test was executed". The schema puts them under the top-level testsuite
> > element, but we put them under the testcase element.
> >
> > Define an xml schema for the xunit report format, and update the xml
> > headers to link to the schema file. This enables consumers of the
> > reports to check mechanically that the incoming document follows the
> > format.
>
> One thing is, does the official XMLs use tabs as indents?
XML doesn't care one way or another:
https://www.w3.org/TR/xml/#sec-white-space
> We got some lines definitely too long for human to read.
> Any way to make them a little better?
Eh, I guess I could change them to two spaces, given the indentyness of
the schema.
> But overall, it really defines a good standard for us to follow.
> This is definitely a good start.
<nod>
> >
> > Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> > ---
> [...]
> > + <xs:choice minOccurs="0" maxOccurs="2">
>
> For this, I prefer maxOccurs to be at least 3.
>
> We have 3 different possible outputs:
>
> - $seqnum.out.bad
> - $seqnum.full
> - $seqnum.dmesg
>
> [...]
>
> > + </xs:annotation>
> > + <xs:simpleType>
> > + <xs:restriction base="pre-string">
> > + <xs:whiteSpace value="preserve"/>
> > + </xs:restriction>
> > + </xs:simpleType>
> > + </xs:element>
> > + <xs:element name="system-err" minOccurs="0" maxOccurs="1">
> > + <xs:annotation>
> > + <xs:documentation xml:lang="en">Data that was written to standard error while the test was executed</xs:documentation>
>
> We don't use stderr, but $seqnum.full and $seqnum.dmesg.
>
> Or can we just rename the "system-out" and "system-err" to something fstests
> specific? E.g.
>
> - system-output
> - system-full
> - system-dmesg
>
> Or the system-err/out thing is mostly to keep the compatibility?
> If so, I'd prefer some properties to make it explicit which output
> represents which fstests specific output.
I'll change those, since that's one of the major divergences from the
upstream junit xml schema. junit says system-out should capture the
stdout of the whole testsuite, not an individual testcase.
>
> > + </xs:annotation>
> > + <xs:simpleType>
> > + <xs:restriction base="pre-string">
> > + <xs:whiteSpace value="preserve"/>
> > + </xs:restriction>
> > + </xs:simpleType>
> > + </xs:element>
> > + </xs:choice>
> > + </xs:sequence>
> > + <xs:attribute name="name" type="xs:token" use="required">
> > + <xs:annotation>
> > + <xs:documentation xml:lang="en">Name of the test method</xs:documentation>
>
> Can we update the description to something more fstests specific, better
> with an example?
> Like "test case number, e.g. generic/001".
>
> This can apply to most description copied from the JUnit doc.
Ok.
> [...]
> > + <xs:attribute name="timestamp" type="ISO8601_DATETIME_PATTERN" use="required">
> > + <xs:annotation>
> > + <xs:documentation xml:lang="en">when the test was executed. Timezone may not be specified.</xs:documentation>
> > + </xs:annotation>
>
> This means the start time, thus all our existing timestamp is not following
> the spec already.
I wrote my comments about this part in the thread about patch 1, so
let's leave the discussion there.
--D
> Thanks,
> Qu
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 3/8] report: capture the time zone in the test report timestamp
2022-12-20 0:01 [PATCHSET RFC 0/8] fstests: improve junit xml reporting Darrick J. Wong
2022-12-20 0:01 ` [PATCH 1/8] check: generate section reports between tests Darrick J. Wong
2022-12-20 0:01 ` [PATCH 2/8] report: derive an xml schema for the xunit report Darrick J. Wong
@ 2022-12-20 0:01 ` Darrick J. Wong
2022-12-20 2:19 ` Qu Wenruo
2022-12-20 0:01 ` [PATCH 4/8] report: sort properties by name Darrick J. Wong
` (4 subsequent siblings)
7 siblings, 1 reply; 19+ messages in thread
From: Darrick J. Wong @ 2022-12-20 0:01 UTC (permalink / raw)
To: djwong, zlang; +Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs
From: Darrick J. Wong <djwong@kernel.org>
Make sure we put the time zone of the system running the test in the
timestamp that is recorded in the xunit report. `date "+%F %T"' reports
the local time zone, not UTC.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 9 ++++++---
doc/xunit.xsd | 2 +-
2 files changed, 7 insertions(+), 4 deletions(-)
diff --git a/common/report b/common/report
index 1d84650270..1817132d51 100644
--- a/common/report
+++ b/common/report
@@ -38,6 +38,7 @@ _xunit_make_section_report()
local bad_count="$3"
local notrun_count="$4"
local sect_time="$5"
+ local timestamp
if [ $sect_name == '-no-sections-' ]; then
sect_name='global'
@@ -45,8 +46,10 @@ _xunit_make_section_report()
local report=$tmp.report.xunit.$sect_name.xml
# Header
echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" > $REPORT_DIR/result.xml
- if [ -z "$date_time" ]; then
- date_time=$(date +"%F %T")
+ if [ -n "$date_time" ]; then
+ timestamp="$(date -Iseconds --date="$date_time")"
+ else
+ timestamp="$(date -Iseconds)"
fi
local fstests_ns="https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git"
@@ -58,7 +61,7 @@ _xunit_make_section_report()
name="xfstests"
failures="$bad_count" skipped="$notrun_count" tests="$tests_count" time="$sect_time"
- hostname="$HOST" timestamp="${date_time/ /T}">
+ hostname="$HOST" timestamp="$timestamp">
ENDL
# Properties
diff --git a/doc/xunit.xsd b/doc/xunit.xsd
index 0aef8a9839..67f586b816 100644
--- a/doc/xunit.xsd
+++ b/doc/xunit.xsd
@@ -12,7 +12,7 @@
<xs:element name="testsuite" type="testsuite"/>
<xs:simpleType name="ISO8601_DATETIME_PATTERN">
<xs:restriction base="xs:dateTime">
- <xs:pattern value="[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}"/>
+ <xs:pattern value="[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}[+-][0-9]{2}:[0-9]{2}"/>
</xs:restriction>
</xs:simpleType>
<xs:element name="testsuites">
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 3/8] report: capture the time zone in the test report timestamp
2022-12-20 0:01 ` [PATCH 3/8] report: capture the time zone in the test report timestamp Darrick J. Wong
@ 2022-12-20 2:19 ` Qu Wenruo
0 siblings, 0 replies; 19+ messages in thread
From: Qu Wenruo @ 2022-12-20 2:19 UTC (permalink / raw)
To: Darrick J. Wong, zlang; +Cc: linux-xfs, fstests, guan, leah.rumancik
On 2022/12/20 08:01, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
>
> Make sure we put the time zone of the system running the test in the
> timestamp that is recorded in the xunit report. `date "+%F %T"' reports
> the local time zone, not UTC.
>
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Qu Wenruo <wqu@suse.com>
Thanks,
Qu
> ---
> common/report | 9 ++++++---
> doc/xunit.xsd | 2 +-
> 2 files changed, 7 insertions(+), 4 deletions(-)
>
>
> diff --git a/common/report b/common/report
> index 1d84650270..1817132d51 100644
> --- a/common/report
> +++ b/common/report
> @@ -38,6 +38,7 @@ _xunit_make_section_report()
> local bad_count="$3"
> local notrun_count="$4"
> local sect_time="$5"
> + local timestamp
>
> if [ $sect_name == '-no-sections-' ]; then
> sect_name='global'
> @@ -45,8 +46,10 @@ _xunit_make_section_report()
> local report=$tmp.report.xunit.$sect_name.xml
> # Header
> echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>" > $REPORT_DIR/result.xml
> - if [ -z "$date_time" ]; then
> - date_time=$(date +"%F %T")
> + if [ -n "$date_time" ]; then
> + timestamp="$(date -Iseconds --date="$date_time")"
> + else
> + timestamp="$(date -Iseconds)"
> fi
>
> local fstests_ns="https://git.kernel.org/pub/scm/fs/xfs/xfstests-dev.git"
> @@ -58,7 +61,7 @@ _xunit_make_section_report()
>
> name="xfstests"
> failures="$bad_count" skipped="$notrun_count" tests="$tests_count" time="$sect_time"
> - hostname="$HOST" timestamp="${date_time/ /T}">
> + hostname="$HOST" timestamp="$timestamp">
> ENDL
>
> # Properties
> diff --git a/doc/xunit.xsd b/doc/xunit.xsd
> index 0aef8a9839..67f586b816 100644
> --- a/doc/xunit.xsd
> +++ b/doc/xunit.xsd
> @@ -12,7 +12,7 @@
> <xs:element name="testsuite" type="testsuite"/>
> <xs:simpleType name="ISO8601_DATETIME_PATTERN">
> <xs:restriction base="xs:dateTime">
> - <xs:pattern value="[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}"/>
> + <xs:pattern value="[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}[+-][0-9]{2}:[0-9]{2}"/>
> </xs:restriction>
> </xs:simpleType>
> <xs:element name="testsuites">
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 4/8] report: sort properties by name
2022-12-20 0:01 [PATCHSET RFC 0/8] fstests: improve junit xml reporting Darrick J. Wong
` (2 preceding siblings ...)
2022-12-20 0:01 ` [PATCH 3/8] report: capture the time zone in the test report timestamp Darrick J. Wong
@ 2022-12-20 0:01 ` Darrick J. Wong
2022-12-20 0:01 ` [PATCH 5/8] report: pass property value to _xunit_add_property Darrick J. Wong
` (3 subsequent siblings)
7 siblings, 0 replies; 19+ messages in thread
From: Darrick J. Wong @ 2022-12-20 0:01 UTC (permalink / raw)
To: djwong, zlang; +Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs
From: Darrick J. Wong <djwong@kernel.org>
When we're generating a junit xml report, always sort the properties by
name. This makes it easier for humans to find a particular property.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/common/report b/common/report
index 1817132d51..b415a2641d 100644
--- a/common/report
+++ b/common/report
@@ -27,7 +27,7 @@ _xunit_add_property()
local value="${!name}"
if [ ! -z "$value" ]; then
- echo -e "\t\t<property name=\"$name\" value=\"$value\"/>" >> $REPORT_DIR/result.xml
+ echo -e "\t\t<property name=\"$name\" value=\"$value\"/>"
fi
}
_xunit_make_section_report()
@@ -68,7 +68,7 @@ ENDL
echo -e "\t<properties>" >> $REPORT_DIR/result.xml
for p in "${REPORT_ENV_LIST[@]}"; do
_xunit_add_property "$p"
- done
+ done | sort >> $REPORT_DIR/result.xml
echo -e "\t</properties>" >> $REPORT_DIR/result.xml
if [ -f $report ]; then
cat $report >> $REPORT_DIR/result.xml
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 5/8] report: pass property value to _xunit_add_property
2022-12-20 0:01 [PATCHSET RFC 0/8] fstests: improve junit xml reporting Darrick J. Wong
` (3 preceding siblings ...)
2022-12-20 0:01 ` [PATCH 4/8] report: sort properties by name Darrick J. Wong
@ 2022-12-20 0:01 ` Darrick J. Wong
2022-12-20 0:01 ` [PATCH 6/8] report: collect basic information about a test run Darrick J. Wong
` (2 subsequent siblings)
7 siblings, 0 replies; 19+ messages in thread
From: Darrick J. Wong @ 2022-12-20 0:01 UTC (permalink / raw)
To: djwong, zlang; +Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs
From: Darrick J. Wong <djwong@kernel.org>
Change this helper to require the caller to pass the value as the second
parameter. This prepares us to start reporting a lot more information
about a test run, not all of which are encoded as bash variables.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/common/report b/common/report
index b415a2641d..642e0426a6 100644
--- a/common/report
+++ b/common/report
@@ -24,9 +24,9 @@ encode_xml()
_xunit_add_property()
{
local name="$1"
- local value="${!name}"
+ local value="$2"
- if [ ! -z "$value" ]; then
+ if [ -n "$value" ]; then
echo -e "\t\t<property name=\"$name\" value=\"$value\"/>"
fi
}
@@ -67,7 +67,7 @@ ENDL
# Properties
echo -e "\t<properties>" >> $REPORT_DIR/result.xml
for p in "${REPORT_ENV_LIST[@]}"; do
- _xunit_add_property "$p"
+ _xunit_add_property "$p" "${!p}"
done | sort >> $REPORT_DIR/result.xml
echo -e "\t</properties>" >> $REPORT_DIR/result.xml
if [ -f $report ]; then
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 6/8] report: collect basic information about a test run
2022-12-20 0:01 [PATCHSET RFC 0/8] fstests: improve junit xml reporting Darrick J. Wong
` (4 preceding siblings ...)
2022-12-20 0:01 ` [PATCH 5/8] report: pass property value to _xunit_add_property Darrick J. Wong
@ 2022-12-20 0:01 ` Darrick J. Wong
2022-12-20 3:29 ` Theodore Ts'o
2022-12-20 0:01 ` [PATCH 7/8] report: record xfs-specific " Darrick J. Wong
2022-12-20 0:01 ` [PATCH 8/8] report: record ext*-specific " Darrick J. Wong
7 siblings, 1 reply; 19+ messages in thread
From: Darrick J. Wong @ 2022-12-20 0:01 UTC (permalink / raw)
To: djwong, zlang; +Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs
From: Darrick J. Wong <djwong@kernel.org>
Record various generic information about an fstests run when generating
a junit xml report. This includes the cpu architecture, the kernel
revision, the CPU, memory, and numa node counts, and some information
about the block devices passed in.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 44 +++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 43 insertions(+), 1 deletion(-)
diff --git a/common/report b/common/report
index 642e0426a6..68646a7709 100644
--- a/common/report
+++ b/common/report
@@ -19,6 +19,42 @@ encode_xml()
-e 's/"/\"/g'
}
+# Fill out REPORT_VARS with information about the block device referred to by
+# the passed-in bash variable.
+__generate_blockdev_report_vars() {
+ local bdev_var="$1"
+ local bdev="${!bdev_var}"
+
+ test -z "$bdev" && return
+ test -b "$bdev" || return
+ local sysfs_bdev="$(_sysfs_dev "$bdev")"
+
+ REPORT_VARS["${bdev_var}_SIZE_KB"]="$(( "$(blockdev --getsz "$bdev")" / 2 ))"
+ REPORT_VARS["${bdev_var}_ROTATIONAL"]="$(cat "$sysfs_bdev/queue/rotational" 2>/dev/null)"
+ REPORT_VARS["${bdev_var}_DAX"]="$(cat "$sysfs_bdev/queue/dax" 2>/dev/null)"
+ REPORT_VARS["${bdev_var}_DISCARD"]="$(sed -e 's/[1-9][0-9]*/1/g' "$sysfs_bdev/queue/discard_max_bytes" 2>/dev/null)"
+ REPORT_VARS["${bdev_var}_WRITE_ZEROES"]="$(sed -e 's/[1-9][0-9]*/1/g' "$sysfs_bdev/queue/write_zeroes_max_bytes" 2>/dev/null)"
+ REPORT_VARS["${bdev_var}_PHYS_BLOCK_BYTES"]="$(cat "$sysfs_bdev/queue/physical_block_size" 2>/dev/null)"
+ REPORT_VARS["${bdev_var}_LBA_BYTES"]="$(cat "$sysfs_bdev/queue/logical_block_size" 2>/dev/null)"
+ REPORT_VARS["${bdev_var}_ZONES"]="$(cat "$sysfs_bdev/queue/nr_zones" 2>/dev/null)"
+}
+
+# Fill out REPORT_VARS with tidbits about our test runner configuration.
+# Caller is required to declare REPORT_VARS to be an associative array.
+__generate_report_vars() {
+ REPORT_VARS["ARCH"]="$(uname -m)"
+ REPORT_VARS["KERNEL"]="$(uname -r)"
+ REPORT_VARS["CPUS"]="$(getconf _NPROCESSORS_ONLN 2>/dev/null)"
+ REPORT_VARS["MEM_KB"]="$(grep MemTotal: /proc/meminfo | awk '{print $2}')"
+ REPORT_VARS["SWAP_KB"]="$(grep SwapTotal: /proc/meminfo | awk '{print $2}')"
+
+ test -e /sys/devices/system/node/possible && \
+ REPORT_VARS["NUMA_NODES"]="$(cat /sys/devices/system/node/possible 2>/dev/null)"
+
+ __generate_blockdev_report_vars "TEST_DEV"
+ __generate_blockdev_report_vars "SCRATCH_DEV"
+}
+
#
# Xunit format report functions
_xunit_add_property()
@@ -64,11 +100,17 @@ _xunit_make_section_report()
hostname="$HOST" timestamp="$timestamp">
ENDL
+ declare -A REPORT_VARS
+ __generate_report_vars
+
# Properties
echo -e "\t<properties>" >> $REPORT_DIR/result.xml
+ (for key in "${!REPORT_VARS[@]}"; do
+ _xunit_add_property "$key" "${REPORT_VARS["$key"]}"
+ done;
for p in "${REPORT_ENV_LIST[@]}"; do
_xunit_add_property "$p" "${!p}"
- done | sort >> $REPORT_DIR/result.xml
+ done) | sort >> $REPORT_DIR/result.xml
echo -e "\t</properties>" >> $REPORT_DIR/result.xml
if [ -f $report ]; then
cat $report >> $REPORT_DIR/result.xml
^ permalink raw reply related [flat|nested] 19+ messages in thread* Re: [PATCH 6/8] report: collect basic information about a test run
2022-12-20 0:01 ` [PATCH 6/8] report: collect basic information about a test run Darrick J. Wong
@ 2022-12-20 3:29 ` Theodore Ts'o
2023-02-14 18:59 ` Darrick J. Wong
0 siblings, 1 reply; 19+ messages in thread
From: Theodore Ts'o @ 2022-12-20 3:29 UTC (permalink / raw)
To: Darrick J. Wong
Cc: zlang, linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs
On Mon, Dec 19, 2022 at 04:01:37PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
>
> Record various generic information about an fstests run when generating
> a junit xml report. This includes the cpu architecture, the kernel
> revision, the CPU, memory, and numa node counts, and some information
> about the block devices passed in.
It would be nice if there was a way that the test runner could pass
information that would be added to the xunit properties. As I
mentioned in another e-mail, I currently do this via a post-processing
step which adds the properties to the junit xml file via a python
script. And there are a number of additional properties that are used
by my report generator[1] which takes the junit xml file as input, and
generates a summary report which is convenient for humans.
[1] https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/usr/local/bin/gen_results_summary
Some of these properties include the version of xfstests, xfsprogs,
and other key software components (for example, I've had test failures
traced to bugs in fio, so knowing the version of fio that is used is
super-handy).
So maybe we could pass in a properties file, either via a command-line
option or an environment variable? My script[2] uses a colon
separated format, but I'm not wedded to that delimiter.
CMDLINE: "-c f2fs/default -g auto"
FSTESTIMG: gce-xfstests/xfstests-amd64-202212131454
FSTESTPRJ: gce-xfstests
KERNEL: kernel 6.1.0-xfstests #2 SMP PREEMPT_DYNAMIC Mon Dec 12 16:09:40 EST 2022 x86_64
FSTESTVER: blktests 068bd2a (Fri, 18 Nov 2022 08:38:35 +0900)
FSTESTVER: fio fio-3.31 (Tue, 9 Aug 2022 14:41:25 -0600)
FSTESTVER: fsverity v1.5 (Sun, 6 Feb 2022 10:59:13 -0800)
FSTESTVER: ima-evm-utils v1.3.2 (Wed, 28 Oct 2020 13:18:08 -0400)
FSTESTVER: nvme-cli v1.16 (Thu, 11 Nov 2021 13:09:06 -0800)
FSTESTVER: quota v4.05-52-gf7e24ee (Tue, 1 Nov 2022 11:45:06 +0100)
FSTESTVER: util-linux v2.38.1 (Thu, 4 Aug 2022 11:06:21 +0200)
FSTESTVER: xfsprogs v6.0.0 (Mon, 14 Nov 2022 12:06:23 +0100)
FSTESTVER: xfstests-bld 65edab38 (Wed, 30 Nov 2022 12:11:57 -0500)
FSTESTVER: xfstests v2022.11.27-8-g3c178050c (Wed, 30 Nov 2022 10:25:39 -0500)
FSTESTVER: zz_build-distro bullseye
FSTESTCFG: "f2fs/default"
FSTESTSET: "-g auto"
FSTESTEXC: ""
FSTESTOPT: "aex"
MNTOPTS: ""
CPUS: "2"
MEM: "7680"
DMI_MEM: 8 GB (Max capacity)
PARAM_MEM: 7680 (restricted by cmdline)
GCE ID: "3198461547210171740"
MACHINE TYPE: "e2-standard-2"
TESTRUNID: tytso-20221213150813
[2] https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/usr/local/bin/update_properties_xunit
Cheers,
- Ted
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 6/8] report: collect basic information about a test run
2022-12-20 3:29 ` Theodore Ts'o
@ 2023-02-14 18:59 ` Darrick J. Wong
0 siblings, 0 replies; 19+ messages in thread
From: Darrick J. Wong @ 2023-02-14 18:59 UTC (permalink / raw)
To: Theodore Ts'o
Cc: zlang, linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs
On Mon, Dec 19, 2022 at 10:29:58PM -0500, Theodore Ts'o wrote:
> On Mon, Dec 19, 2022 at 04:01:37PM -0800, Darrick J. Wong wrote:
> > From: Darrick J. Wong <djwong@kernel.org>
> >
> > Record various generic information about an fstests run when generating
> > a junit xml report. This includes the cpu architecture, the kernel
> > revision, the CPU, memory, and numa node counts, and some information
> > about the block devices passed in.
>
> It would be nice if there was a way that the test runner could pass
> information that would be added to the xunit properties. As I
> mentioned in another e-mail, I currently do this via a post-processing
> step which adds the properties to the junit xml file via a python
> script. And there are a number of additional properties that are used
> by my report generator[1] which takes the junit xml file as input, and
> generates a summary report which is convenient for humans.
>
> [1] https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/usr/local/bin/gen_results_summary
>
> Some of these properties include the version of xfstests, xfsprogs,
> and other key software components (for example, I've had test failures
> traced to bugs in fio, so knowing the version of fio that is used is
> super-handy).
>
> So maybe we could pass in a properties file, either via a command-line
> option or an environment variable? My script[2] uses a colon
> separated format, but I'm not wedded to that delimiter.
>
> CMDLINE: "-c f2fs/default -g auto"
> FSTESTIMG: gce-xfstests/xfstests-amd64-202212131454
> FSTESTPRJ: gce-xfstests
> KERNEL: kernel 6.1.0-xfstests #2 SMP PREEMPT_DYNAMIC Mon Dec 12 16:09:40 EST 2022 x86_64
> FSTESTVER: blktests 068bd2a (Fri, 18 Nov 2022 08:38:35 +0900)
> FSTESTVER: fio fio-3.31 (Tue, 9 Aug 2022 14:41:25 -0600)
> FSTESTVER: fsverity v1.5 (Sun, 6 Feb 2022 10:59:13 -0800)
> FSTESTVER: ima-evm-utils v1.3.2 (Wed, 28 Oct 2020 13:18:08 -0400)
> FSTESTVER: nvme-cli v1.16 (Thu, 11 Nov 2021 13:09:06 -0800)
> FSTESTVER: quota v4.05-52-gf7e24ee (Tue, 1 Nov 2022 11:45:06 +0100)
> FSTESTVER: util-linux v2.38.1 (Thu, 4 Aug 2022 11:06:21 +0200)
> FSTESTVER: xfsprogs v6.0.0 (Mon, 14 Nov 2022 12:06:23 +0100)
> FSTESTVER: xfstests-bld 65edab38 (Wed, 30 Nov 2022 12:11:57 -0500)
> FSTESTVER: xfstests v2022.11.27-8-g3c178050c (Wed, 30 Nov 2022 10:25:39 -0500)
Do you want the version numbers of each dependency to have a unique
name attribute here?
<property name="FSTESTVER: xfstests" value="v2022.11.27-8-g3c178050c..."/>
Though ... technically speaking, the @name attributes aren't required to
be unique, so this is valid:
<property name="FSTESTVER" value="xfstests-bld 65edab38..."/>
<property name="FSTESTVER" value="xfstests v2022.11.27-8-g3c178050c..."/>
Or I could go with what I've been rambling about on the ext4 concall for
some time now: set EXTRA_REPORT_VARS to a path to a file containing
"name: value" strings, one per line, split on the colon. You all can
translate this into such a format however you like. :)
--D
> FSTESTVER: zz_build-distro bullseye
> FSTESTCFG: "f2fs/default"
> FSTESTSET: "-g auto"
> FSTESTEXC: ""
> FSTESTOPT: "aex"
> MNTOPTS: ""
> CPUS: "2"
> MEM: "7680"
> DMI_MEM: 8 GB (Max capacity)
> PARAM_MEM: 7680 (restricted by cmdline)
> GCE ID: "3198461547210171740"
> MACHINE TYPE: "e2-standard-2"
> TESTRUNID: tytso-20221213150813
>
> [2] https://github.com/tytso/xfstests-bld/blob/master/test-appliance/files/usr/local/bin/update_properties_xunit
>
> Cheers,
>
> - Ted
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 7/8] report: record xfs-specific information about a test run
2022-12-20 0:01 [PATCHSET RFC 0/8] fstests: improve junit xml reporting Darrick J. Wong
` (5 preceding siblings ...)
2022-12-20 0:01 ` [PATCH 6/8] report: collect basic information about a test run Darrick J. Wong
@ 2022-12-20 0:01 ` Darrick J. Wong
2022-12-20 0:01 ` [PATCH 8/8] report: record ext*-specific " Darrick J. Wong
7 siblings, 0 replies; 19+ messages in thread
From: Darrick J. Wong @ 2022-12-20 0:01 UTC (permalink / raw)
To: djwong, zlang; +Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs
From: Darrick J. Wong <djwong@kernel.org>
Report various XFS-specific information about a test run.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/report | 2 ++
common/xfs | 10 ++++++++++
2 files changed, 12 insertions(+)
diff --git a/common/report b/common/report
index 68646a7709..f2b60c317e 100644
--- a/common/report
+++ b/common/report
@@ -53,6 +53,8 @@ __generate_report_vars() {
__generate_blockdev_report_vars "TEST_DEV"
__generate_blockdev_report_vars "SCRATCH_DEV"
+
+ test "$FSTYP" = "xfs" && __generate_xfs_report_vars
}
#
diff --git a/common/xfs b/common/xfs
index 7eee76c0ee..add3a42fa4 100644
--- a/common/xfs
+++ b/common/xfs
@@ -2,6 +2,16 @@
# XFS specific common functions.
#
+__generate_xfs_report_vars() {
+ __generate_blockdev_report_vars TEST_RTDEV
+ __generate_blockdev_report_vars TEST_LOGDEV
+ __generate_blockdev_report_vars SCRATCH_RTDEV
+ __generate_blockdev_report_vars SCRATCH_LOGDEV
+
+ REPORT_VARS["XFS_ALWAYS_COW"]="$(cat /sys/fs/xfs/debug/always_cow 2>/dev/null)"
+ REPORT_VARS["XFS_LARP"]="$(cat /sys/fs/xfs/debug/larp 2>/dev/null)"
+}
+
_setup_large_xfs_fs()
{
fs_size=$1
^ permalink raw reply related [flat|nested] 19+ messages in thread* [PATCH 8/8] report: record ext*-specific information about a test run
2022-12-20 0:01 [PATCHSET RFC 0/8] fstests: improve junit xml reporting Darrick J. Wong
` (6 preceding siblings ...)
2022-12-20 0:01 ` [PATCH 7/8] report: record xfs-specific " Darrick J. Wong
@ 2022-12-20 0:01 ` Darrick J. Wong
7 siblings, 0 replies; 19+ messages in thread
From: Darrick J. Wong @ 2022-12-20 0:01 UTC (permalink / raw)
To: djwong, zlang; +Cc: linux-xfs, fstests, guan, leah.rumancik, quwenruo.btrfs
From: Darrick J. Wong <djwong@kernel.org>
Report various ext* specific information about a test run.
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
common/ext4 | 5 +++++
common/report | 1 +
2 files changed, 6 insertions(+)
diff --git a/common/ext4 b/common/ext4
index 8fd6dbc682..3dcbfe17c9 100644
--- a/common/ext4
+++ b/common/ext4
@@ -2,6 +2,11 @@
# ext4 specific common functions
#
+__generate_ext4_report_vars() {
+ __generate_blockdev_report_vars TEST_LOGDEV
+ __generate_blockdev_report_vars SCRATCH_LOGDEV
+}
+
_setup_large_ext4_fs()
{
local fs_size=$1
diff --git a/common/report b/common/report
index f2b60c317e..199851084c 100644
--- a/common/report
+++ b/common/report
@@ -55,6 +55,7 @@ __generate_report_vars() {
__generate_blockdev_report_vars "SCRATCH_DEV"
test "$FSTYP" = "xfs" && __generate_xfs_report_vars
+ [[ "$FSTYP" == ext[0-9]* ]] && __generate_ext4_report_vars
}
#
^ permalink raw reply related [flat|nested] 19+ messages in thread