Re: [PATCH 1/8] check: generate section reports between tests

["Darrick J. Wong" <djwong@kernel.org>]

On Tue, Dec 20, 2022 at 09:14:07AM +0800, Qu Wenruo wrote:
> 
> 
> On 2022/12/20 08:01, Darrick J. Wong wrote:
> > From: Darrick J. Wong <djwong@kernel.org>
> > 
> > Generate the section report between tests so that the summary report
> > always reflects the outcome of the most recent test.  Two usecases are
> > envisioned here -- if a cluster-based test runner anticipates that the
> > testrun could crash the VM, they can set REPORT_DIR to (say) an NFS
> > mount to preserve the intermediate results.  If the VM does indeed
> > crash, the scheduler can examine the state of the crashed VM and move
> > the tests to another VM.  The second usecase is a reporting agent that
> > runs in the VM to upload live results to a test dashboard.
> > 
> > Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> > ---
> >   check |   10 ++++++++++
> >   1 file changed, 10 insertions(+)
> > 
> > 
> > diff --git a/check b/check
> > index 1ff0f44af1..70a0b537b1 100755
> > --- a/check
> > +++ b/check
> > @@ -842,6 +842,16 @@ function run_section()
> >   		fi
> >   		seqres="$REPORT_DIR/$seqnum"
> > +		# Generate the entire section report with whatever test results
> > +		# we have so far.  Leave the $sect_time parameter empty so that
> > +		# it's a little more obvious that this test run is incomplete.
> > +		if $do_report; then
> > +			local sect_now=`_wallclock`
> > +			_make_section_report "$section" "${#try[*]}" \
> > +					     "${#bad[*]}" "${#notrun[*]}" \
> > +					     "" &> /dev/null
> > +		fi
> > +
> 
> In fact that's exactly what I just wanted to do next...
> 
> However one problem I can't work around is the timestamp.
> 
> Currently our timestamp is the finish time, with this change our timestamp
> is the start time.

Er... what do you mean exactly by start and finish time?  The most
recent test?  The entire run?

From check:

	if [ -w /dev/kmsg ]; then
		export date_time=`date +"%F %T"`
		echo "run fstests $seqnum at $date_time" > /dev/kmsg
		...
	fi

From common/report:

	if [ -z "$date_time" ]; then
		date_time=$(date +"%F %T")
	fi

	...timestamp="${date_time/ /T}">

The "date_time" variable can be set by ./check just prior to starting
each test, but only if /dev/kmsg is a writable file.  If it's set, then
the timestamp in the xml file reflects the start time of the most recent
test.

If date_time is not set, then the timestamp in the xml file records the
point in time when the report is generated... which I suppose is a good
enough estimate if we failed to record date_time when the test was
started.

> I'm not sure if there is any strong definition on the timestamp, but
> personally speaking, timestamp for the start time makes more sense to me.
> But there may be some users relying on the timestamp to be the ending time?

The junit xml schema says that the timestamp should be "when the test
was executed".  I think ./check ought to be setting date_time regardless
of /dev/kmsg.  ATM my test dashboard computes the difference between now
and the timestamp= attribute of the last report received to decide if it
should flag a test VM as possibly stalled.

Though to undercut everything I just wrote, the junit xml schema defines
testsuite and testcase as distinct entities and then refers to "the
test".

Given that fstests doesn't strictly follow that schema anyway, I guess
we can define the timestamp as we want, or add more timestamp
attributes?

<testsuite suite_timestamp="..." timestamp="..." report_timestamp="..."...>

--D

> Thanks,
> Qu
> 
> >   		mkdir -p $RESULT_DIR
> >   		rm -f ${RESULT_DIR}/require_scratch*
> >   		rm -f ${RESULT_DIR}/require_test*
> >