Re: [PATCH 1/8] check: generate section reports between tests

Qu Wenruo
2023-02-15T02:07:00Z

On 2023/2/15 02:46, Darrick J. Wong wrote:
> On Tue, Dec 20, 2022 at 09:14:07AM +0800, Qu Wenruo wrote:
>>
>>
>> On 2022/12/20 08:01, Darrick J. Wong wrote:
>>> From: Darrick J. Wong <djwong@kernel.org>
>>>
>>> Generate the section report between tests so that the summary report
>>> always reflects the outcome of the most recent test. Two usecases are
>>> envisioned here -- if a cluster-based test runner anticipates that the
>>> testrun could crash the VM, they can set REPORT_DIR to (say) an NFS
>>> mount to preserve the intermediate results. If the VM does indeed
>>> crash, the scheduler can examine the state of the crashed VM and move
>>> the tests to another VM. The second usecase is a reporting agent that
>>> runs in the VM to upload live results to a test dashboard.
>>>
>>> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
>>> ---
>>> check | 10 ++++++++++
>>> 1 file changed, 10 insertions(+)
>>>
>>>
>>> diff --git a/check b/check
>>> index 1ff0f44af1..70a0b537b1 100755
>>> --- a/check
>>> +++ b/check
>>> @@ -842,6 +842,16 @@ function run_section()
>>> fi
>>> seqres="$REPORT_DIR/$seqnum"
>>> + # Generate the entire section report with whatever test results
>>> + # we have so far. Leave the $sect_time parameter empty so that
>>> + # it's a little more obvious that this test run is incomplete.
>>> + if $do_report; then
>>> + local sect_now=`_wallclock`
>>> + _make_section_report "$section" "${#try[*]}" \
>>> + "${#bad[*]}" "${#notrun[*]}" \
>>> + "" &> /dev/null
>>> + fi
>>> +
>>
>> In fact that's exactly what I just wanted to do next...
>>
>> However one problem I can't work around is the timestamp.
>>
>> Currently our timestamp is the finish time, with this change our timestamp
>> is the start time.
>
> Er... what do you mean exactly by start and finish time? The most
> recent test? The entire run?

Personally speaking, I'd prefer start/finish time of the whole
suite/section at least.

>
> From check:
>
> if [ -w /dev/kmsg ]; then
> export date_time=`date +"%F %T"`
> echo "run fstests $seqnum at $date_time" > /dev/kmsg
> ...
> fi
>
> From common/report:
>
> if [ -z "$date_time" ]; then
> date_time=$(date +"%F %T")
> fi
>
> ...timestamp="${date_time/ /T}">
>
> The "date_time" variable can be set by ./check just prior to starting
> each test, but only if /dev/kmsg is a writable file. If it's set, then
> the timestamp in the xml file reflects the start time of the most recent
> test.
>
> If date_time is not set, then the timestamp in the xml file records the
> point in time when the report is generated... which I suppose is a good
> enough estimate if we failed to record date_time when the test was
> started. >
>> I'm not sure if there is any strong definition on the timestamp, but
>> personally speaking, timestamp for the start time makes more sense to me.
>> But there may be some users relying on the timestamp to be the ending time?
>
> The junit xml schema says that the timestamp should be "when the test
> was executed". I think ./check ought to be setting date_time regardless
> of /dev/kmsg. ATM my test dashboard computes the difference between now
> and the timestamp= attribute of the last report received to decide if it
> should flag a test VM as possibly stalled.
>
> Though to undercut everything I just wrote, the junit xml schema defines
> testsuite and testcase as distinct entities and then refers to "the
> test".
>
> Given that fstests doesn't strictly follow that schema anyway, I guess
> we can define the timestamp as we want, or add more timestamp
> attributes?
>
> <testsuite suite_timestamp="..." timestamp="..." report_timestamp="..."...>

OK, then this looks much better.

Thanks,
Qu

>
> --D
>
>> Thanks,
>> Qu
>>
>>> mkdir -p $RESULT_DIR
>>> rm -f ${RESULT_DIR}/require_scratch*
>>> rm -f ${RESULT_DIR}/require_test*
>>>