Re: [PATCH] check: annotate good and expunged tests in results

[Jeff Mahoney <jeffm@suse.com>]

On 4/30/18 4:48 PM, Jeff Mahoney wrote:
> On 4/27/18 10:45 PM, Eryu Guan wrote:
>> On Fri, Apr 27, 2018 at 05:00:09PM +0000, Luis R. Rodriguez wrote:
>>> On Fri, Apr 27, 2018 at 07:23:56PM +0800, Eryu Guan wrote:
>>>>
>>>> But the test run & pass info is already available from the check output
>>>> and the test result summary at the end of check. Is that sufficient for
>>>> you? Also, we already have mechanism to generate a test report in xunit
>>>> format, i.e. ./check -R xunit -g auto, which includes results for passed
>>>> & failed & notrun tests.
>>>
>>> Do we have a way to parse the results *after* a run? For instance,
>>> if you forgot -R xunit ?
>>
>> There's a tools/compare-failures script that takes the outputs of check
>> as inputs and compares the results. But, TBH, I never run it after
>> reviewing it.. Perhaps it could be enhanced somehow.
> 
> Yeah, that takes check output.  Without capturing it at runtime, it
> can't compare anything.
> 
> The XML report may do most of what I want provided we can enable it by
> default and write it someplace safe rather than clean it up
> automatically when the test run is interrupted.
> 
> I already have test code that extends it to output expunged tests and to
> add an explicit <pass/> element.  It saves the timestamps already, so
> that's a plus.

And shortly after writing this, I had a test run hang.  Since it hung,
even writing it on exit wouldn't have worked.  There's not enough
information leftover to generate it.  I think I'd still like the results
directory to contain the information required to generate it as a
post-mortem.

-Jeff

-- 
Jeff Mahoney
SUSE Labs