Re: [PATCH] check: annotate good and expunged tests in results

Luis Chamberlain
2019-02-12T17:05:01Z

On Tue, May 1, 2018 at 8:44 AM Jeff Mahoney <jeffm@suse.com> wrote:
>
> On 5/1/18 9:34 AM, Jeff Mahoney wrote:
> > On 4/30/18 4:48 PM, Jeff Mahoney wrote:
> >> On 4/27/18 10:45 PM, Eryu Guan wrote:
> >>> On Fri, Apr 27, 2018 at 05:00:09PM +0000, Luis R. Rodriguez wrote:
> >>>> On Fri, Apr 27, 2018 at 07:23:56PM +0800, Eryu Guan wrote:
> >>>>>
> >>>>> But the test run & pass info is already available from the check output
> >>>>> and the test result summary at the end of check. Is that sufficient for
> >>>>> you? Also, we already have mechanism to generate a test report in xunit
> >>>>> format, i.e. ./check -R xunit -g auto, which includes results for passed
> >>>>> & failed & notrun tests.
> >>>>
> >>>> Do we have a way to parse the results *after* a run? For instance,
> >>>> if you forgot -R xunit ?
> >>>
> >>> There's a tools/compare-failures script that takes the outputs of check
> >>> as inputs and compares the results. But, TBH, I never run it after
> >>> reviewing it.. Perhaps it could be enhanced somehow.
> >>
> >> Yeah, that takes check output. Without capturing it at runtime, it
> >> can't compare anything.
> >>
> >> The XML report may do most of what I want provided we can enable it by
> >> default and write it someplace safe rather than clean it up
> >> automatically when the test run is interrupted.
> >>
> >> I already have test code that extends it to output expunged tests and to
> >> add an explicit <pass/> element. It saves the timestamps already, so
> >> that's a plus.
> >
> > And shortly after writing this, I had a test run hang. Since it hung,
> > even writing it on exit wouldn't have worked. There's not enough
> > information leftover to generate it. I think I'd still like the results
> > directory to contain the information required to generate it as a
> > post-mortem.
>
> After thinking on it a bit more, since you object to writing a bunch of
> files by default, we could accomplish the same goal by adding a "files"
> report type that does this without dropping files for everyone. I'm
> working that up now.

How's that going BTW? :)

Luis