Re: [xen-4.6-testing test] 65112: regressions - FAIL

[Ian Campbell <ian.campbell@citrix.com>]

On Fri, 2015-11-27 at 14:03 +0000, Ian Campbell wrote:
> On Fri, 2015-11-27 at 13:24 +0000, Ian Jackson wrote:
> > Ian Campbell writes ("Re: [Xen-devel] [xen-4.6-testing test] 65112:
> > regressions - FAIL"):
> > > On Fri, 2015-11-27 at 12:02 +0000, Ian Jackson wrote:
> > > > As explained below, in 65112 this step did not run because the
> > > > earlier
> > > > step `guest-localmigrate' failed:
> > > >   http://logs.test-lab.xenproject.org/osstest/logs/65112/test-amd64
> > > > -
> > > > amd64-xl-qemut-stubdom-debianhvm-amd64-xsm/info.html
> > > 
> > > Would it be possible to arrange for "blocked" to appear somewhere in
> > > the
> > > results for the job? e.g. "blocked fail in XXX REGR. vs. YYY".
> > > README.email
> > > says "The results normally start with the result in this flight" and
> > > I
> > > think this would be in keeping with that.
> > 
> > But it might not be true that it was blocked.
> 
> Can't sg-run-job tell if it was blocked vs something else though?

I meant sg-report-flight, of course.

> 
> I was suggesting to only add blocked if it was blocked, I'm not sure what
> I
> was suggesting to do for other reason not to run, because I hadn't really
> considered it, but those would be unusual I think?
> 
> >   Maybe the version of
> > osstest used didn't have that step at all, for example.
> 
> In which case would it still be considering the step for failures at all?
> 
> i.e. if:
> 
> flight 100 had test-foo == pass
> flight 200 had test-foo == fail (blocking)
> flight 201 had test-foo == blocked; fail in 201 vs 100
> flight 202 had no test-foo present at all
> 
> Would the decision for flight 202 really be to consider the test-foo
> results in 100, 200 and 201, and therefore block?
> 
> > The best you could say would be something like
> >   "not run; fail in XXX REGR. vs. YYY"
> > but that poses more questions than it answers.
> 
> Right.
> 
> > 
> > > Otherwise I think people naturally tend to just read the "and are
> > > blocking"
> > > section and forget to consider that non-blocking stuff further down
> > > may
> > > have (tolerably) failed but then blocking something else which is
> > > then
> > > blocking the push.
> > 
> > Perhaps sg-report-flight could, if there are any blockages of the form
> > `fail in XXX REGR. vs YYY', add a note below the blockage section,
> > saying something like `XXX examined since needed to justify other
> > failures, see below'.
> > 
> > I'm a bit reluctant to suggest this because it is, essentially,
> > boilerplate - it would always say the same thing about any `fail in
> > XXX' - and filling reports like this with boilerplate isn't always a
> > good idea.
> 
> In general I agree, in this case it might be worth it to counteract a
> (perfectly understandable IMHO) natural tendency to only look at the
> section labelled blocking, it's basically "don't forget that this non-
> blocking stuff might actually be relevant to the blockage".
> 
> Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel