Re: [OE-core] [PATCH v3 0/6] scripts/resulttool/regression: add metadata filtering

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Sat, 2023-02-25 at 16:59 +0100, Alexis Lothoré wrote:
> Hello Richard,
> as usual, thanks for the prompt feedback !
> 
> On 2/25/23 13:32, Richard Purdie wrote:
> > On Sat, 2023-02-25 at 09:15 +0000, Richard Purdie via
> > lists.openembedded.org wrote:
> > > On Fri, 2023-02-24 at 18:06 +0000, Richard Purdie via
> > > lists.openembedded.org wrote:
> > > > Hi Alexis,
> > > > 
> > > > Firstly, this looks very much improved, thanks. It is great to start to
> > > > see some meaningful data from this.
> > > > 
> > > > On Fri, 2023-02-24 at 17:45 +0100, Alexis Lothoré via
> > > > lists.openembedded.org wrote:
> > > > > After manual inspection on some entries, the remaining oeselftest regression
> > > > > raised in the report seems valid. There are still some issues to tackle:
> > > > > - it seems that now one major remaining source of noise is on the "runtime"
> > > > >   tests (comparison to tests not run on "target" results)
> > > > > - when a ptest managed by oe-selftest fails, I guess the remaining tests are not
> > > > >   run, so when 1 failure is logged, we have many "PASSED->None" transitions in
> > > > >   regression report, we should probably silence it.
> > > > > - some transitions appear as regression while those are in fact improvements
> > > > >   (e.g: "UNRESOLVED->PASSED")
> > > > 
> > > > I had quick play. Firstly, if I try "yocto_testresults_query.py
> > > > regression-report 4.2_M1 4.2_M2" in an openembedded-core repository
> > > > instead of poky, it breaks. That isn't surprising but we should either
> > > > make it work or show a sensible error.
> 
> Oh right, I am working in a Poky build configuration, so I have assumed that this
> would be the unique use case.
> Since the test results commits are tightly coupled to revisions in poky (so not
> oecore), I plan to merely log an error about not found revision (and suggesting
> the user to check that the repository is poky and not oecore).
> But please let me know if I miss a major use case here and that a smarter
> fallback plan (shallow-clone poky if we are running in oecore ?) is needed

I'm happy to for it just to give an human readable error, someone can
add this functionality if they need/want it.

> > > I think I might be tempted to merge this series and then we can change
> > > the code to improve from here as this is clearly a vast improvement on
> > > where we were! Improvements can be incremental on top of these changes.
> 
> I am in favor of this :) If it is OK for you, I will just re-submit a series with
> the fix for the proper error logging when running the tool from oecore and not poky.
> 
> Next we could introduce all the suggestions you have suggested, but I feel that
> with the quick increase of "hotfixes" count to support issues with older test
> results, and for the sake of maintainability of resulttool and its submodules,
> those specific hotfixes need to be properly isolated (and documented), like in a
> "regression_quirks.py" or something like that. What do you think ?

I'm hoping we don't have many of these quirks. We have a huge history
at this point so it would be sad if the tool can't work with it. From
what I've seen so far, we can manage with the code in the regression
module itself. I've tried to add some comments.

I wondered what to do with this series since I needed to get M3 built.
Since this series was available and mostly usable, it would be better
to have a nicer report this time, it is a good test of the code.

In the end I've merged most of it, along with my two tweaks to handle
LTP and the bigger ptest results issue. I couldn't take one set of the
selftests since they simply don't work. This will give us a useful
realworld test of the M3 report.

I'm working on the assumption you'll send a follow up series with the
tests, the oe-core check and some of the other issues I've mentioned in
other emails?

Cheers,

Richard