Re: [RFC PATCH 0/5] revive runtime/cases/_ptest.py

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Thu, 2017-07-20 at 10:09 +0800, Robert Yang wrote:
> 
> On 07/19/2017 04:16 PM, Robert Yang wrote:
> > 
> > Hello,
> > 
> > These patches can make ptest test case work, RP has suggested we
> > write a tool to
> > do the regression check on ptest result, I think that the use case
> > is like:
> > 
> > $ bitbake <image> -ctestiamge # Suppose we add ptest to default
> > test cases in the future
> > # Upgrade a recipe form V1.0 to V1.1
> > $ bitbake <image> -ctestiamge # Run the test again
> > 
> > Then the regression check tool can report what's different (passed,
> > failed,
> > skipped) between V1.0 and V1.1.
> > 
> > Currently, I'm not sure about where to save the ptest results, I
> > saved it to
> > ${WORKDIR}/testimage/ptest_log atm, e.g.:
> > tmp/work/qemux86-poky-linux/core-image-minimal/1.0-
> > r0/testimage/ptest_log
> I still have to think about more about where to put the test result.

Would buildhistory make sense for these since we already have a
mechanism to store results there between builds?

> > But it will be removed after -cclean, then no regression check can
> > be made, so
> > I'd like to save the ptest results to DEPLOY_DIR_IMAGE if no
> > objections, and
> > make -cclean not remove them. (Or only keep the latest 2 results).
> > 
> > And I'm not sure where to add the regression check/tool, maybe one
> > of:
> > 1) Add a separate tool in oe-core/scripts, this can make it easy to
> > do regression
> >    check among different build directories, and
> > runtime/cases/_ptest.py can invoke it.
> I'm leaning to add a script called ptest-regression-check which is
> more flexible, the cases/ptest.py or buildhistory (also the user) can
> run it when needed.

I'd either add a script, or perhaps an option to the buildhistory-diff? 
Paul might have some adivce on that. I would like to understand if
there is a good reason we can't use buildhistory to at least help with
some of this?

Cheers,

Richard