Re: OE recipe tree quality

[Graeme Gregory <dp@xora.org.uk>]

 On 30/07/10 09:22, Koen Kooi wrote:
> On 30-07-10 09:21, Esben Haabendal wrote:
> > On Thu, Jul 29, 2010 at 2:07 PM, Philip Balister
> <philip@balister.org> wrote:
> >>
> >>
> >> On 07/29/2010 05:45 AM, Koen Kooi wrote:
> >>>
> >>> -----BEGIN PGP SIGNED MESSAGE-----
> >>> Hash: SHA1
> >>>
> >>> On 29-07-10 10:50, Frans Meulenbroeks wrote:
> >>>>
> >>>> Dear all,
> >>>>
> >>>> Given the discussions on quality that sometimes pop up (and also
> >>>> triggered
> >>>> by Robert's message), I decided to kick off a bitbake -k world.
> >>>
> >>> Could you first explain to me why 'bitbake world' is a good way to
> >>> measure quality?
> >>>
> >>> I would think that building something like console-image and
> looking at
> >>> the following would be a much better metric:
> >>>
> >>> * does it build?
> >>> * are all the rootfs types working?
> >>> * does the image do what it is supposed to do?
> >>> * Are all the licenses of the output packages correct?
> >>> * Do the output packages have any spurious deps?
> >>> * Is the content of the output packages correct?
> >>> * Are there any known CVEs in the resulting packages?
> >>> * Did packaged-staging do its job?
> >>> * What kind of QA errors and warnings were raised?
> >>> * Did all recipes pass recipe_sanity?
> >>> * Did all recipes conform to oe-stylize.py?
> >>>
> >>> etc
> >>>
> >>> I would actually advocate removing the 'world' feature from bitbake/OE
> >>> to stop people from wasting time on looking at bitbake world and have
> >>> them fix actual problems.
> >>
> >> bitbake world seems to be the source of pointless listserv
> discussions. Does
> >> it serve any purpose?
>
> > Pointless or not really depends on how you look at quality.
>
> > If you look at it as you, Koen and other OE long-timers, yes, it looks
> > rather pointless to have bitbake world.
> > But for those of us who have a different view on what quality is, then
> > bitbake world serves a purpose.
>
> As Thomas points out, as soon as you start blacklisting things (which
> actually increases quality), bitbake world doesn't work anymore.
> That alone should be enough to kill it.
Time to jump in the cage here.

"Quality" is achieved by comparing a set of known specifications against
a known data set. In the software case this means we need a good set of
specifications which we are testing against. We also need to know in
detail what we are testing against this set of specifications.

bitbake world meets neither of these criteria. You have no idea what
your testing against, you also have no idea what you are submitting for
test. A random selection of 8500 files in OE. It also doesn't take into
account known combinations which always fail. Angstrom took care of
these known failures by creating the concept of a blacklist.

In fact if you tell me bitbake world fails, I would actually suggest
that is a test pass, it is an expected fail.

Graeme