Quality of meta-oe metadata

[Martin Jansa <martin.jansa@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 4170 bytes --]

Hi, sorry for longer e-mail, this is one of topic I would like to discuss
on OEDAM (http://openembedded.org/wiki/OEDAM), but having some feedback and
thoughts in advance will be very useful.

As people can notice from my "State of bitbake world" e-mails or
http://www.openembedded.org/wiki/Bitbake_World_Status
we never had "green" builds. There are always 20+ failed tasks in those
big builds and just reading the numbers isn't good indicator of quality,
because sooner you break something in dependency tree, fewer recipes will
be actually tested, so fewer failed tasks often means that something
important is broken.

There are IMHO at least 3 reasons for this depressing state:

1) Nobody is paid/dedicated to fix them, there is no company behind meta-oe
   layers, now SWAT team, not even dedicated maintainers to which I could send
   error from latest build and ask them to fix it before the end of
   week/month.

   Kudos to people who are sometimes sending patches for such issues!

2) There are a lot of changes and component upgrades in oe-core which
   sometimes aren't very straight-forward to adapt to and issues stay in
   meta-oe for months.

   I don't mean it's oe-core fault or that changes to oe-core should slow
   down just because meta-oe, especially when we cannot guarantee when it
   will be prepared for them (because of 1)).

   oe-core is trying to track latest stable versions, but meta-oe often
   contains very old versions and people upgrade to latest stable only the
   recipes they really care about, so it's not so surprising that 2 year old
   version of something isn't compatible with latest greatest freetype or
   some library like that.

3) OE releases work great and don't invalidate sstate signatures so often, so my
   feeling is that most developers and projects are just using releases and
   less and less people do CI. People will start complaining that something
   is broken in meta-oe only when they are upgrading their project from 1.5 to
   1.6 when 1.6 is released and that could be too late for fixing meta-oe
   issues.

What I'm trying to do with it:

a) sending those e-mails and updating wiki, so that people can easily find
   if some build failure is common or something which happens only for them
   (something like oestats-client.bbclass page was providing in oe-classic)
   It also includes log of QA issues which are usually easy to fix and great
   way for new people to learn something about OE.
b) trying to refuse all patches which cause new world issue (or new QA
   warn/err) - sometimes missed in logs, because it's often "hidden" by some
   other issue and hard to compare 40 issues from previous build with 38
   from current.
   Also the issues are often triggered later by changes somewhere else...
c) fixing build/qa issues in recipes I've never used or don't even have
   hardware to test - just based on assumption that something which builds
   is better than broken build, even when it can have some issues in runtime.
d) contacting people who added the recipe which is now failing, often
   without reply for months even when I try it multiple times :/
e) moving to "nonworking" directory to mark it as "known-to-be-broken",
   last resort for recipes where the fix is complicated and it's not known
   if someone is actually using it (because it was broken for months and
   nobody replied).
   + easy to find them, because they are still in repository (instead of
     git rm + revert when someone fixes it)
   - layer index probably doesn't find them, because "nonworking" directory
     level isn't in BBFILES, so maybe meta-broken or meta-nonworking would be
     better
   ? some recipes are "broken" just because their dependency is broken, what
     to do with such recipe, I usually just say that in commit message when
     I'm moving them to "nonworking" with their broken dep.

What can we do better? How to motivate more people to do CI and send fixes?
When we get to "green" state it will be easier to quickly spot new issues and
easier to fix them, because it will be clear what's causing them.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]