Layer model doomed, unless we all work together

Layer model doomed, unless we all work together

[Philip Balister]

As evidence, please review this list:

http://layers.openembedded.org/layerindex/branch/master/duplicates/

I mean FOUR recipes for alsa-plugins?

I am trying to fix the pyqt recipe in meta-oe, and had th eidea to check
for it in other layers. This leads me to meta-ivi-demo, which has an
update for sip-native and another pyqt recipe. To be fair, they are
using Qt5, so it is a little more complex.

At any rate, we need to stop ripping recipes out of layers and maing
local copies, or worse, updating our local copies and not the primary
layer. The intent of the layer concept was not to fragment development
across many separate repositories.

I see a couple of issues we need to start talking about:

1) recipes that need to move closer to core because a range of other
packages use them.

2) people feel they have to remove recipes from layers and make local
copies.

And just so people know what seriously pisses me off :) Copying a recipe
from a layer, updating it, and not sending the recipe to the layer they
got the update from.

Thanks for letting me vent,

Philip

Re: Layer model doomed, unless we all work together

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1513 bytes --]

Hi Philip,

Yes.  What you said.  Some comments:

On 18 November 2014 21:28, Philip Balister <philip@balister.org> wrote:
>
> 1) recipes that need to move closer to core because a range of other
> packages use them.
>

alsa-plugins is one of these - meta-multimedia or maybe oe-core is an
obvious home for this.  As a "maintainer" (ha!) of meta-guacamayo I can
take a look at this but I'm fairly busy at the moment so this may take a
while.

In an attempt to balance karma, I have submitted several new recipes that
were in meta-guacamayo and needed elsewhere to meta-multimedia in the last
week. :)


> 2) people feel they have to remove recipes from layers and make local
> copies.
>

Making local copies if fine if required, but I agree that there's bound to
be a high number of recipes which are forked because that was easier than
writing a bbappend, and the reason for this fork was needed is never
resolved in the original layer.

Looking over the duplicate list, there's some obvious low-hanging fruit.
Angstrom duplicates recipes that are in meta-multimedia. meta-webos and
meta-webos-ports seem to be full of duplicates.  meta-guacamayo has some
recipes that were version bumps two years ago when oe-core was frozen.  The
dpdk duplicates are a bug in the tool with regards to sub-layers.

It's a shame that we can't make this some form of offense.  Maybe we should
spend some time researching
https://nctritech.files.wordpress.com/2008/11/tcpip_punch1.jpg?

Ross

[-- Attachment #2: Type: text/html, Size: 2571 bytes --]

Re: Layer model doomed, unless we all work together

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 590 bytes --]

On Tue, Nov 18, 2014 at 09:57:30PM +0000, Burton, Ross wrote:
> Looking over the duplicate list, there's some obvious low-hanging fruit.
> Angstrom duplicates recipes that are in meta-multimedia. meta-webos and
> meta-webos-ports seem to be full of duplicates.  meta-guacamayo has some

meta-webos-ports is fork of meta-webos, so duplicates between these 2
are expected.

Also meta-webos master branch is compatible only with 1.4 dylan release,
so some of the duplicated recipes are actually backports from newer OE.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 188 bytes --]

Re: Layer model doomed, unless we all work together

[Philip Balister]

[-- Attachment #1: Type: text/plain, Size: 1776 bytes --]

On 11/18/2014 06:03 PM, Martin Jansa wrote:
> On Tue, Nov 18, 2014 at 09:57:30PM +0000, Burton, Ross wrote:
>> Looking over the duplicate list, there's some obvious low-hanging fruit.
>> Angstrom duplicates recipes that are in meta-multimedia. meta-webos and
>> meta-webos-ports seem to be full of duplicates.  meta-guacamayo has some
> 
> meta-webos-ports is fork of meta-webos, so duplicates between these 2
> are expected.

Paul explained you can use the filter button to adjust the layers
scanned for dupes. This should be all layers minus meta-webos-ports:

http://layers.openembedded.org/layerindex/branch/master/duplicates/?l=91&l=123&l=30&l=147&l=168&l=148&l=31&l=65&l=66&l=32&l=124&l=164&l=67&l=149&l=77&l=101&l=118&l=33&l=3&l=34&l=142&l=169&l=35&l=137&l=171&l=139&l=108&l=162&l=81&l=82&l=95&l=125&l=145&l=146&l=114&l=4&l=36&l=68&l=140&l=83&l=132&l=5&l=154&l=120&l=84&l=6&l=93&l=112&l=7&l=37&l=167&l=38&l=8&l=98&l=39&l=165&l=40&l=105&l=69&l=9&l=10&l=110&l=107&l=11&l=12&l=13&l=14&l=41&l=15&l=99&l=151&l=170&l=85&l=42&l=43&l=113&l=159&l=16&l=78&l=92&l=103&l=157&l=97&l=119&l=70&l=152&l=153&l=71&l=126&l=115&l=136&l=44&l=45&l=163&l=17&l=102&l=150&l=46&l=18&l=19&l=79&l=2&l=138&l=174&l=20&l=21&l=128&l=129&l=130&l=131&l=47&l=104&l=48&l=135&l=22&l=173&l=121&l=23&l=160&l=49&l=106&l=24&l=50&l=117&l=143&l=87&l=51&l=52&l=25&l=133&l=116&l=53&l=72&l=122&l=73&l=54&l=55&l=172&l=88&l=109&l=56&l=57&l=100&l=26&l=155&l=90&l=58&l=144&l=59&l=60&l=74&l=61&l=75&l=158&l=134&l=62&l=111&l=27&l=76&l=28&l=141&l=1&l=64

Although that url might fail, you get the idea how to use the tool better :)

Philip


> 
> Also meta-webos master branch is compatible only with 1.4 dylan release,
> so some of the duplicated recipes are actually backports from newer OE.
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 484 bytes --]

Re: Layer model doomed, unless we all work together

[Robert Berger]

Hi,

On 11/19/2014 05:19 PM, Philip Balister wrote:
> 
> http://layers.openembedded.org/layerindex/branch/master/duplicates/?l=91&l=123&l=30&l=147&l=168&l=148&l=31&l=65&l=66&l=32&l=124&l=164&l=67&l=149&l=77&l=101&l=118&l=33&l=3&l=34&l=142&l=169&l=35&l=137&l=171&l=139&l=108&l=162&l=81&l=82&l=95&l=125&l=145&l=146&l=114&l=4&l=36&l=68&l=140&l=83&l=132&l=5&l=154&l=120&l=84&l=6&l=93&l=112&l=7&l=37&l=167&l=38&l=8&l=98&l=39&l=165&l=40&l=105&l=69&l=9&l=10&l=110&l=107&l=11&l=12&l=13&l=14&l=41&l=15&l=99&l=151&l=170&l=85&l=42&l=43&l=113&l=159&l=16&l=78&l=92&l=103&l=157&l=97&l=119&l=70&l=152&l=153&l=71&l=126&l=115&l=136&l=44&l=45&l=163&l=17&l=102&l=150&l=46&l=18&l=19&l=79&l=2&l=138&l=174&l=20&l=21&l=128&l=129&l=130&l=131&l=47&l=104&l=48&l=135&l=22&l=173&l=121&l=23&l=160&l=49&l=106&l=24&l=50&l=117&l=143&l=87&l=51&l=52&l=25&l=133&l=116&l=53&l=72&l=122&l=73&l=54&l=55&l=172&l=88&l=109&l=56&l=57&l=100&l=26&l=155&l=90&l=58&l=144&l=59&l=60&l=74&l=61&l=75&l=158&l=134&l=62&l=111&l=27&l=76&l=28&l=141&l=1&l=64
> 
> Although that url might fail, you get the idea how to use the tool better :)

The url works for me;)

You just mention duplicated classes and recipes here, but there are more
things which should be cleaned up ;)

The worst thing about Yocto/poky/OE is, that people can do and do things
in many mysterious ways. Looking through various meta-layers you don't
see a "common coding style", which means that you either need to twist
your brain to understand what others did, or write things from scratch.

I guess we should first publish "best practices" and then try to enforce
them in a first step with maybe with something like "WARN_QA" and
"ERROR_QA" and/or extend oelint.

What do you think?

> 
> Philip
> 

Regards,

Robert


..."Stroustroup writes in the ARM: C programmers think that memory
allocation is too important to be left to the computer, Lisp programmers
think that memory allocation is too important to be left to the programmer."

My public pgp key is available,at:
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x90320BF1

Re: Layer model doomed, unless we all work together

[Joe MacDonald]

[-- Attachment #1: Type: text/plain, Size: 5133 bytes --]

Hey Philip,

[[yocto] Layer model doomed, unless we all work together] On 14.11.18 (Tue 16:28) Philip Balister wrote:

> As evidence, please review this list:
> 
> http://layers.openembedded.org/layerindex/branch/master/duplicates/
> 
> I mean FOUR recipes for alsa-plugins?
> 
> I am trying to fix the pyqt recipe in meta-oe, and had th eidea to check
> for it in other layers. This leads me to meta-ivi-demo, which has an
> update for sip-native and another pyqt recipe. To be fair, they are
> using Qt5, so it is a little more complex.
> 
> At any rate, we need to stop ripping recipes out of layers and maing
> local copies, or worse, updating our local copies and not the primary
> layer. The intent of the layer concept was not to fragment development
> across many separate repositories.

I completely agree and I'm glad you brought it up.  I'm all for doing
what I can to help on this.  I also second Ross' suggestion, perhaps we
could start working on an implementation of RFC5841 as a first phase,
though I'm thinking that may not be the healthiest thing for me.  :-)

> I see a couple of issues we need to start talking about:
> 
> 1) recipes that need to move closer to core because a range of other
> packages use them.

This was actually the only thing I thought needed further discussion
(everything else should largely be a nod-in-agreement thing), as in some
cases I'm not sure it's always clear what constitutes "closer to the
core".  Poky and oe-core layers are pretty clear, but the next step
beyond that isn't.  Are all layers hosted on git.yoctoproject.org
inherhently "more core" than layers on git.openenbedded.org?  And
there's a number of shells when you start including all of the github
projects, setting aside other open-source project hosting services.

Looking at the link you sent out based on Paul's suggestion, I see I'm
actually on both sides of this equation, so yay!  And I'll limit the
discussion to what's indexed there.

Here're my examples.

iscsi-initiator-utils: both in meta-networking nd meta-openstack.  Both
at the same version currently but wildly different contents.
meta-openstack is a git.yoctoproject.org project, so does that make it
closer to the core?  I would think not, but as I recall there had been
some comment about the openstack layer intending to limit layer
dependencies outside Yocto core when it first appeared, so maybe making
meta-networking a dependency is a non-starter for them.

That said, I feel sorry for Alex's meta-cgl layer since it looks like it
uses both meta-networking and meta-openstack and bbappends to
iscsi-initiator-utils, which have significantly different patch sets.
It hasn't bitten me in any of my testing with meta-cgl yet, but I didn't
even notice this before now and I wasn't testing iscsi.  I would not be
at all surprised if different people have different results using iscsi
with meta-cgl now.

On the other side of the coin is the swig recipe in meta-selinux and
meta-oe.  The selinux build requires swig, so the meta-selinux layer
needs to know it is there.  I was actually gearing up a while back to
ditch the swig recipe in meta-selinux since it was so old and obviously
behind the meta-oe version, but didn't ... for reasons that now escape
me but could have been "because that would make meta-selinux depend on
meta-oe", which isn't necessarily a dependency we wanted to bring in for
meta-selinux.  But I'm highly allergic to duplicating (and inevitably
diverging) data, so I'd still like to kick swig out of meta-selinux.
But I also almost never build meta-selinux projects without some of
meta-oe kicking around too.  Maybe that's everyone who uses
meta-selinux, I don't know.

Anyway, just to keep it really interesting, is meta-selinux and
meta-security, both including recipes for libcap-ng, both at the same
version, both different again (though not nearly so bad as the example
above).  Which of these is more core?  I honestly don't know, since I
have used them together and separately on roughly equal number of
projects and I expect that trend will continue for a good long while for
me.  I think this might be a scenario where meta-selinux and
meta-security could make a case for trying to push this recipe further
toward the core.

Also, in light of this discovery, I'll be sending a patch for
meta-security soon, since the meta-selinux changes for libcap-ng are
almost certainly wanted there too.  But that's neither here nor there.

> 2) people feel they have to remove recipes from layers and make local
> copies.
> 
> And just so people know what seriously pisses me off :) Copying a recipe
> from a layer, updating it, and not sending the recipe to the layer they
> got the update from.

I don't know if that's what has happened with any of the recipes in the
layers I baby-sit, but if it is, it was absolutely not intentional and
I'll see what I can do about recovering a bit of karma by tidying up
about the place a bit.  :-)

-J.

> 
> Thanks for letting me vent,
> 
> Philip
-- 
-Joe MacDonald.
:wq

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 501 bytes --]

Re: Layer model doomed, unless we all work together

[Paul Eggleton]

On Wednesday 19 November 2014 22:34:41 Joe MacDonald wrote:
> [[yocto] Layer model doomed, unless we all work together] On 14.11.18 (Tue 
16:28) Philip Balister wrote:
> > I see a couple of issues we need to start talking about:
> > 
> > 1) recipes that need to move closer to core because a range of other
> > packages use them.
> 
> This was actually the only thing I thought needed further discussion
> (everything else should largely be a nod-in-agreement thing), as in some
> cases I'm not sure it's always clear what constitutes "closer to the
> core".  Poky and oe-core layers are pretty clear, but the next step
> beyond that isn't.  Are all layers hosted on git.yoctoproject.org
> inherhently "more core" than layers on git.openenbedded.org?  And
> there's a number of shells when you start including all of the github
> projects, setting aside other open-source project hosting services.

The answer to me there is certainly not. I've said this recently in other 
discussions, but I'll say it here anyway in case anyone else isn't sure - 
layers on git.yoctoproject.org should not be considered in any way more official 
than anywhere else solely based on them being hosted there. Anyone who wants 
to maintain and publish a layer suitable for others to use can get a 
repository on git.yoctoproject.org - there's no official review, validation, or 
acceptance criteria in the general case just for having a repository there 
(beyond being related to the project, that is).

To me this isn't so much about "closer to the core", it's about:

 1) sensible recipe groupings, e.g. meta-networking rather than a particular 
project's mixed recipes layer

 2) good maintenance, i.e. recipes are semi-regularly updated when upstream 
releases happen, fixed when needed to accommodate changes that happen in other 
layers it depends upon such as OE-Core, and the maintainer is reasonably 
responsive to patches or questions relating to the layer.

We need both of those things to encourage re-use, and if we have both then it 
doesn't really matter where a layer is hosted as long as it's listed in the 
layer index.
 
> Looking at the link you sent out based on Paul's suggestion, I see I'm
> actually on both sides of this equation, so yay!  And I'll limit the
> discussion to what's indexed there.
> 
> Here're my examples.
> 
> iscsi-initiator-utils: both in meta-networking nd meta-openstack.  Both
> at the same version currently but wildly different contents.
> meta-openstack is a git.yoctoproject.org project, so does that make it
> closer to the core?  I would think not, but as I recall there had been
> some comment about the openstack layer intending to limit layer
> dependencies outside Yocto core when it first appeared, so maybe making
> meta-networking a dependency is a non-starter for them.

So I've talked to Bruce about meta-openstack before, with particular regard to 
the number of python recipes that the layer ships and future overlap with 
meta-python, and apparently the policy there is not to pull in other layers 
for some dependencies with the aim of avoiding breakage on upgrades. I don't 
like that very much at all, to be frank, but I can at least understand it 
given how huge OpenStack is. It does of course mean that the overlap with that 
layer in particular is only going to increase as time goes on.
 
Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Layer model doomed, unless we all work together

[Joe MacDonald]

[-- Attachment #1: Type: text/plain, Size: 5295 bytes --]

Hi Paul,

[Re: [yocto] Layer model doomed, unless we all work together] On 14.11.20 (Thu 13:43) Paul Eggleton wrote:

> On Wednesday 19 November 2014 22:34:41 Joe MacDonald wrote:
> > [[yocto] Layer model doomed, unless we all work together] On 14.11.18 (Tue 
> 16:28) Philip Balister wrote:
> > > I see a couple of issues we need to start talking about:
> > > 
> > > 1) recipes that need to move closer to core because a range of other
> > > packages use them.
> > 
> > This was actually the only thing I thought needed further discussion
> > (everything else should largely be a nod-in-agreement thing), as in some
> > cases I'm not sure it's always clear what constitutes "closer to the
> > core".  Poky and oe-core layers are pretty clear, but the next step
> > beyond that isn't.  Are all layers hosted on git.yoctoproject.org
> > inherhently "more core" than layers on git.openenbedded.org?  And
> > there's a number of shells when you start including all of the github
> > projects, setting aside other open-source project hosting services.
> 
> The answer to me there is certainly not. I've said this recently in other 
> discussions, but I'll say it here anyway in case anyone else isn't sure - 
> layers on git.yoctoproject.org should not be considered in any way more official 
> than anywhere else solely based on them being hosted there. Anyone who wants 
> to maintain and publish a layer suitable for others to use can get a 
> repository on git.yoctoproject.org - there's no official review, validation, or 
> acceptance criteria in the general case just for having a repository there 
> (beyond being related to the project, that is).

That certainly makes sense to me, though I can respect if a project
maintainer wanted to try to keep everything they put there contained to
other projects hosted there.  Probably that's not even the case now,
I've not looked, but if I were starting a layer that I wanted to be
dependent only on, say, oe-core, and it was still useful to the
community at large, I would consider git.yoctoproject.org to be the most
sensible place to host it.

Regardless, I think your clarification makes perfect sense.

> To me this isn't so much about "closer to the core", it's about:
> 
>  1) sensible recipe groupings, e.g. meta-networking rather than a particular 
> project's mixed recipes layer
> 
>  2) good maintenance, i.e. recipes are semi-regularly updated when upstream 
> releases happen, fixed when needed to accommodate changes that happen in other 
> layers it depends upon such as OE-Core, and the maintainer is reasonably 
> responsive to patches or questions relating to the layer.
> 
> We need both of those things to encourage re-use, and if we have both then it 
> doesn't really matter where a layer is hosted as long as it's listed in the 
> layer index.

Fair enough.  Though I do think that a natural consequence of the
groupings in #1 above would tend to have recipes that appear in multiple
project's mixed recipe layers move toward the core.  But I also agree
that it needs to make sense for the layers involved.

> > Looking at the link you sent out based on Paul's suggestion, I see I'm
> > actually on both sides of this equation, so yay!  And I'll limit the
> > discussion to what's indexed there.
> > 
> > Here're my examples.
> > 
> > iscsi-initiator-utils: both in meta-networking nd meta-openstack.  Both
> > at the same version currently but wildly different contents.
> > meta-openstack is a git.yoctoproject.org project, so does that make it
> > closer to the core?  I would think not, but as I recall there had been
> > some comment about the openstack layer intending to limit layer
> > dependencies outside Yocto core when it first appeared, so maybe making
> > meta-networking a dependency is a non-starter for them.
> 
> So I've talked to Bruce about meta-openstack before, with particular regard to 
> the number of python recipes that the layer ships and future overlap with 
> meta-python, and apparently the policy there is not to pull in other layers 
> for some dependencies with the aim of avoiding breakage on upgrades. I don't 
> like that very much at all, to be frank, but I can at least understand it 
> given how huge OpenStack is. It does of course mean that the overlap with that 
> layer in particular is only going to increase as time goes on.

Hmm.  Okay, I don't want to veer too far off the topic, but what I'm
getting from this is that meta-openstack is a layer usable for building
OpenStack, but otherwise probably isn't an ideal base for other layers.
There's two obvious points of divergence between meta-networking recipes
and meta-openstack ones, both of which have the same priority, and it
sounds like there's going to be an increasing number of these with
meta-python.  I don't know that, given how large OpenStack is, adding
other layer dependencies would amount to more than a drop in the bucket,
but as a philosophical discussion rather than a technical one, I'll
stick to knowing to regularly check the index and time I'm looking at
recipes or working with a new layer.  That's probably really good advice
for anyone, honestly.

That's for the info, Paul.

-- 
-Joe MacDonald.
:wq

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 501 bytes --]