Moving to layered structure of openembedded metadata

Moving to layered structure of openembedded metadata

[Khem Raj]

Hi

In recent discussions there seems to be a lot of interest in moving
towards more layered structure in OE where we discussed machine
layers and arch based layers. There certainly are advantages to this
approach as discussed in other threads on mailing list
I think poky has already achieved that to a certain extent and has
made metadata changes
needed for such a structure. There are other polishing to metadata
which is beneficial in general like recipe licenses, gcc runtime
demystification etc. I would like to suggest that we adopt this
structure and use Poky as core layer which we
should always maintain in coherence and add the extra
machines,architectures and recipes as additional layers around
the base layer.

This will also help us to scale the project and
reduce the to and fro in merges as well as communities at large will
benefit by seamless flow
of patches and other contributions.

This will of course need a lot of work and I wanted to bring to
everyone to know if this would be a good approach moving
forward.

Thanks

-Khem

Re: Moving to layered structure of openembedded metadata

[Denys Dmytriyenko]

On Thu, Nov 04, 2010 at 04:34:42PM -0700, Khem Raj wrote:
> Hi
> 
> In recent discussions there seems to be a lot of interest in moving
> towards more layered structure in OE where we discussed machine
> layers and arch based layers. There certainly are advantages to this
> approach as discussed in other threads on mailing list
> I think poky has already achieved that to a certain extent and has
> made metadata changes
> needed for such a structure. There are other polishing to metadata
> which is beneficial in general like recipe licenses, gcc runtime
> demystification etc. I would like to suggest that we adopt this
> structure and use Poky as core layer which we
> should always maintain in coherence and add the extra
> machines,architectures and recipes as additional layers around
> the base layer.
> 
> This will also help us to scale the project and
> reduce the to and fro in merges as well as communities at large will
> benefit by seamless flow
> of patches and other contributions.
> 
> This will of course need a lot of work and I wanted to bring to
> everyone to know if this would be a good approach moving
> forward.

Khem,

That is already being discussed and decided. See the following previous topics 
on the mailing list...

The raw OEDEM meeting notes, where if you read carefully (and between the 
lines :) you can see the suggestion of using Poky as a new core being 
discussed:
http://article.gmane.org/gmane.comp.handhelds.openembedded/39108

Call for open discussion and future plans for advancing OpenEmbedded and 
Yocto Project:
http://thread.gmane.org/gmane.comp.handhelds.openembedded/39114

-- 
Denys

Re: Moving to layered structure of openembedded metadata

[Frans Meulenbroeks]

2010/11/5 Denys Dmytriyenko <denis@denix.org>:
> On Thu, Nov 04, 2010 at 04:34:42PM -0700, Khem Raj wrote:
>> Hi
>>
>> In recent discussions there seems to be a lot of interest in moving
>> towards more layered structure in OE where we discussed machine
>> layers and arch based layers. There certainly are advantages to this
>> approach as discussed in other threads on mailing list
>> I think poky has already achieved that to a certain extent and has
>> made metadata changes
>> needed for such a structure. There are other polishing to metadata
>> which is beneficial in general like recipe licenses, gcc runtime
>> demystification etc. I would like to suggest that we adopt this
>> structure and use Poky as core layer which we
>> should always maintain in coherence and add the extra
>> machines,architectures and recipes as additional layers around
>> the base layer.
>>
>> This will also help us to scale the project and
>> reduce the to and fro in merges as well as communities at large will
>> benefit by seamless flow
>> of patches and other contributions.
>>
>> This will of course need a lot of work and I wanted to bring to
>> everyone to know if this would be a good approach moving
>> forward.
>
> Khem,
>
> That is already being discussed and decided. See the following previous topics
> on the mailing list...
>
> The raw OEDEM meeting notes, where if you read carefully (and between the
> lines :) you can see the suggestion of using Poky as a new core being
> discussed:
> http://article.gmane.org/gmane.comp.handhelds.openembedded/39108
>
> Call for open discussion and future plans for advancing OpenEmbedded and
> Yocto Project:
> http://thread.gmane.org/gmane.comp.handhelds.openembedded/39114
>
> --
> Denys

Denys, I don't think it is decided yet.
At OEDEM it was suggested and discussed, but not really decided. (and
actually I doubt if OEDEM would be the right place to decide this,
especially in such a short timeframe without getting the rest of the
community involved).
And the call for open discussion is just that, a call for open discussion.

However, I do see consensus emerging, and personally I think the
proposal of Khem is a good idea.
Actually I already did a small test. It seems the most troublesome
issues in merging things are
- the legacy staging that still exists in quite some OE recipes
- bbclasses that are out of sync (or do not exist in yocto, but in
that case they could be in an OE overlay)
- toolchain related issues. E.g. for ppc there is no spe support yet
in yocto; however this is being worked on)

I know merging it will still be a lot of work and won't happen without fallout.
Then again I am more than willing to do my share of the work. My main
concern is how to deal with all the old stuff we accumulated.
In order to move to a layered structure with quality recipes we might
need to move them over one by one (fixing things like LICENSE while
we're at it).

Frans

Re: Moving to layered structure of openembedded metadata

[Stefan Schmidt]

Hello.

On Thu, 2010-11-04 at 21:00, Denys Dmytriyenko wrote:
> 
> That is already being discussed and decided. See the following previous topics 
> on the mailing list...

To clear this up a bit. There can't be a real decision for now. Not all
developers have been present at the OEDEM and we haven't discussed  every detail
their either. :)

Most, if not all, people liked the idea during the OEDEM discussions, but we
would need more input on problem that this may rise. As it stands I'm happy to
see Khem coming up to put this on the ml. :)

One I currently see is how we can keep it all working when the layers are
maintained in different places and what kid of combinational explosion we will
have with layers in different places.

regards
Stefan Schmidt

Re: Moving to layered structure of openembedded metadata

[Frans Meulenbroeks]

2010/11/5 Stefan Schmidt <stefan@datenfreihafen.org>:
> Hello.
>
> On Thu, 2010-11-04 at 21:00, Denys Dmytriyenko wrote:
>>
>> That is already being discussed and decided. See the following previous topics
>> on the mailing list...
>
> To clear this up a bit. There can't be a real decision for now. Not all
> developers have been present at the OEDEM and we haven't discussed  every detail
> their either. :)
>
> Most, if not all, people liked the idea during the OEDEM discussions, but we
> would need more input on problem that this may rise. As it stands I'm happy to
> see Khem coming up to put this on the ml. :)
>
> One I currently see is how we can keep it all working when the layers are
> maintained in different places and what kid of combinational explosion we will
> have with layers in different places.
>

Good points!

To keep things working I feel it is important to minimize the inter
layer dependencies (and make the ones that exist explicit).
My suggestion is that every layer has a maintainer/custodian/owner or
whatever name you want to give it who is responsible to keep the layer
in good shape.
Presumably this means we get a structure where there are two versions
of a layer: the last validated/integrated/accepted one and the working
one.

Frans

Re: Moving to layered structure of openembedded metadata

[Denys Dmytriyenko]

On Fri, Nov 05, 2010 at 08:34:48AM +0100, Stefan Schmidt wrote:
> Hello.
> 
> On Thu, 2010-11-04 at 21:00, Denys Dmytriyenko wrote:
> > 
> > That is already being discussed and decided. See the following previous topics 
> > on the mailing list...
> 
> To clear this up a bit. There can't be a real decision for now. Not all
> developers have been present at the OEDEM and we haven't discussed  every detail
> their either. :)
> 
> Most, if not all, people liked the idea during the OEDEM discussions, but we
> would need more input on problem that this may rise. As it stands I'm happy to
> see Khem coming up to put this on the ml. :)
> 
> One I currently see is how we can keep it all working when the layers are
> maintained in different places and what kid of combinational explosion we will
> have with layers in different places.

That is exactly what I said - being discussed and (being) decided. I didn't 
say it was or has been already decided. "Being" is not the same as "been"! :) 
I.e. present continuous instead of past tense.

And BTW, it may not be a simple decision after all - while technically Yocto 
seems quite a decent project and embracing it would be reasonable, many large 
companies are still having issues with it's governance model and past issues 
with Intel's other seemingly "open source" projects... The good news is that 
this side of the problem is now being actively looked at for any possible 
solutions, which would benefit all involved...

-- 
Denys

Re: Moving to layered structure of openembedded metadata

[Khem Raj]

On Fri, Nov 5, 2010 at 9:57 AM, Denys Dmytriyenko <denis@denix.org> wrote:
> On Fri, Nov 05, 2010 at 08:34:48AM +0100, Stefan Schmidt wrote:
>> Hello.
>>
>> On Thu, 2010-11-04 at 21:00, Denys Dmytriyenko wrote:
>> >
>> > That is already being discussed and decided. See the following previous topics
>> > on the mailing list...
>>
>> To clear this up a bit. There can't be a real decision for now. Not all
>> developers have been present at the OEDEM and we haven't discussed  every detail
>> their either. :)
>>
>> Most, if not all, people liked the idea during the OEDEM discussions, but we
>> would need more input on problem that this may rise. As it stands I'm happy to
>> see Khem coming up to put this on the ml. :)
>>
>> One I currently see is how we can keep it all working when the layers are
>> maintained in different places and what kid of combinational explosion we will
>> have with layers in different places.
>
> That is exactly what I said - being discussed and (being) decided. I didn't
> say it was or has been already decided. "Being" is not the same as "been"! :)
> I.e. present continuous instead of past tense.
>
> And BTW, it may not be a simple decision after all - while technically Yocto
> seems quite a decent project and embracing it would be reasonable, many large
> companies are still having issues with it's governance model and past issues
> with Intel's other seemingly "open source" projects... The good news is that
> this side of the problem is now being actively looked at for any possible
> solutions, which would benefit all involved...
>

My proposal is to align OE and poky and in turn for benefit of community.
Yocto as I understand has a bigger scope and uses OE/poky for the
build system that it uses. Which in
general is also good for OE development. If we design the layers well it will be
very portable and componentized. How yocto itself operates is probably
is a separate thing
they may make some good moves which benefits OE and thats what we
should be interested in
and concerned about and I am sure that if we do something that is
beneficial then yocto using poky
build system will definitely like it too.

> --
> Denys
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>