OpenEmbedded Core - Ready for extended users

OpenEmbedded Core - Ready for extended users

[Richard Purdie]

Hi,

The TSC has been tasked with working out how OE and Yocto could work
together and people have no doubt seen the discussions about OE-Core.

We're now pleased to formally announce that we feel OE-Core is ready for
exposure to more users and we'd like to encourage people to have a look
at it and experiment and develop with it.

Just to recap on the background for this, splitting up OE into
components has been a topic of discussion for a long time, certainly at
every OEDEM. Until now this hasn't happened partly due to resource
issues but also lack of tooling and the lack of a well thought out plan.
We now feel these issues have been addressed and that we have a good
plan in place. The promotion and use of layers is the only real way OE
can scale as a project yet stay maintainable and testable.

Things that aren't part of oe-core fall to the meta-oe layer which is
where we intend the current wider support contained in oe-devel to be
maintained. The people working on meta-oe so far would love to see more
contributions. Both repositories are currently working on a "pull" model
and people using them so far have felt a benefit to this for code
quality.

Discussion and development on oe-core is happening on the
openembedded-core mailing list since otherwise it would be hard to
filter out which patches and discussion were for which repository. We
look forward to seeing more people getting involved there!

(http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core)

Cheers,

Richard
(Communicating on behalf of the TSC)

Re: OpenEmbedded Core - Ready for extended users

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 16-05-11 13:26, Richard Purdie wrote:

> Things that aren't part of oe-core fall to the meta-oe layer which is
> where we intend the current wider support contained in oe-devel to be
> maintained. The people working on meta-oe so far would love to see more
> contributions. Both repositories are currently working on a "pull" model
> and people using them so far have felt a benefit to this for code
> quality.

The idea is that every layer has a top level README describing how to
contribute.

For meta-oe:
http://cgit.openembedded.org/cgit.cgi/meta-openembedded/tree/meta-oe/README

And for meta-angstrom:
http://git.angstrom-distribution.org/cgi-bin/cgit.cgi/meta-angstrom/tree/README

If someone wants to come up with a standard for the layer README files,
please do!

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFN0RFEMkyGM64RGpERAps4AJ9WHjtvVSCIkp8AvJ9GtLij/11RGgCePlMc
oMLNqa9vr5yd99HcV6aqrpY=
=Gb4s
-----END PGP SIGNATURE-----

Re: OpenEmbedded Core - Ready for extended users

[Frans Meulenbroeks]

2011/5/16 Richard Purdie <richard.purdie@linuxfoundation.org>

> Hi,
>
> The TSC has been tasked with working out how OE and Yocto could work
> together and people have no doubt seen the discussions about OE-Core.
>
> We're now pleased to formally announce that we feel OE-Core is ready for
> exposure to more users and we'd like to encourage people to have a look
> at it and experiment and develop with it.
>
> Just to recap on the background for this, splitting up OE into
> components has been a topic of discussion for a long time, certainly at
> every OEDEM. Until now this hasn't happened partly due to resource
> issues but also lack of tooling and the lack of a well thought out plan.
> We now feel these issues have been addressed and that we have a good
> plan in place. The promotion and use of layers is the only real way OE
> can scale as a project yet stay maintainable and testable.
>
> Things that aren't part of oe-core fall to the meta-oe layer which is
> where we intend the current wider support contained in oe-devel to be
> maintained. The people working on meta-oe so far would love to see more
> contributions. Both repositories are currently working on a "pull" model
> and people using them so far have felt a benefit to this for code
> quality.
>
> Discussion and development on oe-core is happening on the
> openembedded-core mailing list since otherwise it would be hard to
> filter out which patches and discussion were for which repository. We
> look forward to seeing more people getting involved there!
>
> (http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core)
>
> Cheers,
>
> Richard
> (Communicating on behalf of the TSC)
>
>
> Richard, thanks for the announcement.

I think the community could benefit if you add info (a pointer) in this
thread
- how to set up a build environment
- policies on what goes into oe-core and meta-oe and what not (and how to
deal with things that do not go into it)
- a migration scenario from the current repo to the new structure (what are
we doing with the existing recipes)
- how to add new distro's and bsp's

I know some (if not all) info has been sent around over time, but I feel it
would help if this announcement would carry the info to get people started
(and/or point to a web page that has the above info).

Best regards, Frans

Re: OpenEmbedded Core - Ready for extended users

[Phil Blundell]

On Mon, 2011-05-16 at 14:52 +0200, Frans Meulenbroeks wrote:
> I think the community could benefit if you add info (a pointer) in this
> thread
> - how to set up a build environment
> - policies on what goes into oe-core and meta-oe and what not (and how to
> deal with things that do not go into it)
> - a migration scenario from the current repo to the new structure (what are
> we doing with the existing recipes)
> - how to add new distro's and bsp's
> 
> I know some (if not all) info has been sent around over time, but I feel it
> would help if this announcement would carry the info to get people started
> (and/or point to a web page that has the above info).

Yes, agreed, I think that would be useful.  The build environment setup
is certainly a bit different to "classic" oe and it took me a while to
figure it out.  At the time there didn't seem to be much documentation
on how that stuff was meant to work, though with hindsight I now suspect
the Poky manual might have been the place to look.

p.

Re: OpenEmbedded Core - Ready for extended users

[Anders Darander]

On Mon, May 16, 2011 at 18:49, Phil Blundell <philb@gnu.org> wrote:
> On Mon, 2011-05-16 at 14:52 +0200, Frans Meulenbroeks wrote:
>> I think the community could benefit if you add info (a pointer) in this
>> thread
>> - how to set up a build environment
>> - policies on what goes into oe-core and meta-oe and what not (and how to
>> deal with things that do not go into it)

Yes, this is definitely needed. Which type of new
applications/libraries can get into meta-oe, and which are supposed to
be kept in higher layers...

In what cases can additional versions of the recipe in oe-core be
added? E.g. dbus is built using 'EXTRA_OECONF = "--with-x"', which at
least on head-less systems seems to be overkill. Still, it's
preferable (from my point of view) to use as much recipes as possible
from oe-core, instead of maintaining separate copy in a local layer.

>> - a migration scenario from the current repo to the new structure (what are
>> we doing with the existing recipes)

This is part of what I'm trying to get some time to do now. I've at
least got sofar as to get the build starting, however, before
proceeding further on, I've got to double check our local/modified
copies of a number of recipes (see e.g. the point on dbus above).

When I'm "finished" converting our distribution/repo; I might get some
time to try to write up a short text on this (unless someone already
has done this). However, this is probably a couple of weeks ahead, at
least.

>> I know some (if not all) info has been sent around over time, but I feel it
>> would help if this announcement would carry the info to get people started
>> (and/or point to a web page that has the above info).

Yes, an agregated / summarized source for this info is definitely needed.

> Yes, agreed, I think that would be useful.  The build environment setup
> is certainly a bit different to "classic" oe and it took me a while to
> figure it out.  At the time there didn't seem to be much documentation
> on how that stuff was meant to work, though with hindsight I now suspect
> the Poky manual might have been the place to look.

I guess that might be right. Currently the move that I'm doing will be
quite similar to the old oe-style. I've got quite good help by
studying the angstrom setup scripts. The Poky manual seems to use a
style where other meta-layers are installed in the poky-tree, however
I prefer a layout more similar to e.g. angstrom:

own-top level
|- conf/
|- bitbake/
|- openembedded-core/
|- meta-openembedded/

or the angstrom version:
own-top level
|- conf/
|- bitbake/      // not sure on which level bitbake is kept.
\ sources/
  |- openembedded-core
  |- meta-openembedded

Such a layout makes it a lot easier to keep the top-level in a
private/public repo, just referencing the oe-core and meta-oe layers
using git submodules.

Regards,
Anders Darander

Re: OpenEmbedded Core - Ready for extended users

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 17-05-11 08:23, Anders Darander wrote:
> On Mon, May 16, 2011 at 18:49, Phil Blundell <philb@gnu.org> wrote:
>> On Mon, 2011-05-16 at 14:52 +0200, Frans Meulenbroeks wrote:
>>> I think the community could benefit if you add info (a pointer) in this
>>> thread
>>> - how to set up a build environment
>>> - policies on what goes into oe-core and meta-oe and what not (and how to
>>> deal with things that do not go into it)
> 
> Yes, this is definitely needed. Which type of new
> applications/libraries can get into meta-oe, and which are supposed to
> be kept in higher layers...
> 
> In what cases can additional versions of the recipe in oe-core be
> added? E.g. dbus is built using 'EXTRA_OECONF = "--with-x"', which at
> least on head-less systems seems to be overkill.

Dbus is a really good example where the above knee-jerk reaction is
misplaced :) The --with-x only turns on building dbus-launch, which is x
specific, but that's the only bit that is. All the other dbus
subpackages lack X dependencies.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFN0hfyMkyGM64RGpERAmkdAJ96oZbdWIr0vN9+ys7FNBGoH2HtaACfVyTl
/5bZEgjnUvyLTFB9apyiFQc=
=XEmW
-----END PGP SIGNATURE-----

Re: OpenEmbedded Core - Ready for extended users

[Anders Darander]

On Tue, May 17, 2011 at 08:38, Koen Kooi <koen@dominion.thruhere.net> wrote:
> On 17-05-11 08:23, Anders Darander wrote:
>> Yes, this is definitely needed. Which type of new
>> applications/libraries can get into meta-oe, and which are supposed to
>> be kept in higher layers...
>>
>> In what cases can additional versions of the recipe in oe-core be
>> added? E.g. dbus is built using 'EXTRA_OECONF = "--with-x"', which at
>> least on head-less systems seems to be overkill.
>
> Dbus is a really good example where the above knee-jerk reaction is
> misplaced :) The --with-x only turns on building dbus-launch, which is x
> specific, but that's the only bit that is. All the other dbus
> subpackages lack X dependencies.

Yes, from that point, Dbus was not the optimal example, I agree...

Though, that was only one example of a package that has either runtime
or only build-time dependencies on X in some form or another. Earlier,
in the 2011.03-releases, we had systemd depending on gtk+, which has
been removed in the lates updates. Thats just another example.

In some cases, removing direct or indirect dependencies like these,
would be something that we'd like to see integrated upstreams. Of
course, we are willing to supply patches for that, after a discussion
on how such things should be handled. Is it desired for package Y or
isn't it. This could sometimes be to reduce the image size, but in
cases like dbus, it might only be to reduce build time and
build-directory size.

In other cases, I assume that we're going to continue keeping our
patchsets local, as I guess that we probably have quite uncommon
desires to get a certain packet working without e.g. X or some other
dependency.

Regards,
Anders

Re: OpenEmbedded Core - Ready for extended users

[Petr Štetiar]

Anders Darander <anders.darander@gmail.com> [2011-05-17 09:04:35]:

> Yes, from that point, Dbus was not the optimal example, I agree...
> 
> Though, that was only one example of a package that has either runtime
> or only build-time dependencies on X in some form or another. Earlier,
> in the 2011.03-releases, we had systemd depending on gtk+, which has
> been removed in the lates updates. Thats just another example.

Don't worry, you're not alone in that feeling, actually I think, that we're
quite a large group fighting with this :-) One day the image has 1700 tasks,
after one rebase within a week timespan it grows almost to 3000, don't mention
bluetooth and similar stuff etc. Well, I'm also looking forward to see, how it
will change with oe-core.

-- ynezz

Re: OpenEmbedded Core - Ready for extended users

[Richard Purdie]

On Tue, 2011-05-17 at 11:25 +0200, Petr Štetiar wrote:
> Anders Darander <anders.darander@gmail.com> [2011-05-17 09:04:35]:
> 
> > Yes, from that point, Dbus was not the optimal example, I agree...
> > 
> > Though, that was only one example of a package that has either runtime
> > or only build-time dependencies on X in some form or another. Earlier,
> > in the 2011.03-releases, we had systemd depending on gtk+, which has
> > been removed in the lates updates. Thats just another example.
> 
> Don't worry, you're not alone in that feeling, actually I think, that we're
> quite a large group fighting with this :-) One day the image has 1700 tasks,
> after one rebase within a week timespan it grows almost to 3000, don't mention
> bluetooth and similar stuff etc. Well, I'm also looking forward to see, how it
> will change with oe-core.

These are issues we need to address I think everyone agrees on that.
What we do need is some overall architectural way to handle it though
rather and doing it adhoc on a case by case basis. There has been some
discussion on the OE-Core list about it but I don't think we've reached
any conclusion.

Out of interesting are there any new proposals on how to handle this?
I'm interested in proposals that are thought through to details like how
distros handle differences and being able to identify which
configurations are used when presented with packages...

Cheers,

Richard

Re: OpenEmbedded Core - Ready for extended users

[Anders Darander]

On Tue, May 17, 2011 at 21:49, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Tue, 2011-05-17 at 11:25 +0200, Petr Štetiar wrote:
>> Anders Darander <anders.darander@gmail.com> [2011-05-17 09:04:35]:
>> > Though, that was only one example of a package that has either runtime
>> > or only build-time dependencies on X in some form or another. Earlier,
>> > in the 2011.03-releases, we had systemd depending on gtk+, which has
>> > been removed in the lates updates. Thats just another example.
>>
>> Don't worry, you're not alone in that feeling, actually I think, that we're
>> quite a large group fighting with this :-) One day the image has 1700 tasks,
>> after one rebase within a week timespan it grows almost to 3000, don't mention
>> bluetooth and similar stuff etc. Well, I'm also looking forward to see, how it
>> will change with oe-core.
>
> These are issues we need to address I think everyone agrees on that.
> What we do need is some overall architectural way to handle it though
> rather and doing it adhoc on a case by case basis. There has been some
> discussion on the OE-Core list about it but I don't think we've reached
> any conclusion.

I had some trouble locating these discussions on the oe-core list. Do
you have hint/link towards those discussions? I'd be interested to
read them and see what ideas has been discussed.

> Out of interesting are there any new proposals on how to handle this?
> I'm interested in proposals that are thought through to details like how
> distros handle differences and being able to identify which
> configurations are used when presented with packages...

Has there been any proposals earlier?

Sorry, no real ideas at the moment. The main alterniatives, I guess,
are to conditionally add depends and configure options based on
either:
* More variables. Although this could lead to _a lot more_ variables
in the end. (I.e. see the recent discussion on the oe-core list on
task-base etc.)
*  On if certain other recipes are selected. I'm not sure that this is
even possible? (I haven't looked that much into bitbake and recipe
parsing) This is solution is obviously a lot easier when using e.g.
Kconfig-based configurations, like Buildroot.

I'm not sure of which option is the best, or if there is another
better option...

Regards,
Anders