RFC: User configurable recipe features

RFC: User configurable recipe features

[Darren Hart]

As part of working on meta-tiny, I've come across a need (want?) to
present users with the ability to select some set of features in a local
configuration file that will impact the build of the image and a set of
recipes.

It is currently possible to affect which packages are installed in an
image with variables like POKY_EXTRA_INSTALL. What I'm not finding a way
to do is specify some set of features that will impact how a recipe is
built.

For example, a user may or may not want networking support or virtual
terminal support in their image. This impacts both the kernel and
busybox (at least). The linux-yocto infrastructure provides us with
config fragment functionality, something similar will need to be added
to busybox. Access to that is still bound to the machine config by means
of the SRC_URI machine override mechanism, but it would be useful to be
able to influence it from the image config or the user's local config.

For example, when building a tiny image I may decide I do not want VT
nor INET support. I might wish to specify this like this (by removing
them from the default features):

local.conf:
#CORE_IMAGE_TINY_FEATURES = "VT INET MDEV"
CORE_IMAGE_TINY_FEATURES = "MDEV"

I would want this to affect linux-yocto-tiny by dropping the vt.cfg and
inet.cfg fragments from the SRC_URI (or from the .scc descriptor files
assembled by the linux-yocto meta indrastructure).

Busybox would need a similar configuration mechanism, and would also
need to add a "no-vt-support.patch" patch to the SRC_URI to avoid a
bug/oversight in the busybox init routine.

I'd appreciate some help determining the proper bitbake way of doing
this. I want to avoid having to create a new machine.conf and/or recipes
for every possible combination of features that a user may want to turn
on or off.

Thanks,

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: RFC: User configurable recipe features

[McClintock Matthew-B29882]

On Mon, Oct 10, 2011 at 1:41 PM, Darren Hart <dvhart@linux.intel.com> wrote:
> I would want this to affect linux-yocto-tiny by dropping the vt.cfg and
> inet.cfg fragments from the SRC_URI (or from the .scc descriptor files
> assembled by the linux-yocto meta indrastructure).
>
> Busybox would need a similar configuration mechanism, and would also
> need to add a "no-vt-support.patch" patch to the SRC_URI to avoid a
> bug/oversight in the busybox init routine.

Just chiming in on this point - I think busybox will need something
very similar to config fragments. It has a lot of functionality that
should be configurable.

-M

Re: RFC: User configurable recipe features

[Darren Hart]

On 10/11/2011 08:53 AM, McClintock Matthew-B29882 wrote:
> On Mon, Oct 10, 2011 at 1:41 PM, Darren Hart <dvhart@linux.intel.com> wrote:
>> I would want this to affect linux-yocto-tiny by dropping the vt.cfg and
>> inet.cfg fragments from the SRC_URI (or from the .scc descriptor files
>> assembled by the linux-yocto meta indrastructure).
>>
>> Busybox would need a similar configuration mechanism, and would also
>> need to add a "no-vt-support.patch" patch to the SRC_URI to avoid a
>> bug/oversight in the busybox init routine.
> 
> Just chiming in on this point - I think busybox will need something
> very similar to config fragments. It has a lot of functionality that
> should be configurable.

Agreed, and it can probably use a very similar mechanism to what I've
been working to get into the upstream Linux kernel.

I've been looking at the uclibc recipe configuration functions. It looks
like it tries to solve a similar problem using feature lists and
programatically modifying the .config. More research needed there...

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: RFC: User configurable recipe features

[Richard Purdie]

On Mon, 2011-10-10 at 11:41 -0700, Darren Hart wrote:
> As part of working on meta-tiny, I've come across a need (want?) to
> present users with the ability to select some set of features in a local
> configuration file that will impact the build of the image and a set of
> recipes.
> 
> It is currently possible to affect which packages are installed in an
> image with variables like POKY_EXTRA_INSTALL. What I'm not finding a way
> to do is specify some set of features that will impact how a recipe is
> built.
> 
> For example, a user may or may not want networking support or virtual
> terminal support in their image. This impacts both the kernel and
> busybox (at least). The linux-yocto infrastructure provides us with
> config fragment functionality, something similar will need to be added
> to busybox. Access to that is still bound to the machine config by means
> of the SRC_URI machine override mechanism, but it would be useful to be
> able to influence it from the image config or the user's local config.
> 
> For example, when building a tiny image I may decide I do not want VT
> nor INET support. I might wish to specify this like this (by removing
> them from the default features):
> 
> local.conf:
> #CORE_IMAGE_TINY_FEATURES = "VT INET MDEV"
> CORE_IMAGE_TINY_FEATURES = "MDEV"
>
> I would want this to affect linux-yocto-tiny by dropping the vt.cfg and
> inet.cfg fragments from the SRC_URI (or from the .scc descriptor files
> assembled by the linux-yocto meta indrastructure).
> 
> Busybox would need a similar configuration mechanism, and would also
> need to add a "no-vt-support.patch" patch to the SRC_URI to avoid a
> bug/oversight in the busybox init routine.
> 
> I'd appreciate some help determining the proper bitbake way of doing
> this. I want to avoid having to create a new machine.conf and/or recipes
> for every possible combination of features that a user may want to turn
> on or off.

We have a few mechanisms around for this but its a difficult problem to
do totally generically since everyone has their own ideas about what
should/shouldn't happen.

One tricky aspect is that some people care about package feeds and the
output into those needs to be deterministic. This is why DISTRO_FEATURES
exist which state things like "does x11 make sense"? This means dbus may
or may not be compiled with X but given a set of policy decisions by the
distro, the output is determined.

Recently we've taken the idea of PACKAGECONFIG on board. This is recipe
level policy which can enable/disable features in a given recipe (e.g.
does gsteamer depend on and build flac or not?). Whilst we have a high
level setup for this for autotools recipes, this is probably something
we need to do a more custom implementation of for busybox and the
features you mention above would map well to this. It would be good to
have a standardised way of representing this (and we may also want to
look at moving the kernel feature control towards this variable name
too).

What we need to be really really careful about is getting the
namespacing right and your CORE_IMAGE_TINY_FEATURES = "VT INET MDEV"
example above scares me as it mixes up several different things. My
worry is for example trying to build two different versions of busybox
in the same tmpdir depending on what image you build, for example what
does "bitbake core-image-tiny core-image-sato" do?

Contrast this to some settings:

PACKAGECONFIG_pn-linux-yocto = "vt inet"
PACKAGECONFIG_pn-busybox = "mdev"

which then mean you have one set of configuration for these recipes and
its clear what the bitbake command above would result in.

One of the bigger problems we're going to have with tiny is its
effectively a different set of distro settings to our normal builds. The
side effect of that is that you couldn't share a tmpdir with a "big"
build but I'm not sure that is an issue in practise, we just need to do
it in a way which doesn't give us the nasty configuration corner cases.

So I guess what I'm saying is the end result of your work is likely a
"poky-tiny" distro setting which would take the "poky" distro but tweak
some pieces for really small images. It would need a separate tmpdir and
we should look in the PACKAGECONFIG variable direction for handling
recipe specific customisations...

Cheers,

Richard

Re: RFC: User configurable recipe features

[Tim Bird]

On 10/10/2011 11:41 AM, Darren Hart wrote:
> As part of working on meta-tiny, I've come across a need (want?) to
> present users with the ability to select some set of features in a local
> configuration file that will impact the build of the image and a set of
> recipes.

Can you tell me more about meta-tiny?  this is the first I've heard
about this (sorry if discussion went by on the mailing list and I
missed it), and I'm very interested.

I'm currently doing some size-related work for Sony (including
some work to support 4K stacks).

Thanks,
 -- Tim
=============================
Tim Bird
Architecture Group Chair, CE Workgroup of the Linux Foundation
Senior Staff Engineer, Sony Network Entertainment
=============================

Re: RFC: User configurable recipe features

[Khem Raj]

On Tue, Oct 11, 2011 at 3:15 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Mon, 2011-10-10 at 11:41 -0700, Darren Hart wrote:
>> As part of working on meta-tiny, I've come across a need (want?) to
>> present users with the ability to select some set of features in a local
>> configuration file that will impact the build of the image and a set of
>> recipes.
>>
>> It is currently possible to affect which packages are installed in an
>> image with variables like POKY_EXTRA_INSTALL. What I'm not finding a way
>> to do is specify some set of features that will impact how a recipe is
>> built.
>>
>> For example, a user may or may not want networking support or virtual
>> terminal support in their image. This impacts both the kernel and
>> busybox (at least). The linux-yocto infrastructure provides us with
>> config fragment functionality, something similar will need to be added
>> to busybox. Access to that is still bound to the machine config by means
>> of the SRC_URI machine override mechanism, but it would be useful to be
>> able to influence it from the image config or the user's local config.
>>
>> For example, when building a tiny image I may decide I do not want VT
>> nor INET support. I might wish to specify this like this (by removing
>> them from the default features):
>>
>> local.conf:
>> #CORE_IMAGE_TINY_FEATURES = "VT INET MDEV"
>> CORE_IMAGE_TINY_FEATURES = "MDEV"
>>
>> I would want this to affect linux-yocto-tiny by dropping the vt.cfg and
>> inet.cfg fragments from the SRC_URI (or from the .scc descriptor files
>> assembled by the linux-yocto meta indrastructure).
>>
>> Busybox would need a similar configuration mechanism, and would also
>> need to add a "no-vt-support.patch" patch to the SRC_URI to avoid a
>> bug/oversight in the busybox init routine.
>>
>> I'd appreciate some help determining the proper bitbake way of doing
>> this. I want to avoid having to create a new machine.conf and/or recipes
>> for every possible combination of features that a user may want to turn
>> on or off.
>
> We have a few mechanisms around for this but its a difficult problem to
> do totally generically since everyone has their own ideas about what
> should/shouldn't happen.
>
> One tricky aspect is that some people care about package feeds and the
> output into those needs to be deterministic. This is why DISTRO_FEATURES
> exist which state things like "does x11 make sense"? This means dbus may
> or may not be compiled with X but given a set of policy decisions by the
> distro, the output is determined.
>
> Recently we've taken the idea of PACKAGECONFIG on board. This is recipe
> level policy which can enable/disable features in a given recipe (e.g.
> does gsteamer depend on and build flac or not?). Whilst we have a high
> level setup for this for autotools recipes, this is probably something
> we need to do a more custom implementation of for busybox and the
> features you mention above would map well to this. It would be good to
> have a standardised way of representing this (and we may also want to
> look at moving the kernel feature control towards this variable name
> too).
>
> What we need to be really really careful about is getting the
> namespacing right and your CORE_IMAGE_TINY_FEATURES = "VT INET MDEV"
> example above scares me as it mixes up several different things. My
> worry is for example trying to build two different versions of busybox
> in the same tmpdir depending on what image you build, for example what
> does "bitbake core-image-tiny core-image-sato" do?
>
> Contrast this to some settings:
>
> PACKAGECONFIG_pn-linux-yocto = "vt inet"
> PACKAGECONFIG_pn-busybox = "mdev"
>
> which then mean you have one set of configuration for these recipes and
> its clear what the bitbake command above would result in.
>
> One of the bigger problems we're going to have with tiny is its
> effectively a different set of distro settings to our normal builds. The
> side effect of that is that you couldn't share a tmpdir with a "big"
> build but I'm not sure that is an issue in practise, we just need to do
> it in a way which doesn't give us the nasty configuration corner cases.
>
> So I guess what I'm saying is the end result of your work is likely a
> "poky-tiny" distro setting which would take the "poky" distro but tweak
> some pieces for really small images. It would need a separate tmpdir and
> we should look in the PACKAGECONFIG variable direction for handling
> recipe specific customisations...
>

FWIW I agree it seems like a new distro to me. I think if we add more to mix
it just will complicate the customizations and may even make it
difficult to share
things.

> Cheers,
>
> Richard
>
>
>
>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>

Re: RFC: User configurable recipe features

[Philip Balister]

On 10/11/2011 07:51 PM, Khem Raj wrote:
> On Tue, Oct 11, 2011 at 3:15 PM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>> On Mon, 2011-10-10 at 11:41 -0700, Darren Hart wrote:
>>> As part of working on meta-tiny, I've come across a need (want?) to
>>> present users with the ability to select some set of features in a local
>>> configuration file that will impact the build of the image and a set of
>>> recipes.
>>>
>>> It is currently possible to affect which packages are installed in an
>>> image with variables like POKY_EXTRA_INSTALL. What I'm not finding a way
>>> to do is specify some set of features that will impact how a recipe is
>>> built.
>>>
>>> For example, a user may or may not want networking support or virtual
>>> terminal support in their image. This impacts both the kernel and
>>> busybox (at least). The linux-yocto infrastructure provides us with
>>> config fragment functionality, something similar will need to be added
>>> to busybox. Access to that is still bound to the machine config by means
>>> of the SRC_URI machine override mechanism, but it would be useful to be
>>> able to influence it from the image config or the user's local config.
>>>
>>> For example, when building a tiny image I may decide I do not want VT
>>> nor INET support. I might wish to specify this like this (by removing
>>> them from the default features):
>>>
>>> local.conf:
>>> #CORE_IMAGE_TINY_FEATURES = "VT INET MDEV"
>>> CORE_IMAGE_TINY_FEATURES = "MDEV"
>>>
>>> I would want this to affect linux-yocto-tiny by dropping the vt.cfg and
>>> inet.cfg fragments from the SRC_URI (or from the .scc descriptor files
>>> assembled by the linux-yocto meta indrastructure).
>>>
>>> Busybox would need a similar configuration mechanism, and would also
>>> need to add a "no-vt-support.patch" patch to the SRC_URI to avoid a
>>> bug/oversight in the busybox init routine.
>>>
>>> I'd appreciate some help determining the proper bitbake way of doing
>>> this. I want to avoid having to create a new machine.conf and/or recipes
>>> for every possible combination of features that a user may want to turn
>>> on or off.
>>
>> We have a few mechanisms around for this but its a difficult problem to
>> do totally generically since everyone has their own ideas about what
>> should/shouldn't happen.
>>
>> One tricky aspect is that some people care about package feeds and the
>> output into those needs to be deterministic. This is why DISTRO_FEATURES
>> exist which state things like "does x11 make sense"? This means dbus may
>> or may not be compiled with X but given a set of policy decisions by the
>> distro, the output is determined.
>>
>> Recently we've taken the idea of PACKAGECONFIG on board. This is recipe
>> level policy which can enable/disable features in a given recipe (e.g.
>> does gsteamer depend on and build flac or not?). Whilst we have a high
>> level setup for this for autotools recipes, this is probably something
>> we need to do a more custom implementation of for busybox and the
>> features you mention above would map well to this. It would be good to
>> have a standardised way of representing this (and we may also want to
>> look at moving the kernel feature control towards this variable name
>> too).
>>
>> What we need to be really really careful about is getting the
>> namespacing right and your CORE_IMAGE_TINY_FEATURES = "VT INET MDEV"
>> example above scares me as it mixes up several different things. My
>> worry is for example trying to build two different versions of busybox
>> in the same tmpdir depending on what image you build, for example what
>> does "bitbake core-image-tiny core-image-sato" do?
>>
>> Contrast this to some settings:
>>
>> PACKAGECONFIG_pn-linux-yocto = "vt inet"
>> PACKAGECONFIG_pn-busybox = "mdev"
>>
>> which then mean you have one set of configuration for these recipes and
>> its clear what the bitbake command above would result in.
>>
>> One of the bigger problems we're going to have with tiny is its
>> effectively a different set of distro settings to our normal builds. The
>> side effect of that is that you couldn't share a tmpdir with a "big"
>> build but I'm not sure that is an issue in practise, we just need to do
>> it in a way which doesn't give us the nasty configuration corner cases.
>>
>> So I guess what I'm saying is the end result of your work is likely a
>> "poky-tiny" distro setting which would take the "poky" distro but tweak
>> some pieces for really small images. It would need a separate tmpdir and
>> we should look in the PACKAGECONFIG variable direction for handling
>> recipe specific customisations...
>>
> 
> FWIW I agree it seems like a new distro to me. I think if we add more to mix
> it just will complicate the customizations and may even make it
> difficult to share
> things.

Along the same line of thought, does this overlap with the micro distro?

Philip

Re: RFC: User configurable recipe features

[Darren Hart]

On 10/11/2011 03:15 PM, Richard Purdie wrote:
> On Mon, 2011-10-10 at 11:41 -0700, Darren Hart wrote:
>> As part of working on meta-tiny, I've come across a need (want?) to
>> present users with the ability to select some set of features in a local
>> configuration file that will impact the build of the image and a set of
>> recipes.
>>
>> It is currently possible to affect which packages are installed in an
>> image with variables like POKY_EXTRA_INSTALL. What I'm not finding a way
>> to do is specify some set of features that will impact how a recipe is
>> built.
>>
>> For example, a user may or may not want networking support or virtual
>> terminal support in their image. This impacts both the kernel and
>> busybox (at least). The linux-yocto infrastructure provides us with
>> config fragment functionality, something similar will need to be added
>> to busybox. Access to that is still bound to the machine config by means
>> of the SRC_URI machine override mechanism, but it would be useful to be
>> able to influence it from the image config or the user's local config.
>>
>> For example, when building a tiny image I may decide I do not want VT
>> nor INET support. I might wish to specify this like this (by removing
>> them from the default features):
>>
>> local.conf:
>> #CORE_IMAGE_TINY_FEATURES = "VT INET MDEV"
>> CORE_IMAGE_TINY_FEATURES = "MDEV"
>>
>> I would want this to affect linux-yocto-tiny by dropping the vt.cfg and
>> inet.cfg fragments from the SRC_URI (or from the .scc descriptor files
>> assembled by the linux-yocto meta indrastructure).
>>
>> Busybox would need a similar configuration mechanism, and would also
>> need to add a "no-vt-support.patch" patch to the SRC_URI to avoid a
>> bug/oversight in the busybox init routine.
>>
>> I'd appreciate some help determining the proper bitbake way of doing
>> this. I want to avoid having to create a new machine.conf and/or recipes
>> for every possible combination of features that a user may want to turn
>> on or off.
> 

Hi RP,

Thanks for the thoughts, gives me some places to look for examples and
flesh this idea out more.

> We have a few mechanisms around for this but its a difficult problem to
> do totally generically since everyone has their own ideas about what
> should/shouldn't happen.
> 
> One tricky aspect is that some people care about package feeds and the
> output into those needs to be deterministic. This is why DISTRO_FEATURES
> exist which state things like "does x11 make sense"? This means dbus may
> or may not be compiled with X but given a set of policy decisions by the
> distro, the output is determined.
> 
> Recently we've taken the idea of PACKAGECONFIG on board. This is recipe
> level policy which can enable/disable features in a given recipe (e.g.
> does gsteamer depend on and build flac or not?). Whilst we have a high
> level setup for this for autotools recipes, this is probably something
> we need to do a more custom implementation of for busybox and the
> features you mention above would map well to this. It would be good to
> have a standardised way of representing this (and we may also want to
> look at moving the kernel feature control towards this variable name
> too).
> 
> What we need to be really really careful about is getting the
> namespacing right and your CORE_IMAGE_TINY_FEATURES = "VT INET MDEV"
> example above scares me as it mixes up several different things. My

Sure, I'm not attached to it, it was just for illustration.

> worry is for example trying to build two different versions of busybox
> in the same tmpdir depending on what image you build, for example what
> does "bitbake core-image-tiny core-image-sato" do?

Yes, I see the problem. I notice that uclibc uses a different tmp space
that the eglibc build.

> 
> Contrast this to some settings:
> 
> PACKAGECONFIG_pn-linux-yocto = "vt inet"
> PACKAGECONFIG_pn-busybox = "mdev"

OK, but it would need to be:

PACKAGECONFIG_pn-linux-yocto = "vt inet"
PACKAGECONFIG_pn-busybox = "vt inet mdev"

Since busybox needs to disable various vt support configs if the kernel
doesn't support vt. If don't support networking in the kernel, we might
as well disable it in busybox (and eglibc or uclibc for that matter).
The idea behind the single features variable was to avoid having to keep
them all in sync. Perhaps the busybox one could also read
PACKAGECONFIG_pn-virtual/kernel, but if they have explicitly different
settings - because someone wants a generic busybox recipe for multiple
kernels for example - I'm not sure how we would reconcile that.

> 
> which then mean you have one set of configuration for these recipes and
> its clear what the bitbake command above would result in.
> 
> One of the bigger problems we're going to have with tiny is its
> effectively a different set of distro settings to our normal builds. The
> side effect of that is that you couldn't share a tmpdir with a "big"
> build but I'm not sure that is an issue in practise, we just need to do
> it in a way which doesn't give us the nasty configuration corner cases.

For now it has been working alright. I wrote a task-core-tiny task and a
core-image-tiny image recipe which just install less stuff, so those
work out fine, and get us all the way down to a 3.2M rootfs (down from
11M with minimal) without rewriting recipes or reconfiguring them. So we
can do a lot in the existing tmp space.

Obviously with reconfigured recipes or altered sources, we'd either need
a new tmp space, or perhaps renamed recipes. I already use
linux-yocto-tiny, perhaps busybox-tiny would make sense (although it
sounds a tad redundant ;). I don't much like the -tiny recipes as they
duplicate code - and the PACKAGECONFIG should do all we need.

> 
> So I guess what I'm saying is the end result of your work is likely a
> "poky-tiny" distro setting which would take the "poky" distro but tweak
> some pieces for really small images. It would need a separate tmpdir and
> we should look in the PACKAGECONFIG variable direction for handling
> recipe specific customisations...

Thanks RP,

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: RFC: User configurable recipe features

[Darren Hart]

On 10/11/2011 05:18 PM, Philip Balister wrote:
> On 10/11/2011 07:51 PM, Khem Raj wrote:
>> On Tue, Oct 11, 2011 at 3:15 PM, Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>>> On Mon, 2011-10-10 at 11:41 -0700, Darren Hart wrote:
>>>> As part of working on meta-tiny, I've come across a need (want?) to
>>>> present users with the ability to select some set of features in a local
>>>> configuration file that will impact the build of the image and a set of
>>>> recipes.
>>>>
>>>> It is currently possible to affect which packages are installed in an
>>>> image with variables like POKY_EXTRA_INSTALL. What I'm not finding a way
>>>> to do is specify some set of features that will impact how a recipe is
>>>> built.
>>>>
>>>> For example, a user may or may not want networking support or virtual
>>>> terminal support in their image. This impacts both the kernel and
>>>> busybox (at least). The linux-yocto infrastructure provides us with
>>>> config fragment functionality, something similar will need to be added
>>>> to busybox. Access to that is still bound to the machine config by means
>>>> of the SRC_URI machine override mechanism, but it would be useful to be
>>>> able to influence it from the image config or the user's local config.
>>>>
>>>> For example, when building a tiny image I may decide I do not want VT
>>>> nor INET support. I might wish to specify this like this (by removing
>>>> them from the default features):
>>>>
>>>> local.conf:
>>>> #CORE_IMAGE_TINY_FEATURES = "VT INET MDEV"
>>>> CORE_IMAGE_TINY_FEATURES = "MDEV"
>>>>
>>>> I would want this to affect linux-yocto-tiny by dropping the vt.cfg and
>>>> inet.cfg fragments from the SRC_URI (or from the .scc descriptor files
>>>> assembled by the linux-yocto meta indrastructure).
>>>>
>>>> Busybox would need a similar configuration mechanism, and would also
>>>> need to add a "no-vt-support.patch" patch to the SRC_URI to avoid a
>>>> bug/oversight in the busybox init routine.
>>>>
>>>> I'd appreciate some help determining the proper bitbake way of doing
>>>> this. I want to avoid having to create a new machine.conf and/or recipes
>>>> for every possible combination of features that a user may want to turn
>>>> on or off.
>>>
>>> We have a few mechanisms around for this but its a difficult problem to
>>> do totally generically since everyone has their own ideas about what
>>> should/shouldn't happen.
>>>
>>> One tricky aspect is that some people care about package feeds and the
>>> output into those needs to be deterministic. This is why DISTRO_FEATURES
>>> exist which state things like "does x11 make sense"? This means dbus may
>>> or may not be compiled with X but given a set of policy decisions by the
>>> distro, the output is determined.
>>>
>>> Recently we've taken the idea of PACKAGECONFIG on board. This is recipe
>>> level policy which can enable/disable features in a given recipe (e.g.
>>> does gsteamer depend on and build flac or not?). Whilst we have a high
>>> level setup for this for autotools recipes, this is probably something
>>> we need to do a more custom implementation of for busybox and the
>>> features you mention above would map well to this. It would be good to
>>> have a standardised way of representing this (and we may also want to
>>> look at moving the kernel feature control towards this variable name
>>> too).
>>>
>>> What we need to be really really careful about is getting the
>>> namespacing right and your CORE_IMAGE_TINY_FEATURES = "VT INET MDEV"
>>> example above scares me as it mixes up several different things. My
>>> worry is for example trying to build two different versions of busybox
>>> in the same tmpdir depending on what image you build, for example what
>>> does "bitbake core-image-tiny core-image-sato" do?
>>>
>>> Contrast this to some settings:
>>>
>>> PACKAGECONFIG_pn-linux-yocto = "vt inet"
>>> PACKAGECONFIG_pn-busybox = "mdev"
>>>
>>> which then mean you have one set of configuration for these recipes and
>>> its clear what the bitbake command above would result in.
>>>
>>> One of the bigger problems we're going to have with tiny is its
>>> effectively a different set of distro settings to our normal builds. The
>>> side effect of that is that you couldn't share a tmpdir with a "big"
>>> build but I'm not sure that is an issue in practise, we just need to do
>>> it in a way which doesn't give us the nasty configuration corner cases.
>>>
>>> So I guess what I'm saying is the end result of your work is likely a
>>> "poky-tiny" distro setting which would take the "poky" distro but tweak
>>> some pieces for really small images. It would need a separate tmpdir and
>>> we should look in the PACKAGECONFIG variable direction for handling
>>> recipe specific customisations...
>>>
>>
>> FWIW I agree it seems like a new distro to me. I think if we add more to mix
>> it just will complicate the customizations and may even make it
>> difficult to share
>> things.
> 
> Along the same line of thought, does this overlap with the micro distro?
> 

Hrm... the micro distro? What's this?

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: RFC: User configurable recipe features

[Koen Kooi]

Op 12 okt. 2011, om 17:41 heeft Darren Hart het volgende geschreven:

> 
> 
> On 10/11/2011 05:18 PM, Philip Balister wrote:
>> On 10/11/2011 07:51 PM, Khem Raj wrote:
>>> On Tue, Oct 11, 2011 at 3:15 PM, Richard Purdie
>>> <richard.purdie@linuxfoundation.org> wrote:
>>>> On Mon, 2011-10-10 at 11:41 -0700, Darren Hart wrote:
>>>>> As part of working on meta-tiny, I've come across a need (want?) to
>>>>> present users with the ability to select some set of features in a local
>>>>> configuration file that will impact the build of the image and a set of
>>>>> recipes.
>>>>> 
>>>>> It is currently possible to affect which packages are installed in an
>>>>> image with variables like POKY_EXTRA_INSTALL. What I'm not finding a way
>>>>> to do is specify some set of features that will impact how a recipe is
>>>>> built.
>>>>> 
>>>>> For example, a user may or may not want networking support or virtual
>>>>> terminal support in their image. This impacts both the kernel and
>>>>> busybox (at least). The linux-yocto infrastructure provides us with
>>>>> config fragment functionality, something similar will need to be added
>>>>> to busybox. Access to that is still bound to the machine config by means
>>>>> of the SRC_URI machine override mechanism, but it would be useful to be
>>>>> able to influence it from the image config or the user's local config.
>>>>> 
>>>>> For example, when building a tiny image I may decide I do not want VT
>>>>> nor INET support. I might wish to specify this like this (by removing
>>>>> them from the default features):
>>>>> 
>>>>> local.conf:
>>>>> #CORE_IMAGE_TINY_FEATURES = "VT INET MDEV"
>>>>> CORE_IMAGE_TINY_FEATURES = "MDEV"
>>>>> 
>>>>> I would want this to affect linux-yocto-tiny by dropping the vt.cfg and
>>>>> inet.cfg fragments from the SRC_URI (or from the .scc descriptor files
>>>>> assembled by the linux-yocto meta indrastructure).
>>>>> 
>>>>> Busybox would need a similar configuration mechanism, and would also
>>>>> need to add a "no-vt-support.patch" patch to the SRC_URI to avoid a
>>>>> bug/oversight in the busybox init routine.
>>>>> 
>>>>> I'd appreciate some help determining the proper bitbake way of doing
>>>>> this. I want to avoid having to create a new machine.conf and/or recipes
>>>>> for every possible combination of features that a user may want to turn
>>>>> on or off.
>>>> 
>>>> We have a few mechanisms around for this but its a difficult problem to
>>>> do totally generically since everyone has their own ideas about what
>>>> should/shouldn't happen.
>>>> 
>>>> One tricky aspect is that some people care about package feeds and the
>>>> output into those needs to be deterministic. This is why DISTRO_FEATURES
>>>> exist which state things like "does x11 make sense"? This means dbus may
>>>> or may not be compiled with X but given a set of policy decisions by the
>>>> distro, the output is determined.
>>>> 
>>>> Recently we've taken the idea of PACKAGECONFIG on board. This is recipe
>>>> level policy which can enable/disable features in a given recipe (e.g.
>>>> does gsteamer depend on and build flac or not?). Whilst we have a high
>>>> level setup for this for autotools recipes, this is probably something
>>>> we need to do a more custom implementation of for busybox and the
>>>> features you mention above would map well to this. It would be good to
>>>> have a standardised way of representing this (and we may also want to
>>>> look at moving the kernel feature control towards this variable name
>>>> too).
>>>> 
>>>> What we need to be really really careful about is getting the
>>>> namespacing right and your CORE_IMAGE_TINY_FEATURES = "VT INET MDEV"
>>>> example above scares me as it mixes up several different things. My
>>>> worry is for example trying to build two different versions of busybox
>>>> in the same tmpdir depending on what image you build, for example what
>>>> does "bitbake core-image-tiny core-image-sato" do?
>>>> 
>>>> Contrast this to some settings:
>>>> 
>>>> PACKAGECONFIG_pn-linux-yocto = "vt inet"
>>>> PACKAGECONFIG_pn-busybox = "mdev"
>>>> 
>>>> which then mean you have one set of configuration for these recipes and
>>>> its clear what the bitbake command above would result in.
>>>> 
>>>> One of the bigger problems we're going to have with tiny is its
>>>> effectively a different set of distro settings to our normal builds. The
>>>> side effect of that is that you couldn't share a tmpdir with a "big"
>>>> build but I'm not sure that is an issue in practise, we just need to do
>>>> it in a way which doesn't give us the nasty configuration corner cases.
>>>> 
>>>> So I guess what I'm saying is the end result of your work is likely a
>>>> "poky-tiny" distro setting which would take the "poky" distro but tweak
>>>> some pieces for really small images. It would need a separate tmpdir and
>>>> we should look in the PACKAGECONFIG variable direction for handling
>>>> recipe specific customisations...
>>>> 
>>> 
>>> FWIW I agree it seems like a new distro to me. I think if we add more to mix
>>> it just will complicate the customizations and may even make it
>>> difficult to share
>>> things.
>> 
>> Along the same line of thought, does this overlap with the micro distro?
>> 
> 
> Hrm... the micro distro? What's this?

http://cgit.openembedded.org/meta-micro/

Re: RFC: User configurable recipe features

[Paul Eggleton]

On Wednesday 12 October 2011 16:41:56 Darren Hart wrote:
> Hrm... the micro distro? What's this?

http://cgit.openembedded.org/meta-micro/

It's maintained by Phil Blundell. I believe he's been pretty successful in 
reducing size of the resulting image whilst keeping the system fairly 
functional. He is using uclibc however.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: RFC: User configurable recipe features

[Darren Hart]

Hi Tim,

On 10/11/2011 04:49 PM, Tim Bird wrote:
> On 10/10/2011 11:41 AM, Darren Hart wrote:
>> As part of working on meta-tiny, I've come across a need (want?) to
>> present users with the ability to select some set of features in a local
>> configuration file that will impact the build of the image and a set of
>> recipes.
> 
> Can you tell me more about meta-tiny?  this is the first I've heard
> about this (sorry if discussion went by on the mailing list and I
> missed it), and I'm very interested.


Tim,

I'll be presenting things in more detail at ELCE, Friday @ 3:45 PM in
the Kepler room. The summary is that I have received a lot of questions
along the lines of "How small of an image can I build with Yocto?".
core-image-minimal does a decent job at a small general purpose image,
but it still has a lot of baggage that a truly size conscious developer
doesn't need for a custom BSP.

meta-tiny is my experimental layer where I'm looking at what we can
build with our existing sources and infrastructure. I've found that we
can cut the image size to about 10% of core-image-minimal without
changes to source code, but dropping a lot of functionality. We can get
to something like 20% while still maintaining ipv4 networking.

This "recipe features" thread stems from this work. Before I can
integrate something like this into Yocto, it needs a more suitable user
exposed configuration mechanism.

I'm working from a public git repository, but it isn't in sync with my
local development branch as things are changing by the minute still. I
will be pushing this to a Yocto hosted git repository in time for ELCE,
but it will still be development/experimental. If you are interested in
seeing what is there now, you can have a look to get an idea of what I'm
doing. Please note this tree WILL rebase, it WILL fail to build, it WILL
destroy your tmp dir, your sstate, and waste your time as you try and
pick through the various options to get something that builds and boots
;-) Also, this repo will disappear in the next few days, but if you just
want to see it, it's here:

http://git.infradead.org/users/dvhart/meta-tiny.git

> 
> I'm currently doing some size-related work for Sony (including
> some work to support 4K stacks).

I'd like to have a look - is there anything out there I can read through?

Post ELCE, I'd like to look at more source level changes we can make. In
particular I'd like to see the >200k bzImage increase for ipv4 come
down. In addition to that, I'm hoping to take a closer look at dynamic
memory usage in the Linux kernel for small systems.

Thanks,

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: RFC: User configurable recipe features

[Darren Hart]

On 10/12/2011 08:52 AM, Paul Eggleton wrote:
> On Wednesday 12 October 2011 16:41:56 Darren Hart wrote:
>> Hrm... the micro distro? What's this?
> 
> http://cgit.openembedded.org/meta-micro/
> 
> It's maintained by Phil Blundell. I believe he's been pretty successful in 
> reducing size of the resulting image whilst keeping the system fairly 
> functional. He is using uclibc however.

I wish I had run across this a few months ago. meta-micro in fact has
both an eglibc and a uclibc distro definition. I haven't tried a build,
but it does seem to be attempting the same basic thing as meta-tiny -
only in a more generally usable way.

This is fine as the end result of meta-tiny is likely to be some
improved configuration mechanisms for things like busybox, libc, and the
kernel, which should prove useful to meta-micro as well.

Thanks,

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: RFC: User configurable recipe features

[Darren Hart]

On 10/11/2011 04:49 PM, Tim Bird wrote:
> On 10/10/2011 11:41 AM, Darren Hart wrote:
>> As part of working on meta-tiny, I've come across a need (want?) to
>> present users with the ability to select some set of features in a local
>> configuration file that will impact the build of the image and a set of
>> recipes.
> 
> Can you tell me more about meta-tiny?  this is the first I've heard
> about this (sorry if discussion went by on the mailing list and I
> missed it), and I'm very interested.
> 
> I'm currently doing some size-related work for Sony (including
> some work to support 4K stacks).
> 

Perhaps while I have the attention of a few interested parties, it would
be a good time for a poll. I'm interested in your motivation for smaller
images.

Are you building SoC's with memory on die and needing to keep the memory
footprint down to save precious die real-estate?

Are you looking at creating mass-market products and saving a few
pennies on the flash storage translates to real money, so you want to
minimize the physical size?

Are you concerned with boot time, and have connected larger image sizes
with longer boot times?

Is there another motivating factor for your interest in small images?

Thanks,

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: RFC: User configurable recipe features

[Tim Bird]

On 10/12/2011 09:59 AM, Darren Hart wrote:
> On 10/11/2011 04:49 PM, Tim Bird wrote:
>> On 10/10/2011 11:41 AM, Darren Hart wrote:
>>> As part of working on meta-tiny, I've come across a need (want?) to
>>> present users with the ability to select some set of features in a local
>>> configuration file that will impact the build of the image and a set of
>>> recipes.
>>
>> Can you tell me more about meta-tiny?  this is the first I've heard
>> about this (sorry if discussion went by on the mailing list and I
>> missed it), and I'm very interested.
>>
>> I'm currently doing some size-related work for Sony (including
>> some work to support 4K stacks).
>>
> 
> Perhaps while I have the attention of a few interested parties, it would
> be a good time for a poll. I'm interested in your motivation for smaller
> images.
> 
> Are you building SoC's with memory on die and needing to keep the memory
> footprint down to save precious die real-estate?

No.

> Are you looking at creating mass-market products and saving a few
> pennies on the flash storage translates to real money, so you want to
> minimize the physical size?

Yes. - this is the primary one for me/Sony.  We have dual-core/dual-OS cameras
where the ram budget for the Linux side of the device is only 10 meg.
We are working on medical products with flash budgets of 8 meg and ram
budgets of 4 meg.

We are currently doing a fair amount with execute in place, to conserve
RAM versus flash.

> Are you concerned with boot time, and have connected larger image sizes
> with longer boot times?
Yes.

> Is there another motivating factor for your interest in small images?
Smaller images should theoretically run faster, due to less pressure
on CPU caches.  I don't know of any meaningful measurements of this,
but it's an interesting possibility.

Also, it's nice to minimize the memory footprint to reduce power
consumption.
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Workgroup of the Linux Foundation
Senior Staff Engineer, Sony Network Entertainment
=============================

Re: RFC: User configurable recipe features

[Khem Raj]

On 10/12/2011 8:55 AM, Darren Hart wrote:
> Hi Tim,
>
> On 10/11/2011 04:49 PM, Tim Bird wrote:
>> On 10/10/2011 11:41 AM, Darren Hart wrote:
>>> As part of working on meta-tiny, I've come across a need (want?) to
>>> present users with the ability to select some set of features in a local
>>> configuration file that will impact the build of the image and a set of
>>> recipes.
>>
>> Can you tell me more about meta-tiny?  this is the first I've heard
>> about this (sorry if discussion went by on the mailing list and I
>> missed it), and I'm very interested.
>
>
> Tim,
>
> I'll be presenting things in more detail at ELCE, Friday @ 3:45 PM in
> the Kepler room. The summary is that I have received a lot of questions
> along the lines of "How small of an image can I build with Yocto?".

I guess yocto needs to define another profile(distro) to really 
demonstrate how small it can get. There are other distros based on 
oe-core e.g. micro and even slugos where the image sizes are really 
small slugos/uclibc image is around 2.7M eglibc based image is 3.5M


> core-image-minimal does a decent job at a small general purpose image,
> but it still has a lot of baggage that a truly size conscious developer
> doesn't need for a custom BSP.
>
> meta-tiny is my experimental layer where I'm looking at what we can
> build with our existing sources and infrastructure. I've found that we
> can cut the image size to about 10% of core-image-minimal without
> changes to source code, but dropping a lot of functionality. We can get
> to something like 20% while still maintaining ipv4 networking.
>
> This "recipe features" thread stems from this work. Before I can
> integrate something like this into Yocto, it needs a more suitable user
> exposed configuration mechanism.

this has a downside. I still favour that distro/machine should be one to 
dictate the features which should control recipe feature enable/disable 
behavior.

>
> I'm working from a public git repository, but it isn't in sync with my
> local development branch as things are changing by the minute still. I
> will be pushing this to a Yocto hosted git repository in time for ELCE,
> but it will still be development/experimental. If you are interested in
> seeing what is there now, you can have a look to get an idea of what I'm
> doing. Please note this tree WILL rebase, it WILL fail to build, it WILL
> destroy your tmp dir, your sstate, and waste your time as you try and
> pick through the various options to get something that builds and boots
> ;-) Also, this repo will disappear in the next few days, but if you just
> want to see it, it's here:
>
> http://git.infradead.org/users/dvhart/meta-tiny.git
>
>>
>> I'm currently doing some size-related work for Sony (including
>> some work to support 4K stacks).
>
> I'd like to have a look - is there anything out there I can read through?
>
> Post ELCE, I'd like to look at more source level changes we can make. In
> particular I'd like to see the>200k bzImage increase for ipv4 come
> down. In addition to that, I'm hoping to take a closer look at dynamic
> memory usage in the Linux kernel for small systems.
>
> Thanks,
>

Re: RFC: User configurable recipe features

[Tim Bird]

On 10/12/2011 08:55 AM, Darren Hart wrote:
> meta-tiny is my experimental layer where I'm looking at what we can
> build with our existing sources and infrastructure. I've found that we
> can cut the image size to about 10% of core-image-minimal without
> changes to source code, but dropping a lot of functionality. We can get
> to something like 20% while still maintaining ipv4 networking.

This sounds cool.

> This "recipe features" thread stems from this work. Before I can
> integrate something like this into Yocto, it needs a more suitable user
> exposed configuration mechanism.
> 
> I'm working from a public git repository, but it isn't in sync with my
> local development branch as things are changing by the minute still. I
> will be pushing this to a Yocto hosted git repository in time for ELCE,
> but it will still be development/experimental. If you are interested in
> seeing what is there now, you can have a look to get an idea of what I'm
> doing. Please note this tree WILL rebase, it WILL fail to build, it WILL
> destroy your tmp dir, your sstate, and waste your time as you try and
> pick through the various options to get something that builds and boots
> ;-) Also, this repo will disappear in the next few days, but if you just
> want to see it, it's here:
> 
> http://git.infradead.org/users/dvhart/meta-tiny.git
> 
>>
>> I'm currently doing some size-related work for Sony (including
>> some work to support 4K stacks).
> 
> I'd like to have a look - is there anything out there I can read through?
Not at the moment.  I've been meaning to send my patch
for 4K stack support for ARM to linux-ARM, but haven't had
the time.  I'll see if I can push it out this week (but it
might collide with the 3.2 merge window, and I wouldn't
expect it to go straight in without it dwelling in someone's
-next tree for a bit.)

In our cameras, this saves about 300K of kernel memory.

I have no problem sending that to you if you'd like.  Just let
me know.  If mainline doesn't take it straight-away for some reason,
I'll probably hand it off to the linux-tiny project, which is back
in full swing.

> Post ELCE, I'd like to look at more source level changes we can make. In
> particular I'd like to see the >200k bzImage increase for ipv4 come
> down. In addition to that, I'm hoping to take a closer look at dynamic
> memory usage in the Linux kernel for small systems.

I have some patches for monitoring slab allocations that are helpful
for this (also something I should have already mainlined, but didn't
get to... ;-(

 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Workgroup of the Linux Foundation
Senior Staff Engineer, Sony Network Entertainment
=============================

Re: RFC: User configurable recipe features

[William Mills]

On 10/12/2011 12:59 PM, Darren Hart wrote:
>
>
> On 10/11/2011 04:49 PM, Tim Bird wrote:
>> On 10/10/2011 11:41 AM, Darren Hart wrote:
>>> As part of working on meta-tiny, I've come across a need (want?) to
>>> present users with the ability to select some set of features in a local
>>> configuration file that will impact the build of the image and a set of
>>> recipes.
>>
>> Can you tell me more about meta-tiny?  this is the first I've heard
>> about this (sorry if discussion went by on the mailing list and I
>> missed it), and I'm very interested.
>>
>> I'm currently doing some size-related work for Sony (including
>> some work to support 4K stacks).
>>
>
> Perhaps while I have the attention of a few interested parties, it would
> be a good time for a poll. I'm interested in your motivation for smaller
> images.

I am not on the front line but here is my take.

>
> Are you building SoC's with memory on die and needing to keep the memory
> footprint down to save precious die real-estate?

no.

However we sometimes run the full fileystem from DDR so there is no 
flash per say.  (Full filesystem image gets transfered at boot time). 
Board with N devices, all with DDR and ethernet and nothing else.  Don't 
use NFS for latency/performance consistency issues.

>
> Are you looking at creating mass-market products and saving a few
> pennies on the flash storage translates to real money, so you want to
> minimize the physical size?

Yes.

We still see flash size == to RAM size or 1/2 RAM.
32 MB RAM, 16 MB Flash
64 MB RAM, 32 MB flash

We had one EVM w/ 8 MB SPI flash and we fit a fairly decent headless 
system into that.  Not sure if that was customer driven or bad EVM 
definition.  Fortunity that EVM also had a MMC/SD card reader for the > 
8MB use case.

Of course once you go above a certain size (128/256 MB), people go 
MMC/SD and then minimums go way up.  1 GB is probably as cheap as 512MB.

We have not seen the situation Tim talked about with RAM < NVMEM (at 
least not for several years)

If you start getting into the oversize microcontroller area then you 
start getting very small.  However you are also probably only a kernel, 
uclibc, a very cut down busybox and one or two custom apps.

>
> Are you concerned with boot time, and have connected larger image sizes
> with longer boot times?
>

That is a nice benefit but not the main objective.

Better use case for seep cases is a partitioned NVMEM configuration with 
small faster flash (32MB-256MB parallel NAND) and slower larger storage 
(1+ GB SD Card etc).

With this config do you:
* treat the NAND as a dynamic FS cache
* use it to store the "core" os (/ but not /usr or /opt)
* use it to store only readahead like data

> Is there another motivating factor for your interest in small images?
>
> Thanks,
>

Re: RFC: User configurable recipe features

[Koen Kooi]

Op 12 okt. 2011, om 20:44 heeft Khem Raj het volgende geschreven:

> On 10/12/2011 8:55 AM, Darren Hart wrote:
>> Hi Tim,
>> 
>> On 10/11/2011 04:49 PM, Tim Bird wrote:
>>> On 10/10/2011 11:41 AM, Darren Hart wrote:
>>>> As part of working on meta-tiny, I've come across a need (want?) to
>>>> present users with the ability to select some set of features in a local
>>>> configuration file that will impact the build of the image and a set of
>>>> recipes.
>>> 
>>> Can you tell me more about meta-tiny?  this is the first I've heard
>>> about this (sorry if discussion went by on the mailing list and I
>>> missed it), and I'm very interested.
>> 
>> 
>> Tim,
>> 
>> I'll be presenting things in more detail at ELCE, Friday @ 3:45 PM in
>> the Kepler room. The summary is that I have received a lot of questions
>> along the lines of "How small of an image can I build with Yocto?".
> 
> I guess yocto needs to define another profile(distro) to really demonstrate how small it can get. There are other distros based on oe-core e.g. micro and even slugos where the image sizes are really small slugos/uclibc image is around 2.7M eglibc based image is 3.5M

And don't forget angstrom, that scales all the way down to an fbdev gui fit inside 80kiB.

Re: RFC: User configurable recipe features

[Khem Raj]

On 10/12/2011 12:30 PM, Koen Kooi wrote:
>
> Op 12 okt. 2011, om 20:44 heeft Khem Raj het volgende geschreven:
>
>> On 10/12/2011 8:55 AM, Darren Hart wrote:
>>> Hi Tim,
>>>
>>> On 10/11/2011 04:49 PM, Tim Bird wrote:
>>>> On 10/10/2011 11:41 AM, Darren Hart wrote:
>>>>> As part of working on meta-tiny, I've come across a need (want?) to
>>>>> present users with the ability to select some set of features in a local
>>>>> configuration file that will impact the build of the image and a set of
>>>>> recipes.
>>>>
>>>> Can you tell me more about meta-tiny?  this is the first I've heard
>>>> about this (sorry if discussion went by on the mailing list and I
>>>> missed it), and I'm very interested.
>>>
>>>
>>> Tim,
>>>
>>> I'll be presenting things in more detail at ELCE, Friday @ 3:45 PM in
>>> the Kepler room. The summary is that I have received a lot of questions
>>> along the lines of "How small of an image can I build with Yocto?".
>>
>> I guess yocto needs to define another profile(distro) to really demonstrate how small it can get. There are other distros based on oe-core e.g. micro and even slugos where the image sizes are really small slugos/uclibc image is around 2.7M eglibc based image is 3.5M
>
> And don't forget angstrom, that scales all the way down to an fbdev gui fit inside 80kiB.

indeed.

Re: RFC: User configurable recipe features

[Darren Hart]

On 10/12/2011 11:44 AM, Khem Raj wrote:
> On 10/12/2011 8:55 AM, Darren Hart wrote:
>> Hi Tim,
>>
>> On 10/11/2011 04:49 PM, Tim Bird wrote:
>>> On 10/10/2011 11:41 AM, Darren Hart wrote:
>>>> As part of working on meta-tiny, I've come across a need (want?) to
>>>> present users with the ability to select some set of features in a local
>>>> configuration file that will impact the build of the image and a set of
>>>> recipes.
>>>
>>> Can you tell me more about meta-tiny?  this is the first I've heard
>>> about this (sorry if discussion went by on the mailing list and I
>>> missed it), and I'm very interested.
>>
>>
>> Tim,
>>
>> I'll be presenting things in more detail at ELCE, Friday @ 3:45 PM in
>> the Kepler room. The summary is that I have received a lot of questions
>> along the lines of "How small of an image can I build with Yocto?".
> 
> I guess yocto needs to define another profile(distro) to really 
> demonstrate how small it can get. There are other distros based on 
> oe-core e.g. micro and even slugos where the image sizes are really 
> small slugos/uclibc image is around 2.7M eglibc based image is 3.5M

That's right about what I'm seeing as well, and it's not particularly
difficult to get there. It was mostly just time consuming. If we can
make it easy to start there and let people build up from there, I think
taht would be a win.

> 
>> core-image-minimal does a decent job at a small general purpose image,
>> but it still has a lot of baggage that a truly size conscious developer
>> doesn't need for a custom BSP.
>>
>> meta-tiny is my experimental layer where I'm looking at what we can
>> build with our existing sources and infrastructure. I've found that we
>> can cut the image size to about 10% of core-image-minimal without
>> changes to source code, but dropping a lot of functionality. We can get
>> to something like 20% while still maintaining ipv4 networking.
>>
>> This "recipe features" thread stems from this work. Before I can
>> integrate something like this into Yocto, it needs a more suitable user
>> exposed configuration mechanism.
> 
> this has a downside. I still favour that distro/machine should be one to 
> dictate the features which should control recipe feature enable/disable 
> behavior.


I don't disagree. The configuration mechanism is still needed though.
For example, the only way to configure bitbake as things stand is with a
new defconfig. By adding config fragment management and some sort of
selectable feature set, each distro definition could provide it's own
config and build from the same recipe. Much the way uclibc and eglibc
can be configured now (and use distro defined variables).

Thanks,

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: RFC: User configurable recipe features

[Jack Mitchell]

On 12/10/2011 17:59, Darren Hart wrote:
>
> On 10/11/2011 04:49 PM, Tim Bird wrote:
>> On 10/10/2011 11:41 AM, Darren Hart wrote:
>>> As part of working on meta-tiny, I've come across a need (want?) to
>>> present users with the ability to select some set of features in a 
>>> local
>>> configuration file that will impact the build of the image and a set of
>>> recipes.
>> Can you tell me more about meta-tiny?  this is the first I've heard
>> about this (sorry if discussion went by on the mailing list and I
>> missed it), and I'm very interested.
>>
>> I'm currently doing some size-related work for Sony (including
>> some work to support 4K stacks).
>>
> Perhaps while I have the attention of a few interested parties, it would
> be a good time for a poll. I'm interested in your motivation for smaller
> images.
>
> Are you building SoC's with memory on die and needing to keep the memory
> footprint down to save precious die real-estate?

no

>
> Are you looking at creating mass-market products and saving a few
> pennies on the flash storage translates to real money, so you want to
> minimize the physical size?

no

>
> Are you concerned with boot time, and have connected larger image sizes
> with longer boot times?

I am concerned with boot time, but don't believe it is image size that 
ramps it up.

>
> Is there another motivating factor for your interest in small images?

Yes, a smaller system which is easier to check, build and maintain. In 
my office I am the leading driver for using linux in a team of 3 (two 
software, one fpga developer) so the less time I spend building, 
rebuilding and checking features I don't need, to ensure they don't 
comprimise the stability of the system, the more faith they have in the 
system I'm putting forward.

>
> Thanks,
>

Re: RFC: User configurable recipe features

[William Mills]

On 10/13/2011 04:30 AM, Jack Mitchell wrote:
> On 12/10/2011 17:59, Darren Hart wrote:
>>
>> On 10/11/2011 04:49 PM, Tim Bird wrote:
>>> On 10/10/2011 11:41 AM, Darren Hart wrote:
>>>> As part of working on meta-tiny, I've come across a need (want?) to
>>>> present users with the ability to select some set of features in a 
>>>> local
>>>> configuration file that will impact the build of the image and a 
>>>> set of
>>>> recipes.
>>> Can you tell me more about meta-tiny?  this is the first I've heard
>>> about this (sorry if discussion went by on the mailing list and I
>>> missed it), and I'm very interested.
>>>
>>> I'm currently doing some size-related work for Sony (including
>>> some work to support 4K stacks).
>>>
>> Perhaps while I have the attention of a few interested parties, it would
>> be a good time for a poll. I'm interested in your motivation for smaller
>> images.
>>
>> Are you building SoC's with memory on die and needing to keep the memory
>> footprint down to save precious die real-estate?
>
> no
>
>>
>> Are you looking at creating mass-market products and saving a few
>> pennies on the flash storage translates to real money, so you want to
>> minimize the physical size?
>
> no
>
>>
>> Are you concerned with boot time, and have connected larger image sizes
>> with longer boot times?
>
> I am concerned with boot time, but don't believe it is image size that 
> ramps it up.
>
>>
>> Is there another motivating factor for your interest in small images?
>
> Yes, a smaller system which is easier to check, build and maintain. In 
> my office I am the leading driver for using linux in a team of 3 (two 
> software, one fpga developer) so the less time I spend building, 
> rebuilding and checking features I don't need, to ensure they don't 
> comprimise the stability of the system, the more faith they have in 
> the system I'm putting forward.
>

Ahhh, nice one Jack.

I had a similar thought this morning.  As the target system gets smaller 
the tolerance for spending X amount of time building non-target code 
goes down and the expectation of being able to use a "modest machine" 
goes up.

What is a modest machine?  Yocto quotes build times for a "refernce 
machine" that is pretty up to date and not on the low end.  To me, a 
modest machine is the laptop Mom & Dad bought "Stacy" when she graduated 
from High School and went off to College.  Stacy is now a junior and is 
exploring embedded Linux.  This might be an i3 2 GB machine.  A China 
based startup may also give its engineers modest machines.  I think many 
TI'ers would claim they have been stuck on modest machines for long periods.

So If a sato image takes 1 hour to build on the reference machine it may 
take 4 hours to build on a modest machine.  Of that time perhaps 1 hr is 
spent building host side stuff.

If your image is just kernel, busybox, and uclibc you probably only 
spend 1/2 hour building that on a modest machine.  Question is does 
oe-core/poky still make you build 1 hr worth of host stuff?

I know Richard's answer will be shared state but I want to see how that 
really works out.  This is an area we plan on playing with over the next 
release cycle.


>> Thanks,
>>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: RFC: User configurable recipe features

[Khem Raj]

On Thu, Oct 13, 2011 at 11:33 AM, William Mills <wmills@ti.com> wrote:
>
>
> On 10/13/2011 04:30 AM, Jack Mitchell wrote:
>>
>> On 12/10/2011 17:59, Darren Hart wrote:
>>>
>>> On 10/11/2011 04:49 PM, Tim Bird wrote:
>>>>
>>>> On 10/10/2011 11:41 AM, Darren Hart wrote:
>>>>>
>>>>> As part of working on meta-tiny, I've come across a need (want?) to
>>>>> present users with the ability to select some set of features in a
>>>>> local
>>>>> configuration file that will impact the build of the image and a set of
>>>>> recipes.
>>>>
>>>> Can you tell me more about meta-tiny?  this is the first I've heard
>>>> about this (sorry if discussion went by on the mailing list and I
>>>> missed it), and I'm very interested.
>>>>
>>>> I'm currently doing some size-related work for Sony (including
>>>> some work to support 4K stacks).
>>>>
>>> Perhaps while I have the attention of a few interested parties, it would
>>> be a good time for a poll. I'm interested in your motivation for smaller
>>> images.
>>>
>>> Are you building SoC's with memory on die and needing to keep the memory
>>> footprint down to save precious die real-estate?
>>
>> no
>>
>>>
>>> Are you looking at creating mass-market products and saving a few
>>> pennies on the flash storage translates to real money, so you want to
>>> minimize the physical size?
>>
>> no
>>
>>>
>>> Are you concerned with boot time, and have connected larger image sizes
>>> with longer boot times?
>>
>> I am concerned with boot time, but don't believe it is image size that
>> ramps it up.
>>
>>>
>>> Is there another motivating factor for your interest in small images?
>>
>> Yes, a smaller system which is easier to check, build and maintain. In my
>> office I am the leading driver for using linux in a team of 3 (two software,
>> one fpga developer) so the less time I spend building, rebuilding and
>> checking features I don't need, to ensure they don't comprimise the
>> stability of the system, the more faith they have in the system I'm putting
>> forward.
>>
>
> Ahhh, nice one Jack.
>
> I had a similar thought this morning.  As the target system gets smaller the
> tolerance for spending X amount of time building non-target code goes down
> and the expectation of being able to use a "modest machine" goes up.
>
> What is a modest machine?  Yocto quotes build times for a "refernce machine"
> that is pretty up to date and not on the low end.  To me, a modest machine
> is the laptop Mom & Dad bought "Stacy" when she graduated from High School
> and went off to College.  Stacy is now a junior and is exploring embedded
> Linux.  This might be an i3 2 GB machine.  A China based startup may also
> give its engineers modest machines.  I think many TI'ers would claim they
> have been stuck on modest machines for long periods.
>
> So If a sato image takes 1 hour to build on the reference machine it may
> take 4 hours to build on a modest machine.  Of that time perhaps 1 hr is
> spent building host side stuff.
>
> If your image is just kernel, busybox, and uclibc you probably only spend
> 1/2 hour building that on a modest machine.  Question is does oe-core/poky
> still make you build 1 hr worth of host stuff?

Usually some of the host distros have sane(for OE sense) versions of
host packages. You might experiment using
ASSUME_PROVIDED for the build host packages this will then use the
packages from your build system
however the results you get might be different than others that's one
reason for host packages

A lot of stuff in OE environment is done to ensure predictability of
builds on different host distros
which is a good thing IMO.

>
> I know Richard's answer will be shared state but I want to see how that
> really works out.  This is an area we plan on playing with over the next
> release cycle.
>
>
>>> Thanks,
>>>
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
>
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto
>