Variable locality too restricted.

Variable locality too restricted.

[David Nyström]

Hi, 

I'm trying to create a setup for qemuppc.

Goals:
core-image-minimal + virtual/kernel. 
core-image-minimal + virtual/kernel with modified .config for debug flavoured kernel.

Problems:
PREFERRED_PROVIDER seems to be restricted to local.conf, distro and machine.

Creating another machine for my debug flavored kernel feel wrong. creating a new distro even worse.
And I don't want 'sed' my local.conf files everytime I want to switch flavors.

Possible solution:
The possibility to set PREFERRED_PROVIDER in image.bb would solve my problem.

Whats the right way to get this done ?

Best Regards,
David

Re: Variable locality too restricted.

[Chris Larson]

On Wed, Dec 7, 2011 at 6:44 AM, David Nyström <David.Nystrom@enea.com> wrote:
> I'm trying to create a setup for qemuppc.
>
> Goals:
> core-image-minimal + virtual/kernel.
> core-image-minimal + virtual/kernel with modified .config for debug flavoured kernel.
>
> Problems:
> PREFERRED_PROVIDER seems to be restricted to local.conf, distro and machine.


This is incorrect. It can be anywhere in the configuration metadata.
bitbake.conf includes a variety of config files, not just
distro/machine. Read that to see other existing files which get
included. Further, you could create a .conf/.inc which you include
from your machine .conf, if your goal is just to avoid duplication.
-- 
Christopher Larson
clarson at kergoth dot com
Senior Software Engineer, Mentor Graphics

Re: Variable locality too restricted.

[Paul Eggleton]

On Wednesday 07 December 2011 06:52:25 Chris Larson wrote:
> On Wed, Dec 7, 2011 at 6:44 AM, David Nyström <David.Nystrom@enea.com> 
wrote:
> > I'm trying to create a setup for qemuppc.
> > 
> > Goals:
> > core-image-minimal + virtual/kernel.
> > core-image-minimal + virtual/kernel with modified .config for debug
> > flavoured kernel.
> > 
> > Problems:
> > PREFERRED_PROVIDER seems to be restricted to local.conf, distro and
> > machine.
> This is incorrect. It can be anywhere in the configuration metadata.
> bitbake.conf includes a variety of config files, not just
> distro/machine. Read that to see other existing files which get
> included. Further, you could create a .conf/.inc which you include
> from your machine .conf, if your goal is just to avoid duplication.

Changing MACHINE has other implications though; do we not have any other way 
to switch out the kernel on a per-image basis?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Variable locality too restricted.

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 1162 bytes --]


Op 7 dec. 2011, om 15:05 heeft Paul Eggleton het volgende geschreven:

> On Wednesday 07 December 2011 06:52:25 Chris Larson wrote:
>> On Wed, Dec 7, 2011 at 6:44 AM, David Nyström <David.Nystrom@enea.com> 
> wrote:
>>> I'm trying to create a setup for qemuppc.
>>> 
>>> Goals:
>>> core-image-minimal + virtual/kernel.
>>> core-image-minimal + virtual/kernel with modified .config for debug
>>> flavoured kernel.
>>> 
>>> Problems:
>>> PREFERRED_PROVIDER seems to be restricted to local.conf, distro and
>>> machine.
>> This is incorrect. It can be anywhere in the configuration metadata.
>> bitbake.conf includes a variety of config files, not just
>> distro/machine. Read that to see other existing files which get
>> included. Further, you could create a .conf/.inc which you include
>> from your machine .conf, if your goal is just to avoid duplication.
> 
> Changing MACHINE has other implications though; do we not have any other way 
> to switch out the kernel on a per-image basis?

And how would that work from a packaging perspective? AIUI changing DISTRO/MACHINE flags in an image is a recipe for failure.

regards,

Koen

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 169 bytes --]

Re: Variable locality too restricted.

[Paul Eggleton]

On Wednesday 07 December 2011 15:07:54 Koen Kooi wrote:
> > Changing MACHINE has other implications though; do we not have any other
> > way to switch out the kernel on a per-image basis?
> 
> And how would that work from a packaging perspective? AIUI changing
> DISTRO/MACHINE flags in an image is a recipe for failure.

I'm not suggesting changing the kernel configuration for the existing kernel; 
naturally you would need a different kernel recipe. Surely you should just be 
able to have a different kernel that is built and installed for a different 
image file? No invalidated packages necessary.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Variable locality too restricted.

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 996 bytes --]


Op 7 dec. 2011, om 15:10 heeft Paul Eggleton het volgende geschreven:

> On Wednesday 07 December 2011 15:07:54 Koen Kooi wrote:
>>> Changing MACHINE has other implications though; do we not have any other
>>> way to switch out the kernel on a per-image basis?
>> 
>> And how would that work from a packaging perspective? AIUI changing
>> DISTRO/MACHINE flags in an image is a recipe for failure.
> 
> I'm not suggesting changing the kernel configuration for the existing kernel; 
> naturally you would need a different kernel recipe. Surely you should just be 
> able to have a different kernel that is built and installed for a different 
> image file? No invalidated packages necessary.

A kernel buld will generate packages and thanks to kernel.bbclass they will have similar names. So what will happen when I do:

bitbake foo-image bar-image

? Will it build 2 kernels? Which kernel will end up being packaged or will there be a mix of both in deploy?

regards,

Koen

[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 169 bytes --]

Re: Variable locality too restricted.

[Paul Eggleton]

On Wednesday 07 December 2011 15:14:28 Koen Kooi wrote:
> A kernel buld will generate packages and thanks to kernel.bbclass they will
> have similar names.

Right, I had missed that important point. I guess the only way at the moment 
is to have separate machines.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Variable locality too restricted.

[David Nyström]

________________________________________
From: Koen Kooi [koen@dominion.thruhere.net]
Sent: Wednesday, December 07, 2011 15:07
To: Paul Eggleton
Cc: Chris Larson; David Nyström; yocto@yoctoproject.org
Subject: Re: [yocto] Variable locality too restricted.

Op 7 dec. 2011, om 15:05 heeft Paul Eggleton het volgende geschreven:

> On Wednesday 07 December 2011 06:52:25 Chris Larson wrote:
>> On Wed, Dec 7, 2011 at 6:44 AM, David Nyström <David.Nystrom@enea.com>
> wrote:
>>> I'm trying to create a setup for qemuppc.
>>>
>>> Goals:
>>> core-image-minimal + virtual/kernel.
>>> core-image-minimal + virtual/kernel with modified .config for debug
>>> flavoured kernel.
>>>
>>> Problems:
>>> PREFERRED_PROVIDER seems to be restricted to local.conf, distro and
>>> machine.
>> This is incorrect. It can be anywhere in the configuration metadata.
>> bitbake.conf includes a variety of config files, not just
>> distro/machine. Read that to see other existing files which get
>> included. Further, you could create a .conf/.inc which you include
>> from your machine .conf, if your goal is just to avoid duplication.
>
> Changing MACHINE has other implications though; do we not have any other way
> to switch out the kernel on a per-image basis?

And how would that work from a packaging perspective? AIUI changing DISTRO/MACHINE flags in an image is a recipe for failure.

regards,

Koen
Op 7 dec. 2011, om 15:10 heeft Paul Eggleton het volgende geschreven:

--
> On Wednesday 07 December 2011 15:07:54 Koen Kooi wrote:
>>> Changing MACHINE has other implications though; do we not have any other
>>> way to switch out the kernel on a per-image basis?
>>
>> And how would that work from a packaging perspective? AIUI changing
>> DISTRO/MACHINE flags in an image is a recipe for failure.
>
> I'm not suggesting changing the kernel configuration for the existing kernel;
> naturally you would need a different kernel recipe. Surely you should just be
> able to have a different kernel that is built and installed for a different
> image file? No invalidated packages necessary.

A kernel buld will generate packages and thanks to kernel.bbclass they will have similar names. So what will happen when I do:

bitbake foo-image bar-image

? Will it build 2 kernels? Which kernel will end up being packaged or will there be a mix of both in deploy?

regards,

Koen

--

Different virtual/kernel recipies depending on kernel configuration might not be the best resolution path for this issue.

How would it be problematic to have multiple binary flavour packages(Still a single source package) 
of the same kernel with the same linux-headers in the distro packages ?
In deploy/images a similiar notation as for rootfs could be used.
Perhaps there are no mechanisms today to do this, but perhaps there could/should be ?
What about a pkgconfig style approach ?

poky configuration contains a logical separation of IMAGE, MACHINE and DISTRO. (Probably more, but this is unrelated to my point below).
What is the roadmap and future of this separation ?

image:
The IMAGE may contain user applications that will not function without the proper kernel modules.(Or possibly non-module available .config items).
i.e. RDEPENDS = kernel-module-xxxx will is impossible to automate since it only scans kernel build for its existence.(if it was configured by kernel .config monolith selected by MACHINE)
This is solved in most recipies as RRECOMMENDS = kernel-module-xxx to avoid build breakage.

So you need to configure your kernel to support all your IMAGEs, from core-image-smaller-than-minimal to core-image-huge-system(or in my case debug, profiling)
This is especially problematic for small embedded systems when it comes to debug and profiling, where included kernel configuration(depending on exactly what) 
will have varying degrees of performance impact.(some huge, others we can probably live with).

machine:
In my head, this would contain machine specific configuration and not deal with other kernel configuration. 
In edison, the kernel configuration is treated as a monolith, perhaps we can improve this by allowing for a more dynamic configuration model of the kernel, where the 
machine logical layer would depend on _only_ the machine specific parts of the kernel configuration. In the current(edison) case, it usually uses a defconfig. 
Great, but let the build modify it.

pipedream:
1. MACHINE selects a defconfig.
2. IMAGE selects packets on rootfs, some depend on kernel-module-xx, or kernel-config-xx.
3. When building image, fist stage scans included recipes for kernel-module-* and kernel-config-* and adds this to the MACHINE selected defconfig before building kernel.
Possibly, all kernel related stuff can be located to kernel-profiling-flavour.bb, where recipes are depending on a flavour instead of directly to the kernel-config-xxx.  
4. Continues with otherwise normal build-flow.
5. deploy/ipk..rpm..deb: Kernel binary packages are given a flavour.(Still the same kernel source package).
6. deploy/images: keeping a scheme of images similiar to rootfs:es.

Impossible ?

Best Regards,
David

Re: Variable locality too restricted.

[Bruce Ashfield]

On Thu, Dec 8, 2011 at 2:24 AM, David Nyström <David.Nystrom@enea.com> wrote:
>
> ________________________________________
> From: Koen Kooi [koen@dominion.thruhere.net]
> Sent: Wednesday, December 07, 2011 15:07
> To: Paul Eggleton
> Cc: Chris Larson; David Nyström; yocto@yoctoproject.org
> Subject: Re: [yocto] Variable locality too restricted.
>
> Op 7 dec. 2011, om 15:05 heeft Paul Eggleton het volgende geschreven:
>
>> On Wednesday 07 December 2011 06:52:25 Chris Larson wrote:
>>> On Wed, Dec 7, 2011 at 6:44 AM, David Nyström <David.Nystrom@enea.com>
>> wrote:
>>>> I'm trying to create a setup for qemuppc.
>>>>
>>>> Goals:
>>>> core-image-minimal + virtual/kernel.
>>>> core-image-minimal + virtual/kernel with modified .config for debug
>>>> flavoured kernel.
>>>>
>>>> Problems:
>>>> PREFERRED_PROVIDER seems to be restricted to local.conf, distro and
>>>> machine.
>>> This is incorrect. It can be anywhere in the configuration metadata.
>>> bitbake.conf includes a variety of config files, not just
>>> distro/machine. Read that to see other existing files which get
>>> included. Further, you could create a .conf/.inc which you include
>>> from your machine .conf, if your goal is just to avoid duplication.
>>
>> Changing MACHINE has other implications though; do we not have any other way
>> to switch out the kernel on a per-image basis?
>
> And how would that work from a packaging perspective? AIUI changing DISTRO/MACHINE flags in an image is a recipe for failure.
>
> regards,
>
> Koen
> Op 7 dec. 2011, om 15:10 heeft Paul Eggleton het volgende geschreven:
>
> --
>> On Wednesday 07 December 2011 15:07:54 Koen Kooi wrote:
>>>> Changing MACHINE has other implications though; do we not have any other
>>>> way to switch out the kernel on a per-image basis?
>>>
>>> And how would that work from a packaging perspective? AIUI changing
>>> DISTRO/MACHINE flags in an image is a recipe for failure.
>>
>> I'm not suggesting changing the kernel configuration for the existing kernel;
>> naturally you would need a different kernel recipe. Surely you should just be
>> able to have a different kernel that is built and installed for a different
>> image file? No invalidated packages necessary.
>
> A kernel buld will generate packages and thanks to kernel.bbclass they will have similar names. So what will happen when I do:
>
> bitbake foo-image bar-image
>
> ? Will it build 2 kernels? Which kernel will end up being packaged or will there be a mix of both in deploy?
>
> regards,
>
> Koen
>
> --
>
> Different virtual/kernel recipies depending on kernel configuration might not be the best resolution path for this issue.
>
> How would it be problematic to have multiple binary flavour packages(Still a single source package)
> of the same kernel with the same linux-headers in the distro packages ?
> In deploy/images a similiar notation as for rootfs could be used.
> Perhaps there are no mechanisms today to do this, but perhaps there could/should be ?
> What about a pkgconfig style approach ?
>
> poky configuration contains a logical separation of IMAGE, MACHINE and DISTRO. (Probably more, but this is unrelated to my point below).
> What is the roadmap and future of this separation ?
>
> image:
> The IMAGE may contain user applications that will not function without the proper kernel modules.(Or possibly non-module available .config items).
> i.e. RDEPENDS = kernel-module-xxxx will is impossible to automate since it only scans kernel build for its existence.(if it was configured by kernel .config monolith selected by MACHINE)
> This is solved in most recipies as RRECOMMENDS = kernel-module-xxx to avoid build breakage.
>
> So you need to configure your kernel to support all your IMAGEs, from core-image-smaller-than-minimal to core-image-huge-system(or in my case debug, profiling)
> This is especially problematic for small embedded systems when it comes to debug and profiling, where included kernel configuration(depending on exactly what)
> will have varying degrees of performance impact.(some huge, others we can probably live with).
>
> machine:
> In my head, this would contain machine specific configuration and not deal with other kernel configuration.
> In edison, the kernel configuration is treated as a monolith, perhaps we can improve this by allowing for a more dynamic configuration model of the kernel, where the
> machine logical layer would depend on _only_ the machine specific parts of the kernel configuration. In the current(edison) case, it usually uses a defconfig.
> Great, but let the build modify it.
>
> pipedream:
> 1. MACHINE selects a defconfig.
> 2. IMAGE selects packets on rootfs, some depend on kernel-module-xx, or kernel-config-xx.
> 3. When building image, fist stage scans included recipes for kernel-module-* and kernel-config-* and adds this to the MACHINE selected defconfig before building kernel.

Hi David,

For what it's worth, if you use the linux-yocto tooling/base kernel,
there are alternate
and modular ways to do something that sounds like what you are
describing with the
kernel configuration. I won't claim to have all the answers with how
this impacts
packaging, image creation, etc. I'm simply talking as a "kernel guy",
and my understanding
of what you are trying to do.

In this scenario, the base BSP inherits the standard/policy kernel configuration
fragments that are part of the kernel tree. It then adds board
specific (hardware) options
and overrides of optional functionality (where optional is defined by
the base configuration).

If you want to then modify the configuration slightly using either
in-tree or add on kernel
configuration fragments you can simply add a .cfg to the SCR_URI or set the
KERNEL_FEATURES variable, just like Darren did recently.

   KERNEL_FEATURES_append_fri2 += " cfg/smp.scc cfg/efi-ext.scc"

Cheers,

Bruce
> Possibly, all kernel related stuff can be located to kernel-profiling-flavour.bb, where recipes are depending on a flavour instead of directly to the kernel-config-xxx.
> 4. Continues with otherwise normal build-flow.
> 5. deploy/ipk..rpm..deb: Kernel binary packages are given a flavour.(Still the same kernel source package).
> 6. deploy/images: keeping a scheme of images similiar to rootfs:es.
>
> Impossible ?
>
> Best Regards,
> David
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto



-- 
"Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end"

Re: Variable locality too restricted.

[Darren Hart]

On 12/08/2011 06:31 AM, Bruce Ashfield wrote:
> On Thu, Dec 8, 2011 at 2:24 AM, David Nyström <David.Nystrom@enea.com> wrote:
>>
>> ________________________________________
>> From: Koen Kooi [koen@dominion.thruhere.net]
>> Sent: Wednesday, December 07, 2011 15:07
>> To: Paul Eggleton
>> Cc: Chris Larson; David Nyström; yocto@yoctoproject.org
>> Subject: Re: [yocto] Variable locality too restricted.
>>
>> Op 7 dec. 2011, om 15:05 heeft Paul Eggleton het volgende geschreven:
>>
>>> On Wednesday 07 December 2011 06:52:25 Chris Larson wrote:
>>>> On Wed, Dec 7, 2011 at 6:44 AM, David Nyström <David.Nystrom@enea.com>
>>> wrote:
>>>>> I'm trying to create a setup for qemuppc.
>>>>>
>>>>> Goals:
>>>>> core-image-minimal + virtual/kernel.
>>>>> core-image-minimal + virtual/kernel with modified .config for debug
>>>>> flavoured kernel.
>>>>>
>>>>> Problems:
>>>>> PREFERRED_PROVIDER seems to be restricted to local.conf, distro and
>>>>> machine.
>>>> This is incorrect. It can be anywhere in the configuration metadata.
>>>> bitbake.conf includes a variety of config files, not just
>>>> distro/machine. Read that to see other existing files which get
>>>> included. Further, you could create a .conf/.inc which you include
>>>> from your machine .conf, if your goal is just to avoid duplication.
>>>
>>> Changing MACHINE has other implications though; do we not have any other way
>>> to switch out the kernel on a per-image basis?
>>
>> And how would that work from a packaging perspective? AIUI changing DISTRO/MACHINE flags in an image is a recipe for failure.
>>
>> regards,
>>
>> Koen
>> Op 7 dec. 2011, om 15:10 heeft Paul Eggleton het volgende geschreven:
>>
>> --
>>> On Wednesday 07 December 2011 15:07:54 Koen Kooi wrote:
>>>>> Changing MACHINE has other implications though; do we not have any other
>>>>> way to switch out the kernel on a per-image basis?
>>>>
>>>> And how would that work from a packaging perspective? AIUI changing
>>>> DISTRO/MACHINE flags in an image is a recipe for failure.
>>>
>>> I'm not suggesting changing the kernel configuration for the existing kernel;
>>> naturally you would need a different kernel recipe. Surely you should just be
>>> able to have a different kernel that is built and installed for a different
>>> image file? No invalidated packages necessary.
>>
>> A kernel buld will generate packages and thanks to kernel.bbclass they will have similar names. So what will happen when I do:
>>
>> bitbake foo-image bar-image
>>
>> ? Will it build 2 kernels? Which kernel will end up being packaged or will there be a mix of both in deploy?
>>
>> regards,
>>
>> Koen
>>
>> --
>>
>> Different virtual/kernel recipies depending on kernel configuration might not be the best resolution path for this issue.
>>
>> How would it be problematic to have multiple binary flavour packages(Still a single source package)
>> of the same kernel with the same linux-headers in the distro packages ?
>> In deploy/images a similiar notation as for rootfs could be used.
>> Perhaps there are no mechanisms today to do this, but perhaps there could/should be ?
>> What about a pkgconfig style approach ?
>>
>> poky configuration contains a logical separation of IMAGE, MACHINE and DISTRO. (Probably more, but this is unrelated to my point below).
>> What is the roadmap and future of this separation ?
>>
>> image:
>> The IMAGE may contain user applications that will not function without the proper kernel modules.(Or possibly non-module available .config items).
>> i.e. RDEPENDS = kernel-module-xxxx will is impossible to automate since it only scans kernel build for its existence.(if it was configured by kernel .config monolith selected by MACHINE)
>> This is solved in most recipies as RRECOMMENDS = kernel-module-xxx to avoid build breakage.
>>
>> So you need to configure your kernel to support all your IMAGEs, from core-image-smaller-than-minimal to core-image-huge-system(or in my case debug, profiling)
>> This is especially problematic for small embedded systems when it comes to debug and profiling, where included kernel configuration(depending on exactly what)
>> will have varying degrees of performance impact.(some huge, others we can probably live with).
>>
>> machine:
>> In my head, this would contain machine specific configuration and not deal with other kernel configuration.
>> In edison, the kernel configuration is treated as a monolith, perhaps we can improve this by allowing for a more dynamic configuration model of the kernel, where the
>> machine logical layer would depend on _only_ the machine specific parts of the kernel configuration. In the current(edison) case, it usually uses a defconfig.
>> Great, but let the build modify it.
>>
>> pipedream:
>> 1. MACHINE selects a defconfig.
>> 2. IMAGE selects packets on rootfs, some depend on kernel-module-xx, or kernel-config-xx.
>> 3. When building image, fist stage scans included recipes for kernel-module-* and kernel-config-* and adds this to the MACHINE selected defconfig before building kernel.
> 
> Hi David,
> 
> For what it's worth, if you use the linux-yocto tooling/base kernel,
> there are alternate
> and modular ways to do something that sounds like what you are
> describing with the
> kernel configuration. I won't claim to have all the answers with how
> this impacts
> packaging, image creation, etc. I'm simply talking as a "kernel guy",
> and my understanding
> of what you are trying to do.
> 
> In this scenario, the base BSP inherits the standard/policy kernel configuration
> fragments that are part of the kernel tree. It then adds board
> specific (hardware) options
> and overrides of optional functionality (where optional is defined by
> the base configuration).
> 
> If you want to then modify the configuration slightly using either
> in-tree or add on kernel
> configuration fragments you can simply add a .cfg to the SCR_URI or set the
> KERNEL_FEATURES variable, just like Darren did recently.
> 
>    KERNEL_FEATURES_append_fri2 += " cfg/smp.scc cfg/efi-ext.scc"


This has the same original variable scope issue as before however. It
would be nice to be able to do this:

$ bitbake core-image-minimal
$ bitbake core-image-minimal-debug

The latter image would use the same machine, but it would build perf,
add "debug" and other configurable options to "APPEND", and add a
configurable set of KERNEL_FEATURES. We should also update the base
kernel tools (non-Yocto) to use merge_config.sh so fragments can be used
in those recipes as well.

The problem I've run into here in the past has been that I cannot
specify things like PREFERRED_PROVIDER in the image recipe. So, for
example, to build RT, the current documented approach is to first add a
PREFERRED_PROVIDER line to either local.conf or bblayers to select
linux-yocto-rt and then to build core-image-rt which merely adds some
relevant packages. It would be much preferable to be able to specify
just an image target and not have to change your configuration because
the image is the intuitive distinguishing factor for people.

--
Darren

> 
> Cheers,
> 
> Bruce
>> Possibly, all kernel related stuff can be located to kernel-profiling-flavour.bb, where recipes are depending on a flavour instead of directly to the kernel-config-xxx.
>> 4. Continues with otherwise normal build-flow.
>> 5. deploy/ipk..rpm..deb: Kernel binary packages are given a flavour.(Still the same kernel source package).
>> 6. deploy/images: keeping a scheme of images similiar to rootfs:es.
>>
>> Impossible ?
>>
>> Best Regards,
>> David
>> _______________________________________________
>> yocto mailing list
>> yocto@yoctoproject.org
>> https://lists.yoctoproject.org/listinfo/yocto
> 
> 
> 

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: Variable locality too restricted.

[Richard Purdie]

On Thu, 2011-12-08 at 10:49 -0800, Darren Hart wrote:
> On 12/08/2011 06:31 AM, Bruce Ashfield wrote:
> > If you want to then modify the configuration slightly using either
> > in-tree or add on kernel
> > configuration fragments you can simply add a .cfg to the SCR_URI or set the
> > KERNEL_FEATURES variable, just like Darren did recently.
> > 
> >    KERNEL_FEATURES_append_fri2 += " cfg/smp.scc cfg/efi-ext.scc"
> 
> 
> This has the same original variable scope issue as before however. It
> would be nice to be able to do this:
> 
> $ bitbake core-image-minimal
> $ bitbake core-image-minimal-debug
> 
> The latter image would use the same machine, but it would build perf,
> add "debug" and other configurable options to "APPEND", and add a
> configurable set of KERNEL_FEATURES. We should also update the base
> kernel tools (non-Yocto) to use merge_config.sh so fragments can be used
> in those recipes as well.
> 
> The problem I've run into here in the past has been that I cannot
> specify things like PREFERRED_PROVIDER in the image recipe. So, for
> example, to build RT, the current documented approach is to first add a
> PREFERRED_PROVIDER line to either local.conf or bblayers to select
> linux-yocto-rt and then to build core-image-rt which merely adds some
> relevant packages. It would be much preferable to be able to specify
> just an image target and not have to change your configuration because
> the image is the intuitive distinguishing factor for people.

I'd like to give the bitbake perspective on this problem.

PREFERRED_PROVIDER in images doesn't really make sense to bitbake.
Imagine you have:

core-image-minimal setting PREFERRED_PROVIDER = "kernelA" and
core-image-minima-debug setting PREFERRED_PROVIDER = "kernelB". Both
would depend on "virtual/kernel". You then run:

"bitbake core-image-minimal core-image-minimal-debug"

What would you expect bitbake to do?

The kernel is special in that doesn't really stage output that is reused
by other parts of the system but we have to consider the general case
where output such as libraries would end up in a shared sysroot. Even
then, the kernel does generate packages which it places in a machine
specific feed and the names don't reflect the different inputs, there is
one kernel-image package for example and in the above case it would be a
race on which one built last.

Bitbake therefore takes PREFERRED_PROVIDER and VERSION decisions from
the core configuration (.conf / .inc / machine / distro / bitbake.conf /
base.bbclass). There is no easy solution to this problem, even recipe
specific sysroots would only get a part solution.

Having said that, if you enable sstate checksums on the stamp files, you
will get a system that is *much* more responsive to change. With that
enabled you could:

bitbake core-image-minimal
<realise you need debug enabled>
<edit some .conf which turns it on>
bitbake core-image-minimal
<have an image with debug enabled>

The reason is that with the checksums on, bitbake can tell exactly what
changed, what it needs to rebuild and it will then do so.

Cheers,

Richard

Re: Variable locality too restricted.

[Darren Hart]

On 12/08/2011 02:47 PM, Richard Purdie wrote:
> On Thu, 2011-12-08 at 10:49 -0800, Darren Hart wrote:
>> On 12/08/2011 06:31 AM, Bruce Ashfield wrote:
>>> If you want to then modify the configuration slightly using either
>>> in-tree or add on kernel
>>> configuration fragments you can simply add a .cfg to the SCR_URI or set the
>>> KERNEL_FEATURES variable, just like Darren did recently.
>>>
>>>    KERNEL_FEATURES_append_fri2 += " cfg/smp.scc cfg/efi-ext.scc"
>>
>>
>> This has the same original variable scope issue as before however. It
>> would be nice to be able to do this:
>>
>> $ bitbake core-image-minimal
>> $ bitbake core-image-minimal-debug
>>
>> The latter image would use the same machine, but it would build perf,
>> add "debug" and other configurable options to "APPEND", and add a
>> configurable set of KERNEL_FEATURES. We should also update the base
>> kernel tools (non-Yocto) to use merge_config.sh so fragments can be used
>> in those recipes as well.
>>
>> The problem I've run into here in the past has been that I cannot
>> specify things like PREFERRED_PROVIDER in the image recipe. So, for
>> example, to build RT, the current documented approach is to first add a
>> PREFERRED_PROVIDER line to either local.conf or bblayers to select
>> linux-yocto-rt and then to build core-image-rt which merely adds some
>> relevant packages. It would be much preferable to be able to specify
>> just an image target and not have to change your configuration because
>> the image is the intuitive distinguishing factor for people.
> 
> I'd like to give the bitbake perspective on this problem.
> 
> PREFERRED_PROVIDER in images doesn't really make sense to bitbake.
> Imagine you have:
> 
> core-image-minimal setting PREFERRED_PROVIDER = "kernelA" and
> core-image-minima-debug setting PREFERRED_PROVIDER = "kernelB". Both
> would depend on "virtual/kernel". You then run:
> 
> "bitbake core-image-minimal core-image-minimal-debug"
> 
> What would you expect bitbake to do?

What I _think_ most people would expect it to do is to build each kernel
and install the right one in each image. I understand this doesn't work
with the way bitbake currently handles kernels, as you describe below.

> The kernel is special in that doesn't really stage output that is reused
> by other parts of the system but we have to consider the general case
> where output such as libraries would end up in a shared sysroot. Even
> then, the kernel does generate packages which it places in a machine
> specific feed and the names don't reflect the different inputs, there is
> one kernel-image package for example and in the above case it would be a
> race on which one built last.

The names not reflecting different inputs seems like it should be
something we can address. I appreciate it isn't trivial, and probably
stomps on some pretty core assumptions dealing with how we build images,
but I believe it would be valuable to be able to build multiple kernel
versions for a given machine. Many Linux distributions do this and I can
see users of Yocto wanting to do the same with the distributions they build.

> 
> Bitbake therefore takes PREFERRED_PROVIDER and VERSION decisions from
> the core configuration (.conf / .inc / machine / distro / bitbake.conf /
> base.bbclass). There is no easy solution to this problem, even recipe
> specific sysroots would only get a part solution.

Would this be something we should consider as a major feature
development item for a future release?

> Having said that, if you enable sstate checksums on the stamp files, you
> will get a system that is *much* more responsive to change. With that
> enabled you could:
> 
> bitbake core-image-minimal
> <realise you need debug enabled>
> <edit some .conf which turns it on>
> bitbake core-image-minimal
> <have an image with debug enabled>
> 
> The reason is that with the checksums on, bitbake can tell exactly what
> changed, what it needs to rebuild and it will then do so.
> 
> Cheers,
> 
> Richard
-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: Variable locality too restricted.

[Richard Purdie]

On Thu, 2011-12-08 at 14:56 -0800, Darren Hart wrote:
> On 12/08/2011 02:47 PM, Richard Purdie wrote:
> > On Thu, 2011-12-08 at 10:49 -0800, Darren Hart wrote:
> >> On 12/08/2011 06:31 AM, Bruce Ashfield wrote:
> >>> If you want to then modify the configuration slightly using either
> >>> in-tree or add on kernel
> >>> configuration fragments you can simply add a .cfg to the SCR_URI or set the
> >>> KERNEL_FEATURES variable, just like Darren did recently.
> >>>
> >>>    KERNEL_FEATURES_append_fri2 += " cfg/smp.scc cfg/efi-ext.scc"
> >>
> >>
> >> This has the same original variable scope issue as before however. It
> >> would be nice to be able to do this:
> >>
> >> $ bitbake core-image-minimal
> >> $ bitbake core-image-minimal-debug
> >>
> >> The latter image would use the same machine, but it would build perf,
> >> add "debug" and other configurable options to "APPEND", and add a
> >> configurable set of KERNEL_FEATURES. We should also update the base
> >> kernel tools (non-Yocto) to use merge_config.sh so fragments can be used
> >> in those recipes as well.
> >>
> >> The problem I've run into here in the past has been that I cannot
> >> specify things like PREFERRED_PROVIDER in the image recipe. So, for
> >> example, to build RT, the current documented approach is to first add a
> >> PREFERRED_PROVIDER line to either local.conf or bblayers to select
> >> linux-yocto-rt and then to build core-image-rt which merely adds some
> >> relevant packages. It would be much preferable to be able to specify
> >> just an image target and not have to change your configuration because
> >> the image is the intuitive distinguishing factor for people.
> > 
> > I'd like to give the bitbake perspective on this problem.
> > 
> > PREFERRED_PROVIDER in images doesn't really make sense to bitbake.
> > Imagine you have:
> > 
> > core-image-minimal setting PREFERRED_PROVIDER = "kernelA" and
> > core-image-minima-debug setting PREFERRED_PROVIDER = "kernelB". Both
> > would depend on "virtual/kernel". You then run:
> > 
> > "bitbake core-image-minimal core-image-minimal-debug"
> > 
> > What would you expect bitbake to do?
> 
> What I _think_ most people would expect it to do is to build each kernel
> and install the right one in each image. I understand this doesn't work
> with the way bitbake currently handles kernels, as you describe below.
>
> > The kernel is special in that doesn't really stage output that is reused
> > by other parts of the system but we have to consider the general case
> > where output such as libraries would end up in a shared sysroot. Even
> > then, the kernel does generate packages which it places in a machine
> > specific feed and the names don't reflect the different inputs, there is
> > one kernel-image package for example and in the above case it would be a
> > race on which one built last.
> 
> The names not reflecting different inputs seems like it should be
> something we can address. I appreciate it isn't trivial, and probably
> stomps on some pretty core assumptions dealing with how we build images,

Images aren't the problem, those are easy. Its the packages. How do you
represent that kernel-image A is built with configuration X and
kernel-image B is built with configuration Y. How do you determine
whether you can switch between A and B or not? How do you decide which
one is the higher version?

> but I believe it would be valuable to be able to build multiple kernel
> versions for a given machine.

Version on its own is easier as packages do have pretty clear ideas
about versions. I suspect you mean different configurations though.

>  Many Linux distributions do this and I can
> see users of Yocto wanting to do the same with the distributions they build.

How do different linux distributions do that exactly? Likely different
package feeds for each kernel?

As far as I can tell, to do this properly we'd need to:

a) Adopt per recipe sysroots
b) Adopt package feeds constructed per image
c) Drop support for anyone wanting traditional distro type package feeds

These things would not be popular with the community due to a loss of
functionality and would utterly destroy any notion of build speed.

> > Bitbake therefore takes PREFERRED_PROVIDER and VERSION decisions from
> > the core configuration (.conf / .inc / machine / distro / bitbake.conf /
> > base.bbclass). There is no easy solution to this problem, even recipe
> > specific sysroots would only get a part solution.
> 
> Would this be something we should consider as a major feature
> development item for a future release?

Not without some idea of how it could be made to work. I can't visualise
a way of making this work as you describe as long as we have packages
around to deal with in the conventional way.

We've talked about special cases for the kernel above but any proposal
needs to work generically too which makes this trickier again :(.

Cheers,

Richard

Re: Variable locality too restricted.

[Darren Hart]

On 12/08/2011 03:12 PM, Richard Purdie wrote:
> On Thu, 2011-12-08 at 14:56 -0800, Darren Hart wrote:
>> On 12/08/2011 02:47 PM, Richard Purdie wrote:
>>> On Thu, 2011-12-08 at 10:49 -0800, Darren Hart wrote:
>>>> On 12/08/2011 06:31 AM, Bruce Ashfield wrote:
>>>>> If you want to then modify the configuration slightly using either
>>>>> in-tree or add on kernel
>>>>> configuration fragments you can simply add a .cfg to the SCR_URI or set the
>>>>> KERNEL_FEATURES variable, just like Darren did recently.
>>>>>
>>>>>    KERNEL_FEATURES_append_fri2 += " cfg/smp.scc cfg/efi-ext.scc"
>>>>
>>>>
>>>> This has the same original variable scope issue as before however. It
>>>> would be nice to be able to do this:
>>>>
>>>> $ bitbake core-image-minimal
>>>> $ bitbake core-image-minimal-debug
>>>>
>>>> The latter image would use the same machine, but it would build perf,
>>>> add "debug" and other configurable options to "APPEND", and add a
>>>> configurable set of KERNEL_FEATURES. We should also update the base
>>>> kernel tools (non-Yocto) to use merge_config.sh so fragments can be used
>>>> in those recipes as well.
>>>>
>>>> The problem I've run into here in the past has been that I cannot
>>>> specify things like PREFERRED_PROVIDER in the image recipe. So, for
>>>> example, to build RT, the current documented approach is to first add a
>>>> PREFERRED_PROVIDER line to either local.conf or bblayers to select
>>>> linux-yocto-rt and then to build core-image-rt which merely adds some
>>>> relevant packages. It would be much preferable to be able to specify
>>>> just an image target and not have to change your configuration because
>>>> the image is the intuitive distinguishing factor for people.
>>>
>>> I'd like to give the bitbake perspective on this problem.
>>>
>>> PREFERRED_PROVIDER in images doesn't really make sense to bitbake.
>>> Imagine you have:
>>>
>>> core-image-minimal setting PREFERRED_PROVIDER = "kernelA" and
>>> core-image-minima-debug setting PREFERRED_PROVIDER = "kernelB". Both
>>> would depend on "virtual/kernel". You then run:
>>>
>>> "bitbake core-image-minimal core-image-minimal-debug"
>>>
>>> What would you expect bitbake to do?
>>
>> What I _think_ most people would expect it to do is to build each kernel
>> and install the right one in each image. I understand this doesn't work
>> with the way bitbake currently handles kernels, as you describe below.
>>
>>> The kernel is special in that doesn't really stage output that is reused
>>> by other parts of the system but we have to consider the general case
>>> where output such as libraries would end up in a shared sysroot. Even
>>> then, the kernel does generate packages which it places in a machine
>>> specific feed and the names don't reflect the different inputs, there is
>>> one kernel-image package for example and in the above case it would be a
>>> race on which one built last.
>>
>> The names not reflecting different inputs seems like it should be
>> something we can address. I appreciate it isn't trivial, and probably
>> stomps on some pretty core assumptions dealing with how we build images,
> 
> Images aren't the problem, those are easy. Its the packages. How do you
> represent that kernel-image A is built with configuration X and
> kernel-image B is built with configuration Y. How do you determine
> whether you can switch between A and B or not? How do you decide which
> one is the higher version?

I'd imagine we change the package name linux-yocto vs. linux-yocto-debug
for example (this is how some of the Linux distributions handle it). I
believe PREFERRED_PROVIDER can then distinguish based on the package
name, and the versioning works the same as it does now.

> 
>> but I believe it would be valuable to be able to build multiple kernel
>> versions for a given machine.
> 
> Version on its own is easier as packages do have pretty clear ideas
> about versions. I suspect you mean different configurations though.

Agreed, that was a generic "version", not PV :-) I believe being able to
select the PN from PREFERRED_PROVIDER in the image recipe is what I'm
saying I'd prefer.


> 
>>  Many Linux distributions do this and I can
>> see users of Yocto wanting to do the same with the distributions they build.
> 
> How do different linux distributions do that exactly? Likely different
> package feeds for each kernel?

They have different names. For example, Red Hat's MRG ships something like:

linux-rt
linux-rt-vanilla
linux-rt-debug
linux-rt-vanilla-debug

It has been a while since I looked, but it was something along those lines.

> 
> As far as I can tell, to do this properly we'd need to:
> 
> a) Adopt per recipe sysroots

Yes. And if the same recipe can be used to generate different PNs based
on the config option, then that is treated as a separate recipe right?

> b) Adopt package feeds constructed per image

Does the term "feed" refer to "all the packages that can be installed"
or "all the packages that will be installed" ? Note that the distros
allow multiple kernels to be installed at once - which in the -debug
case would probably be preferred, but I can see the argument of not
wanting to go that route for Yocto.

> c) Drop support for anyone wanting traditional distro type package feeds
> 

I'm hoping b and c would not be required.... or could be made not required.

> These things would not be popular with the community due to a loss of
> functionality and would utterly destroy any notion of build speed.
> 
>>> Bitbake therefore takes PREFERRED_PROVIDER and VERSION decisions from
>>> the core configuration (.conf / .inc / machine / distro / bitbake.conf /
>>> base.bbclass). There is no easy solution to this problem, even recipe
>>> specific sysroots would only get a part solution.
>>
>> Would this be something we should consider as a major feature
>> development item for a future release?
> 
> Not without some idea of how it could be made to work. I can't visualise
> a way of making this work as you describe as long as we have packages
> around to deal with in the conventional way.
> 
> We've talked about special cases for the kernel above but any proposal
> needs to work generically too which makes this trickier again :(.

Agreed. I'd like to discuss the package feed concept a bit more. Clearly
more thought is required.

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: Variable locality too restricted.

[David Nyström]

________________________________________
From: Darren Hart [dvhart@linux.intel.com]
Sent: Friday, December 09, 2011 00:52
To: Richard Purdie
Cc: Bruce Ashfield; David Nyström; Paul Eggleton; yocto@yoctoproject.org
Subject: Re: [yocto] Variable locality too restricted.

On 12/08/2011 03:12 PM, Richard Purdie wrote:
> On Thu, 2011-12-08 at 14:56 -0800, Darren Hart wrote:
>> On 12/08/2011 02:47 PM, Richard Purdie wrote:
>>> On Thu, 2011-12-08 at 10:49 -0800, Darren Hart wrote:
>>>> On 12/08/2011 06:31 AM, Bruce Ashfield wrote:
>>>>> If you want to then modify the configuration slightly using either
>>>>> in-tree or add on kernel
>>>>> configuration fragments you can simply add a .cfg to the SCR_URI or set the
>>>>> KERNEL_FEATURES variable, just like Darren did recently.
>>>>>
>>>>>    KERNEL_FEATURES_append_fri2 += " cfg/smp.scc cfg/efi-ext.scc"
>>>>
>>>>
>>>> This has the same original variable scope issue as before however. It
>>>> would be nice to be able to do this:
>>>>
>>>> $ bitbake core-image-minimal
>>>> $ bitbake core-image-minimal-debug
>>>>
>>>> The latter image would use the same machine, but it would build perf,
>>>> add "debug" and other configurable options to "APPEND", and add a
>>>> configurable set of KERNEL_FEATURES. We should also update the base
>>>> kernel tools (non-Yocto) to use merge_config.sh so fragments can be used
>>>> in those recipes as well.
>>>>
>>>> The problem I've run into here in the past has been that I cannot
>>>> specify things like PREFERRED_PROVIDER in the image recipe. So, for
>>>> example, to build RT, the current documented approach is to first add a
>>>> PREFERRED_PROVIDER line to either local.conf or bblayers to select
>>>> linux-yocto-rt and then to build core-image-rt which merely adds some
>>>> relevant packages. It would be much preferable to be able to specify
>>>> just an image target and not have to change your configuration because
>>>> the image is the intuitive distinguishing factor for people.
>>>
>>> I'd like to give the bitbake perspective on this problem.
>>>
>>> PREFERRED_PROVIDER in images doesn't really make sense to bitbake.
>>> Imagine you have:
>>>
>>> core-image-minimal setting PREFERRED_PROVIDER = "kernelA" and
>>> core-image-minima-debug setting PREFERRED_PROVIDER = "kernelB". Both
>>> would depend on "virtual/kernel". You then run:
>>>
>>> "bitbake core-image-minimal core-image-minimal-debug"
>>>
>>> What would you expect bitbake to do?
>>
>> What I _think_ most people would expect it to do is to build each kernel
>> and install the right one in each image. I understand this doesn't work
>> with the way bitbake currently handles kernels, as you describe below.
>>
>>> The kernel is special in that doesn't really stage output that is reused
>>> by other parts of the system but we have to consider the general case
>>> where output such as libraries would end up in a shared sysroot. Even
>>> then, the kernel does generate packages which it places in a machine
>>> specific feed and the names don't reflect the different inputs, there is
>>> one kernel-image package for example and in the above case it would be a
>>> race on which one built last.
>>
>> The names not reflecting different inputs seems like it should be
>> something we can address. I appreciate it isn't trivial, and probably
>> stomps on some pretty core assumptions dealing with how we build images,
>
> Images aren't the problem, those are easy. Its the packages. How do you
> represent that kernel-image A is built with configuration X and
> kernel-image B is built with configuration Y. How do you determine
> whether you can switch between A and B or not? How do you decide which
> one is the higher version?

I'd imagine we change the package name linux-yocto vs. linux-yocto-debug
for example (this is how some of the Linux distributions handle it). I
believe PREFERRED_PROVIDER can then distinguish based on the package
name, and the versioning works the same as it does now.

>
>> but I believe it would be valuable to be able to build multiple kernel
>> versions for a given machine.
>
> Version on its own is easier as packages do have pretty clear ideas
> about versions. I suspect you mean different configurations though.

Agreed, that was a generic "version", not PV :-) I believe being able to
select the PN from PREFERRED_PROVIDER in the image recipe is what I'm
saying I'd prefer.


>
>>  Many Linux distributions do this and I can
>> see users of Yocto wanting to do the same with the distributions they build.
>
> How do different linux distributions do that exactly? Likely different
> package feeds for each kernel?

They have different names. For example, Red Hat's MRG ships something like:

linux-rt
linux-rt-vanilla
linux-rt-debug
linux-rt-vanilla-debug

It has been a while since I looked, but it was something along those lines.

>
> As far as I can tell, to do this properly we'd need to:
>
> a) Adopt per recipe sysroots

Yes. And if the same recipe can be used to generate different PNs based
on the config option, then that is treated as a separate recipe right?

> b) Adopt package feeds constructed per image

Does the term "feed" refer to "all the packages that can be installed"
or "all the packages that will be installed" ? Note that the distros
allow multiple kernels to be installed at once - which in the -debug
case would probably be preferred, but I can see the argument of not
wanting to go that route for Yocto.

> c) Drop support for anyone wanting traditional distro type package feeds
>

I'm hoping b and c would not be required.... or could be made not required.

> These things would not be popular with the community due to a loss of
> functionality and would utterly destroy any notion of build speed.
>
>>> Bitbake therefore takes PREFERRED_PROVIDER and VERSION decisions from
>>> the core configuration (.conf / .inc / machine / distro / bitbake.conf /
>>> base.bbclass). There is no easy solution to this problem, even recipe
>>> specific sysroots would only get a part solution.
>>
>> Would this be something we should consider as a major feature
>> development item for a future release?
>
> Not without some idea of how it could be made to work. I can't visualise
> a way of making this work as you describe as long as we have packages
> around to deal with in the conventional way.
>
> We've talked about special cases for the kernel above but any proposal
> needs to work generically too which makes this trickier again :(.

Agreed. I'd like to discuss the package feed concept a bit more. Clearly
more thought is required.


--------------
Will this have implications really be as severe as this ?

We already have a model where normal packages are packaged according to the default scheme:

udev        - Symbol stripped pkg of udev.
udev-dbg -  Non stripped pkg of udev.
udev-dev -  Development pkg of udev.

A selectable in IMAGE_FEATURES controls what gets installed on rootfs, and 
I guess all of them are available in sysroot with the added .debug directory for -dbg packages.

Slightly off-topic:
My only objection to this method would be the rootfs bloat where both a stripped bash _and_ a bash 
executable with symbols will be installed if dbg-pkgs is configured. Where the debug version is located under the .debug path.
Perhaps we could allow for this to be mutually exclusive when generating the rootfs.

Besides this, it's a good method, perhaps we should not create a new method just because its a kernel package.
Perhaps we want to do they same with the busybox package ?
The only differency with this method is that the recipe do_configure/do_compile/dp_package would be issued once for each flavour,
where the flavours can be selectable in the IMAGE.

The previous discussions of having different kernel recipes for different kernel configs, I dont particularly agree with.
I guess this will lead to separate build directories directly hitting both build time and build disk space.
Having three configs built in the same build directory with increasing selects, will take roughly the same 
amount of time and space as the regular bloated .config monolith build.

Having multiple flavours, and being able to add a flavour with .bbappend would increase bitbake's modularity when it comes to 
multiple meta-layers from different sources.

Lets say I want to create an custom image for mpc8536ds, which has a defined machine in meta-fsl-ppc.
Image consists of :
meta, meta-yocto,      meta-fsl-ppc       meta-davids-layer.
|------YOCTO---------- | ----HW-vendor--- | ----SW-only-guys----- |
    PACKAGES       DISTRO-MACHINE    IMAGES

When looking at poky-edison, to add changes to the kernel .config, I would have to edit meta-fsl-ppc sources and copy some of 
their repo to my own, making the layer structure between HW vendor and SW-guys obsolete since meta-fsl-ppc wont(and shouldn't) accept my user specific modifications.
This is not only related to .config, but also f.ex. MACHINE_FEATURES, KERNEL_IMAGETYPE which can't be overridden from images

Optimally, meta-davids-layer should contain different package configurations(kernel, busybox) and images only. 

Best Regards,
David





--
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: Variable locality too restricted.

[Darren Hart]

On 12/09/2011 12:30 AM, David Nyström wrote:
> 
> ________________________________________
> From: Darren Hart [dvhart@linux.intel.com]
> Sent: Friday, December 09, 2011 00:52
> To: Richard Purdie
> Cc: Bruce Ashfield; David Nyström; Paul Eggleton; yocto@yoctoproject.org
> Subject: Re: [yocto] Variable locality too restricted.
> 
> On 12/08/2011 03:12 PM, Richard Purdie wrote:
>> On Thu, 2011-12-08 at 14:56 -0800, Darren Hart wrote:
>>> On 12/08/2011 02:47 PM, Richard Purdie wrote:
>>>> On Thu, 2011-12-08 at 10:49 -0800, Darren Hart wrote:
>>>>> On 12/08/2011 06:31 AM, Bruce Ashfield wrote:
>>>>>> If you want to then modify the configuration slightly using either
>>>>>> in-tree or add on kernel
>>>>>> configuration fragments you can simply add a .cfg to the SCR_URI or set the
>>>>>> KERNEL_FEATURES variable, just like Darren did recently.
>>>>>>
>>>>>>    KERNEL_FEATURES_append_fri2 += " cfg/smp.scc cfg/efi-ext.scc"
>>>>>
>>>>>
>>>>> This has the same original variable scope issue as before however. It
>>>>> would be nice to be able to do this:
>>>>>
>>>>> $ bitbake core-image-minimal
>>>>> $ bitbake core-image-minimal-debug
>>>>>
>>>>> The latter image would use the same machine, but it would build perf,
>>>>> add "debug" and other configurable options to "APPEND", and add a
>>>>> configurable set of KERNEL_FEATURES. We should also update the base
>>>>> kernel tools (non-Yocto) to use merge_config.sh so fragments can be used
>>>>> in those recipes as well.
>>>>>
>>>>> The problem I've run into here in the past has been that I cannot
>>>>> specify things like PREFERRED_PROVIDER in the image recipe. So, for
>>>>> example, to build RT, the current documented approach is to first add a
>>>>> PREFERRED_PROVIDER line to either local.conf or bblayers to select
>>>>> linux-yocto-rt and then to build core-image-rt which merely adds some
>>>>> relevant packages. It would be much preferable to be able to specify
>>>>> just an image target and not have to change your configuration because
>>>>> the image is the intuitive distinguishing factor for people.
>>>>
>>>> I'd like to give the bitbake perspective on this problem.
>>>>
>>>> PREFERRED_PROVIDER in images doesn't really make sense to bitbake.
>>>> Imagine you have:
>>>>
>>>> core-image-minimal setting PREFERRED_PROVIDER = "kernelA" and
>>>> core-image-minima-debug setting PREFERRED_PROVIDER = "kernelB". Both
>>>> would depend on "virtual/kernel". You then run:
>>>>
>>>> "bitbake core-image-minimal core-image-minimal-debug"
>>>>
>>>> What would you expect bitbake to do?
>>>
>>> What I _think_ most people would expect it to do is to build each kernel
>>> and install the right one in each image. I understand this doesn't work
>>> with the way bitbake currently handles kernels, as you describe below.
>>>
>>>> The kernel is special in that doesn't really stage output that is reused
>>>> by other parts of the system but we have to consider the general case
>>>> where output such as libraries would end up in a shared sysroot. Even
>>>> then, the kernel does generate packages which it places in a machine
>>>> specific feed and the names don't reflect the different inputs, there is
>>>> one kernel-image package for example and in the above case it would be a
>>>> race on which one built last.
>>>
>>> The names not reflecting different inputs seems like it should be
>>> something we can address. I appreciate it isn't trivial, and probably
>>> stomps on some pretty core assumptions dealing with how we build images,
>>
>> Images aren't the problem, those are easy. Its the packages. How do you
>> represent that kernel-image A is built with configuration X and
>> kernel-image B is built with configuration Y. How do you determine
>> whether you can switch between A and B or not? How do you decide which
>> one is the higher version?
> 
> I'd imagine we change the package name linux-yocto vs. linux-yocto-debug
> for example (this is how some of the Linux distributions handle it). I
> believe PREFERRED_PROVIDER can then distinguish based on the package
> name, and the versioning works the same as it does now.
> 
>>
>>> but I believe it would be valuable to be able to build multiple kernel
>>> versions for a given machine.
>>
>> Version on its own is easier as packages do have pretty clear ideas
>> about versions. I suspect you mean different configurations though.
> 
> Agreed, that was a generic "version", not PV :-) I believe being able to
> select the PN from PREFERRED_PROVIDER in the image recipe is what I'm
> saying I'd prefer.
> 
> 
>>
>>>  Many Linux distributions do this and I can
>>> see users of Yocto wanting to do the same with the distributions they build.
>>
>> How do different linux distributions do that exactly? Likely different
>> package feeds for each kernel?
> 
> They have different names. For example, Red Hat's MRG ships something like:
> 
> linux-rt
> linux-rt-vanilla
> linux-rt-debug
> linux-rt-vanilla-debug
> 
> It has been a while since I looked, but it was something along those lines.
> 
>>
>> As far as I can tell, to do this properly we'd need to:
>>
>> a) Adopt per recipe sysroots
> 
> Yes. And if the same recipe can be used to generate different PNs based
> on the config option, then that is treated as a separate recipe right?
> 
>> b) Adopt package feeds constructed per image
> 
> Does the term "feed" refer to "all the packages that can be installed"
> or "all the packages that will be installed" ? Note that the distros
> allow multiple kernels to be installed at once - which in the -debug
> case would probably be preferred, but I can see the argument of not
> wanting to go that route for Yocto.
> 
>> c) Drop support for anyone wanting traditional distro type package feeds
>>
> 
> I'm hoping b and c would not be required.... or could be made not required.
> 
>> These things would not be popular with the community due to a loss of
>> functionality and would utterly destroy any notion of build speed.
>>
>>>> Bitbake therefore takes PREFERRED_PROVIDER and VERSION decisions from
>>>> the core configuration (.conf / .inc / machine / distro / bitbake.conf /
>>>> base.bbclass). There is no easy solution to this problem, even recipe
>>>> specific sysroots would only get a part solution.
>>>
>>> Would this be something we should consider as a major feature
>>> development item for a future release?
>>
>> Not without some idea of how it could be made to work. I can't visualise
>> a way of making this work as you describe as long as we have packages
>> around to deal with in the conventional way.
>>
>> We've talked about special cases for the kernel above but any proposal
>> needs to work generically too which makes this trickier again :(.
> 
> Agreed. I'd like to discuss the package feed concept a bit more. Clearly
> more thought is required.
> 
> 
> --------------

Hi David,

When replying on mailing lists, please use proper quoting and inline
responses.

> Will this have implications really be as severe as this ?
> 
> We already have a model where normal packages are packaged according to the default scheme:
> 
> udev        - Symbol stripped pkg of udev.
> udev-dbg -  Non stripped pkg of udev.
> udev-dev -  Development pkg of udev.
> 
> A selectable in IMAGE_FEATURES controls what gets installed on rootfs, and 
> I guess all of them are available in sysroot with the added .debug directory for -dbg packages.

This really addresses a completely different problem, and we should not
overload the mechanism to handle different types of kernels.

...

> 
> The previous discussions of having different kernel recipes for different kernel configs, I dont particularly agree with.
> I guess this will lead to separate build directories directly hitting both build time and build disk space.
> Having three configs built in the same build directory with increasing selects, will take roughly the same 
> amount of time and space as the regular bloated .config monolith build.

No. We need separate build directories. We need to maintain the build
for each kernel and not overwrite it when building a different
configuration. Note that the "other" kernel might also contain
additional patches, not just .config changes. But even if it didn't,
.config changes are significant enough to warrant a new build tree.

> 
> Having multiple flavours, and being able to add a flavour with .bbappend would increase bitbake's modularity when it comes to 
> multiple meta-layers from different sources.
> 
> Lets say I want to create an custom image for mpc8536ds, which has a defined machine in meta-fsl-ppc.
> Image consists of :
> meta, meta-yocto,      meta-fsl-ppc       meta-davids-layer.
> |------YOCTO---------- | ----HW-vendor--- | ----SW-only-guys----- |
>     PACKAGES       DISTRO-MACHINE    IMAGES
> 
> When looking at poky-edison, to add changes to the kernel .config, I would have to edit meta-fsl-ppc sources and copy some of 
> their repo to my own, making the layer structure between HW vendor and SW-guys obsolete since meta-fsl-ppc wont(and shouldn't) accept my user specific modifications.
> This is not only related to .config, but also f.ex. MACHINE_FEATURES, KERNEL_IMAGETYPE which can't be overridden from images
> 
> Optimally, meta-davids-layer should contain different package configurations(kernel, busybox) and images only. 


I'm not following you here. As it stands now, you should be able to use
a bbappend of the kernel recipe to modify how the kernel is built
(without duplicating the recipe). This is easier with linux-yocto based
recipes with support for configuration fragments and feature descriptors.

-- 
Darren Hart
Intel Open Source Technology Center
Yocto Project - Linux Kernel

Re: Variable locality too restricted.

[Trevor Woerner]

On Thu, Dec 8, 2011 at 5:47 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Imagine you have:
>
> core-image-minimal setting PREFERRED_PROVIDER = "kernelA" and
> core-image-minima-debug setting PREFERRED_PROVIDER = "kernelB". Both
> would depend on "virtual/kernel". You then run:
>
> "bitbake core-image-minimal core-image-minimal-debug"
>
> What would you expect bitbake to do?

Would it be possible to have both kernels built and installed to the
image with the PREFERRED_PROVIDER selecting the default kernel to boot
(i.e. grub/lilo/<whatever> configuration setting)?

Re: Variable locality too restricted.

[Trevor Woerner]

On Thu, Dec 8, 2011 at 5:47 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> Having said that, if you enable sstate checksums on the stamp files, you
> will get a system that is *much* more responsive to change. With that
> enabled you could:

Isn't this the default behaviour (i.e. sstate checksums enabled on the
stamp files)?