What are _virtual providers? and other Suffixes?

What are _virtual providers? and other Suffixes?

[Brad Litterell]

[-- Attachment #1: Type: text/plain, Size: 1088 bytes --]

I searched the Yocto Mega Manual, but am still somewhat mystified by the suffix formatting of various variable - especially virtual ones like this:


PREFERRED_PROVIDER_virtual/gettext = "gettext"

PREFERRED_PROVIDER_virtual/kernel_am335x-evm = "linux-ti-staging"

PREFERRED_PROVIDER_virtual/bootloader_am335x-evm = "u-boot-ti-staging"


Also, there seems to be some sort of prioritization going on with the multiple levels underscore characters in places like this:


# Set the PREFERRED_PROVIDER for jpeg functionality based on the MACHINE

# architecture.  For armv7a devices libjpeg-turbo should be used to take

# advantage of the SIMD instructions.

PREFERRED_PROVIDER_jpeg = "libjpeg-turbo"

PREFERRED_PROVIDER_jpeg_armv5te = "jpeg"


(all examples from the arago project in arago-prefs.inc)

Is there a succinct way of viewing these suffixes that will allow me to wrap my head around their various & myriad versions?

Am I right in assuming the "variable name" consists of the uppercase parse (regardless of number of underscores)?

Thanks,
Brad


[-- Attachment #2: Type: text/html, Size: 4556 bytes --]

Re: What are _virtual providers? and other Suffixes?

[Paul Eggleton]

Hi Brad,

On Monday 19 August 2013 22:51:04 Brad Litterell wrote:
> I searched the Yocto Mega Manual, but am still somewhat mystified by the
> suffix formatting of various variable - especially virtual ones like this:
> 
> 
> PREFERRED_PROVIDER_virtual/gettext = "gettext"
> 
> PREFERRED_PROVIDER_virtual/kernel_am335x-evm = "linux-ti-staging"
> 
> PREFERRED_PROVIDER_virtual/bootloader_am335x-evm = "u-boot-ti-staging"

These are setting PREFERRED_PROVIDER for a number of targets. "virtual/xyz" is
a convention we use where there are multiple providers for a particular
component, so a recipe that requires that component can have virtual/xyz in
its DEPENDS, each provider declares virtual/xyz in its PROVIDES, and then in
configuration the desired recipe to provide that can be selected via 
PREFERRED_PROVIDER_virtual/xyz = "...".

The second two examples have an additional override which makes them only
applicable for a specific machine (am335x-evm). If you want to know how this
mechanism works have a look at this:

http://docs.openembedded.org/bitbake/html/ch02.html#id439023

(We're in the process of updating the BitBake manual, so this old version will have to suffice.)

> Also, there seems to be some sort of prioritization going on with the
> multiple levels underscore characters in places like this:
> 
> 
> # Set the PREFERRED_PROVIDER for jpeg functionality based on the MACHINE
> 
> # architecture.  For armv7a devices libjpeg-turbo should be used to take
> 
> # advantage of the SIMD instructions.
> 
> PREFERRED_PROVIDER_jpeg = "libjpeg-turbo"
> 
> PREFERRED_PROVIDER_jpeg_armv5te = "jpeg"
> 
> 
> (all examples from the arago project in arago-prefs.inc)
> 
> Is there a succinct way of viewing these suffixes that will allow me to wrap
> my head around their various & myriad versions?

OVERRIDES is listed in order so that overrides for the later items are
preferred. If you run bitbake -e | less and search for OVERRIDES (with the /
key) you can see what the value is for the global environment; here is an
example for my current configuration:

# $OVERRIDES [3 operations]
#   set conf/bitbake.conf:670
#     "${TARGET_OS}:${TRANSLATED_TARGET_ARCH}:build-${BUILD_OS}:pn-${PN}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${CLASSOVERRIDE}:forcevariable"
#   set conf/bitbake.conf:671
#     [vardepsexclude] "MACHINEOVERRIDES"
#   postdot /home/paul/poky/poky/meta/conf/distro/include/tclibc-eglibc.inc:9
#     "${LIBCOVERRIDE}"
# computed:
#   "${TARGET_OS}:${TRANSLATED_TARGET_ARCH}:build-${BUILD_OS}:pn-${PN}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${CLASSOVERRIDE}:forcevariable${LIBCOVERRIDE}"
OVERRIDES="linux:i586:build-linux:pn-bblayers:x86:qemuall:qemux86:poky:class-target:forcevariable:libc-glibc"

Putting the values of all these variables into OVERRIDES gives us a way of
setting variable values per-machine, per-architecture etc. Based on the
above value, if I had:

SOMEVAR = "value1"
SOMEVAR_qemux86 = "anothervalue"
SOMEVAR_i586 = "yetanothervalue"

The computed value of SOMEVAR would be "anothervalue". If MACHINE were
changed to, say, "qemumips", then OVERRIDES would change and the value 
of SOMEVAR would become "value1".

> Am I right in assuming the "variable name" consists of the uppercase parse
> (regardless of number of underscores)?

That is our (fairly strict) convention in OpenEmbedded, but FWIW I don't
think bitbake itself enforces that; it's all dependent on what is in OVERRIDES
and where the _ character appears in the left hand side of the assignment
statement.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: What are _virtual providers? and other Suffixes?

[Brad Litterell]

Hi Paul,

Thanks for taking the time to explain, very enlightening.  I think I understood it, please allow me to play back my understanding:

1.  _virtual is part of the variable name, and is not a special type of override.
2.  PREFERRED_PROVIDER_virtual/kernel_am335x-evm breaks into: "PREFERRED_PROVIDER_virtual/kernel" with an override condition of  "am335x-evm"
3. So for the jpeg case:

>> PREFERRED_PROVIDER_jpeg = "libjpeg-turbo"
>> 
>> PREFERRED_PROVIDER_jpeg_armv5te = "jpeg"

Could this have also been _virtual/jpeg?  It's just that some components use the _virtual convention and others don't?


Do I have it right?

Thanks,
Brad



On Aug 20, 2013, at 3:27 PM, Paul Eggleton <paul.eggleton@linux.intel.com>
 wrote:

> Hi Brad,
> 
> On Monday 19 August 2013 22:51:04 Brad Litterell wrote:
>> I searched the Yocto Mega Manual, but am still somewhat mystified by the
>> suffix formatting of various variable - especially virtual ones like this:
>> 
>> 
>> PREFERRED_PROVIDER_virtual/gettext = "gettext"
>> 
>> PREFERRED_PROVIDER_virtual/kernel_am335x-evm = "linux-ti-staging"
>> 
>> PREFERRED_PROVIDER_virtual/bootloader_am335x-evm = "u-boot-ti-staging"
> 
> These are setting PREFERRED_PROVIDER for a number of targets. "virtual/xyz" is
> a convention we use where there are multiple providers for a particular
> component, so a recipe that requires that component can have virtual/xyz in
> its DEPENDS, each provider declares virtual/xyz in its PROVIDES, and then in
> configuration the desired recipe to provide that can be selected via 
> PREFERRED_PROVIDER_virtual/xyz = "...".
> 
> The second two examples have an additional override which makes them only
> applicable for a specific machine (am335x-evm). If you want to know how this
> mechanism works have a look at this:
> 
> http://docs.openembedded.org/bitbake/html/ch02.html#id439023
> 
> (We're in the process of updating the BitBake manual, so this old version will have to suffice.)
> 
>> Also, there seems to be some sort of prioritization going on with the
>> multiple levels underscore characters in places like this:
>> 
>> 
>> # Set the PREFERRED_PROVIDER for jpeg functionality based on the MACHINE
>> 
>> # architecture.  For armv7a devices libjpeg-turbo should be used to take
>> 
>> # advantage of the SIMD instructions.
>> 
>> PREFERRED_PROVIDER_jpeg = "libjpeg-turbo"
>> 
>> PREFERRED_PROVIDER_jpeg_armv5te = "jpeg"
>> 
>> 
>> (all examples from the arago project in arago-prefs.inc)
>> 
>> Is there a succinct way of viewing these suffixes that will allow me to wrap
>> my head around their various & myriad versions?
> 
> OVERRIDES is listed in order so that overrides for the later items are
> preferred. If you run bitbake -e | less and search for OVERRIDES (with the /
> key) you can see what the value is for the global environment; here is an
> example for my current configuration:
> 
> # $OVERRIDES [3 operations]
> #   set conf/bitbake.conf:670
> #     "${TARGET_OS}:${TRANSLATED_TARGET_ARCH}:build-${BUILD_OS}:pn-${PN}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${CLASSOVERRIDE}:forcevariable"
> #   set conf/bitbake.conf:671
> #     [vardepsexclude] "MACHINEOVERRIDES"
> #   postdot /home/paul/poky/poky/meta/conf/distro/include/tclibc-eglibc.inc:9
> #     "${LIBCOVERRIDE}"
> # computed:
> #   "${TARGET_OS}:${TRANSLATED_TARGET_ARCH}:build-${BUILD_OS}:pn-${PN}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${CLASSOVERRIDE}:forcevariable${LIBCOVERRIDE}"
> OVERRIDES="linux:i586:build-linux:pn-bblayers:x86:qemuall:qemux86:poky:class-target:forcevariable:libc-glibc"
> 
> Putting the values of all these variables into OVERRIDES gives us a way of
> setting variable values per-machine, per-architecture etc. Based on the
> above value, if I had:
> 
> SOMEVAR = "value1"
> SOMEVAR_qemux86 = "anothervalue"
> SOMEVAR_i586 = "yetanothervalue"
> 
> The computed value of SOMEVAR would be "anothervalue". If MACHINE were
> changed to, say, "qemumips", then OVERRIDES would change and the value 
> of SOMEVAR would become "value1".
> 
>> Am I right in assuming the "variable name" consists of the uppercase parse
>> (regardless of number of underscores)?
> 
> That is our (fairly strict) convention in OpenEmbedded, but FWIW I don't
> think bitbake itself enforces that; it's all dependent on what is in OVERRIDES
> and where the _ character appears in the left hand side of the assignment
> statement.
> 
> Cheers,
> Paul
> 
> -- 
> 
> Paul Eggleton
> Intel Open Source Technology Centre

Re: What are _virtual providers? and other Suffixes?

[Paul Eggleton]

On Tuesday 20 August 2013 23:16:56 Brad Litterell wrote:
> Thanks for taking the time to explain, very enlightening.  I think I
> understood it, please allow me to play back my understanding:
> 
> 1.  _virtual is part of the variable name, and is not a special type of
> override. 

Actually, virtual/kernel is an override as well. The way a few variables, 
including PREFERRED_PROVIDER, are used is that OVERRIDES is set to include the 
item being dealt with when the variable is read, thus specifically with 
PREFERRED_PROVIDER you always override it with the name of the target you wish 
to select the provider for.

> 2.  PREFERRED_PROVIDER_virtual/kernel_am335x-evm breaks into:
> "PREFERRED_PROVIDER_virtual/kernel" with an override condition of 
> "am335x-evm"

Outside of the part of the code where it's actually read, yes. Within that 
code the override "virtual/kernel" will be applied when it's looking to see 
what the provider for "virtual/kernel" should be, and thus it will get the 
appropriate value for the PREFERRED_PROVIDER variable. So technically the 
variable is just PREFERRED_PROVIDER.

> 3. So for the jpeg case:
> >> PREFERRED_PROVIDER_jpeg = "libjpeg-turbo"
> >> 
> >> PREFERRED_PROVIDER_jpeg_armv5te = "jpeg"
> 
> Could this have also been _virtual/jpeg?  It's just that some components use
> the _virtual convention and others don't?

The prefix is virtual/ not _virtual, and as I mentioned earlier virtual/ is 
just a convention (although there are a few isolated parts of the code that 
specifically look for the virtual/ prefix). The mechanism will work in the same 
way here; recipes that need jpeg decoding have "jpeg" in DEPENDS and providing 
libjpeg-turbo has jpeg in its PROVIDES, which it does, we can select between 
two different recipes providing that - jpeg and libjpeg-turbo (where the latter 
is provided at all of course, i.e. when you have the meta-oe layer in your 
configuration). 

I don't think there's any special reason we don't have virtual/jpeg rather 
than jpeg here other than that having multiple jpeg decoding libraries is a 
relatively new situation compared to others such as virtual/kernel, and there 
are already a bunch of recipes out there referring to "jpeg" in their DEPENDS. 
(One wonders why there is a need for multiple jpeg decoding libraries in the 
first place, but that's a different can of worms...)
 
Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: What are _virtual providers? and other Suffixes?

[Brad Litterell]

Hi Paul,

Thanks - that makes it clearer.  But now I have one other question to ask:

if  virtual/xyz is added to overrides when the item is dealt with, then in that case P_P_virtual/xyz_am335 has two overrides hanging off of the base variable PREFERRED_PROVIDER.

You also said earlier that the latest override applies, so is there some rule for multiple conditionals on a variable?

E.g. What happens in a case like the following?

OVERRIDES="foo1:bar2:car3"

VARIABLE_foo1_bar2 = "both"
VARIABLE_car3 = "last one"

what does VARIABLE wind up?  The first is "more specific" in that it matches two values in overrides, whereas car is last, but less specific.

Thanks & sorry if I'm missing something simple.

Brad



On Aug 20, 2013, at 4:33 PM, Paul Eggleton <paul.eggleton@linux.intel.com>
 wrote:

> On Tuesday 20 August 2013 23:16:56 Brad Litterell wrote:
>> Thanks for taking the time to explain, very enlightening.  I think I
>> understood it, please allow me to play back my understanding:
>> 
>> 1.  _virtual is part of the variable name, and is not a special type of
>> override. 
> 
> Actually, virtual/kernel is an override as well. The way a few variables, 
> including PREFERRED_PROVIDER, are used is that OVERRIDES is set to include the 
> item being dealt with when the variable is read, thus specifically with 
> PREFERRED_PROVIDER you always override it with the name of the target you wish 
> to select the provider for.
> 
>> 2.  PREFERRED_PROVIDER_virtual/kernel_am335x-evm breaks into:
>> "PREFERRED_PROVIDER_virtual/kernel" with an override condition of 
>> "am335x-evm"
> 
> Outside of the part of the code where it's actually read, yes. Within that 
> code the override "virtual/kernel" will be applied when it's looking to see 
> what the provider for "virtual/kernel" should be, and thus it will get the 
> appropriate value for the PREFERRED_PROVIDER variable. So technically the 
> variable is just PREFERRED_PROVIDER.
> 
>> 3. So for the jpeg case:
>>>> PREFERRED_PROVIDER_jpeg = "libjpeg-turbo"
>>>> 
>>>> PREFERRED_PROVIDER_jpeg_armv5te = "jpeg"
>> 
>> Could this have also been _virtual/jpeg?  It's just that some components use
>> the _virtual convention and others don't?
> 
> The prefix is virtual/ not _virtual, and as I mentioned earlier virtual/ is 
> just a convention (although there are a few isolated parts of the code that 
> specifically look for the virtual/ prefix). The mechanism will work in the same 
> way here; recipes that need jpeg decoding have "jpeg" in DEPENDS and providing 
> libjpeg-turbo has jpeg in its PROVIDES, which it does, we can select between 
> two different recipes providing that - jpeg and libjpeg-turbo (where the latter 
> is provided at all of course, i.e. when you have the meta-oe layer in your 
> configuration). 
> 
> I don't think there's any special reason we don't have virtual/jpeg rather 
> than jpeg here other than that having multiple jpeg decoding libraries is a 
> relatively new situation compared to others such as virtual/kernel, and there 
> are already a bunch of recipes out there referring to "jpeg" in their DEPENDS. 
> (One wonders why there is a need for multiple jpeg decoding libraries in the 
> first place, but that's a different can of worms...)
> 
> Cheers,
> Paul
> 
> -- 
> 
> Paul Eggleton
> Intel Open Source Technology Centre

Re: What are _virtual providers? and other Suffixes?

[Paul Eggleton]

Hi Brad,

On Tuesday 20 August 2013 23:42:36 Brad Litterell wrote:
> Thanks - that makes it clearer.  But now I have one other question to ask:
> 
> if  virtual/xyz is added to overrides when the item is dealt with, then in
> that case P_P_virtual/xyz_am335 has two overrides hanging off of the base
> variable PREFERRED_PROVIDER.
> 
> You also said earlier that the latest override applies, so is there some
> rule for multiple conditionals on a variable?

Yes, effectively all must be in OVERRIDES for the assignment statement to take 
effect.
 
> E.g. What happens in a case like the following?
> 
> OVERRIDES="foo1:bar2:car3"
> 
> VARIABLE_foo1_bar2 = "both"
> VARIABLE_car3 = "last one"
> 
> what does VARIABLE wind up?  The first is "more specific" in that it matches
> two values in overrides, whereas car is last, but less specific.

I would have thought that the value would have been "last one" however 
experimentation shows that "both" is what you actually get. I'm not exactly 
sure why. In practice I don't think we hit this kind of situation too often 
though, i.e. a mix of double and single overrides where the single override 
needs to take precedence.
 
Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: What are _virtual providers? and other Suffixes?

[Brad Litterell]

Paul, thanks so much! 

On Aug 21, 2013, at 7:26 AM, Paul Eggleton <paul.eggleton@linux.intel.com> wrote:

> Hi Brad,
> 
> On Tuesday 20 August 2013 23:42:36 Brad Litterell wrote:
>> Thanks - that makes it clearer.  But now I have one other question to ask:
>> 
>> if  virtual/xyz is added to overrides when the item is dealt with, then in
>> that case P_P_virtual/xyz_am335 has two overrides hanging off of the base
>> variable PREFERRED_PROVIDER.
>> 
>> You also said earlier that the latest override applies, so is there some
>> rule for multiple conditionals on a variable?
> 
> Yes, effectively all must be in OVERRIDES for the assignment statement to take 
> effect.
> 
>> E.g. What happens in a case like the following?
>> 
>> OVERRIDES="foo1:bar2:car3"
>> 
>> VARIABLE_foo1_bar2 = "both"
>> VARIABLE_car3 = "last one"
>> 
>> what does VARIABLE wind up?  The first is "more specific" in that it matches
>> two values in overrides, whereas car is last, but less specific.
> 
> I would have thought that the value would have been "last one" however 
> experimentation shows that "both" is what you actually get. I'm not exactly 
> sure why. In practice I don't think we hit this kind of situation too often 
> though, i.e. a mix of double and single overrides where the single override 
> needs to take precedence.
> 
> Cheers,
> Paul
> 
> -- 
> 
> Paul Eggleton
> Intel Open Source Technology Centre