dynamic layer dependencies in meta-security

dynamic layer dependencies in meta-security

[Patrick Ohly]

Hello Armin!

I'm trying to use the latest meta-security master in refkit. Refkit runs
yocto-compat-layer.py against its own layers, which then pulls in
meta-security as one of the dependencies. That's where I am hitting a
snag:

INFO: Detected layers:
INFO: meta-refkit: LayerType.DISTRO, /fast/work/intel-iot-refkit-pr/meta-refkit
INFO: 
INFO: Setting up for meta-refkit(LayerType.DISTRO), /fast/work/intel-iot-refkit-pr/meta-refkit
INFO: Adding layer dependency meta-selftest
INFO: Adding layer dependency meta-intel
INFO: Adding layer dependency meta-refkit-gateway
INFO: Adding layer dependency meta-refkit-core
INFO: Adding layer dependency meta-oic
INFO: Adding layer dependency meta-iot-web
INFO: Adding layer dependency meta-refkit-computervision
INFO: Adding layer dependency meta-intel-realsense
INFO: Adding layer dependency meta-clang
INFO: Adding layer dependency meta-oe
INFO: Adding layer dependency meta-refkit-industrial
INFO: Adding layer dependency meta-python
INFO: Adding layer dependency meta-ros
INFO: Adding layer dependency meta-iotqa
INFO: Adding layer dependency meta-security
INFO: Adding layer dependency meta-perl
INFO: Adding layer dependency meta-security-isafw
INFO: Adding layer dependency meta-filesystems
INFO: Adding layer dependency meta-gnome
INFO: Adding layer dependency meta-networking
INFO: Adding layer dependency meta-flatpak
Traceback (most recent call last):
  File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/yocto-compat-layer.py", line 203, in <module>
    ret =  main()
  File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/yocto-compat-layer.py", line 151, in main
    if not add_layer_dependencies(bblayersconf, layer, dep_layers, logger) or \
  File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/lib/compatlayer/__init__.py", line 195, in add_layer_dependencies
    output = check_command('Getting existing layers failed.', 'bitbake-layers show-layers').decode('utf-8')
  File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/lib/compatlayer/__init__.py", line 228, in check_command
    raise RuntimeError(msg)
RuntimeError: Getting existing layers failed.
Command: bitbake-layers show-layers
Output:
ERROR: Layer 'security' depends on layer 'xfce-layer', but this layer is not enabled in your configuration

The reason is the way how meta-security declares its dependencies:

LAYERDEPENDS_security = "core openembedded-layer perl-layer"
LAYERDEPENDS_security += "${@bb.utils.contains("DISTRO_FEATURES", "x11", "gnome-layer xfce-layer", "", d)}"
LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "tpm", "tpm-layer", "",d)}"
LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "tpm2", "tpm-layer", "",d)}"
LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "vtpm", "tpm-layer meta-filesystems", "",d)}"

I haven't checked how exactly yocto-compat-layer.py deals with this, but
the end result is that it doesn't add meta-xfce when adding
meta-security. Then when "bitbake-layers show-layers" is called, a
distro (poky) has been chosen where x11 is in DISTRO_FEATURES, causing
the dependency check to fail.

This highlights a conceptual issue with the dynamic
LAYERDEPENDS_security above: which distro or machine config is active?
Tools like yocto-compat-layer.py or the layer index have to do their
work before that is defined.

This is also problematic with multiconfig. The same bblayers.conf might
be used with one config where x11 is off and another where it is on.

I suspect LAYERDEPENDS simply can't be dynamic. It either has to include
everything that a layer might depend on (even when the additional
content doesn't end up being used), or it just depends on the essential
parts and then enables additional recipes only when the optional layers
they depend on are present. BBFILES_DYNAMIC can be used for that, or the
recipes themselves check BBCOLLECTIONS in anonymous python code and
raise a SkipRecipe exception when their pre-condition isn't satisfied.

Copying Paul (layer index) and Mark (oe-yocto-compat.py, setup tool) for
their input.

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: dynamic layer dependencies in meta-security

[Mark Hatle]

On 7/19/17 1:57 AM, Patrick Ohly wrote:
> Hello Armin!
> 
> I'm trying to use the latest meta-security master in refkit. Refkit runs
> yocto-compat-layer.py against its own layers, which then pulls in
> meta-security as one of the dependencies. That's where I am hitting a
> snag:
> 
> INFO: Detected layers:
> INFO: meta-refkit: LayerType.DISTRO, /fast/work/intel-iot-refkit-pr/meta-refkit
> INFO: 
> INFO: Setting up for meta-refkit(LayerType.DISTRO), /fast/work/intel-iot-refkit-pr/meta-refkit
> INFO: Adding layer dependency meta-selftest
> INFO: Adding layer dependency meta-intel
> INFO: Adding layer dependency meta-refkit-gateway
> INFO: Adding layer dependency meta-refkit-core
> INFO: Adding layer dependency meta-oic
> INFO: Adding layer dependency meta-iot-web
> INFO: Adding layer dependency meta-refkit-computervision
> INFO: Adding layer dependency meta-intel-realsense
> INFO: Adding layer dependency meta-clang
> INFO: Adding layer dependency meta-oe
> INFO: Adding layer dependency meta-refkit-industrial
> INFO: Adding layer dependency meta-python
> INFO: Adding layer dependency meta-ros
> INFO: Adding layer dependency meta-iotqa
> INFO: Adding layer dependency meta-security
> INFO: Adding layer dependency meta-perl
> INFO: Adding layer dependency meta-security-isafw
> INFO: Adding layer dependency meta-filesystems
> INFO: Adding layer dependency meta-gnome
> INFO: Adding layer dependency meta-networking
> INFO: Adding layer dependency meta-flatpak
> Traceback (most recent call last):
>   File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/yocto-compat-layer.py", line 203, in <module>
>     ret =  main()
>   File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/yocto-compat-layer.py", line 151, in main
>     if not add_layer_dependencies(bblayersconf, layer, dep_layers, logger) or \
>   File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/lib/compatlayer/__init__.py", line 195, in add_layer_dependencies
>     output = check_command('Getting existing layers failed.', 'bitbake-layers show-layers').decode('utf-8')
>   File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/lib/compatlayer/__init__.py", line 228, in check_command
>     raise RuntimeError(msg)
> RuntimeError: Getting existing layers failed.
> Command: bitbake-layers show-layers
> Output:
> ERROR: Layer 'security' depends on layer 'xfce-layer', but this layer is not enabled in your configuration
> 
> The reason is the way how meta-security declares its dependencies:
> 
> LAYERDEPENDS_security = "core openembedded-layer perl-layer"
> LAYERDEPENDS_security += "${@bb.utils.contains("DISTRO_FEATURES", "x11", "gnome-layer xfce-layer", "", d)}"
> LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "tpm", "tpm-layer", "",d)}"
> LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "tpm2", "tpm-layer", "",d)}"
> LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "vtpm", "tpm-layer meta-filesystems", "",d)}"
> 
> I haven't checked how exactly yocto-compat-layer.py deals with this, but
> the end result is that it doesn't add meta-xfce when adding
> meta-security. Then when "bitbake-layers show-layers" is called, a
> distro (poky) has been chosen where x11 is in DISTRO_FEATURES, causing
> the dependency check to fail.

The meta-freescale method of checking if a layer is present and adding features
is how they resolve this issue.  Instead of making it a dependency, they simply
check if it's already present and then add to it.

I agree the above is less then optimal way of doing things, and will prevent the
layer index from getting a clear list of what may be required.

Note, there is also LAYERRECOMMENDS_... in this case, the system could recommend
gnome-layer, xfce-layer, tpm-layer, meta-filesystems, and then use the
meta-freescale approach to only enabling thsoe features if the recommended
layers are present.

> This highlights a conceptual issue with the dynamic
> LAYERDEPENDS_security above: which distro or machine config is active?
> Tools like yocto-compat-layer.py or the layer index have to do their
> work before that is defined.
> 
> This is also problematic with multiconfig. The same bblayers.conf might
> be used with one config where x11 is off and another where it is on.
> 
> I suspect LAYERDEPENDS simply can't be dynamic. It either has to include
> everything that a layer might depend on (even when the additional
> content doesn't end up being used), or it just depends on the essential
> parts and then enables additional recipes only when the optional layers
> they depend on are present. BBFILES_DYNAMIC can be used for that, or the
> recipes themselves check BBCOLLECTIONS in anonymous python code and
> raise a SkipRecipe exception when their pre-condition isn't satisfied.
> 
> Copying Paul (layer index) and Mark (oe-yocto-compat.py, setup tool) for
> their input.

I would certainly caution against dynamic layerdepends/layerrecommends and
instead focus on listing everything between what is required and what you want
to add in addition -- then using the meta-freescale approach of only extending
(the recommends) when present.

The setup program and layer index can handle this situaton.  The
yocto-compat-layer checking may have to be augmented to deal with recommends as
well, but that should be fairly minor.

--Mark

Re: dynamic layer dependencies in meta-security

[akuster808]

On 07/19/2017 05:29 AM, Mark Hatle wrote:
> On 7/19/17 1:57 AM, Patrick Ohly wrote:
>> Hello Armin!
>>
>> I'm trying to use the latest meta-security master in refkit. Refkit runs
>> yocto-compat-layer.py against its own layers, which then pulls in
>> meta-security as one of the dependencies. That's where I am hitting a
>> snag:
>>
>> INFO: Detected layers:
>> INFO: meta-refkit: LayerType.DISTRO, /fast/work/intel-iot-refkit-pr/meta-refkit
>> INFO:
>> INFO: Setting up for meta-refkit(LayerType.DISTRO), /fast/work/intel-iot-refkit-pr/meta-refkit
>> INFO: Adding layer dependency meta-selftest
>> INFO: Adding layer dependency meta-intel
>> INFO: Adding layer dependency meta-refkit-gateway
>> INFO: Adding layer dependency meta-refkit-core
>> INFO: Adding layer dependency meta-oic
>> INFO: Adding layer dependency meta-iot-web
>> INFO: Adding layer dependency meta-refkit-computervision
>> INFO: Adding layer dependency meta-intel-realsense
>> INFO: Adding layer dependency meta-clang
>> INFO: Adding layer dependency meta-oe
>> INFO: Adding layer dependency meta-refkit-industrial
>> INFO: Adding layer dependency meta-python
>> INFO: Adding layer dependency meta-ros
>> INFO: Adding layer dependency meta-iotqa
>> INFO: Adding layer dependency meta-security
>> INFO: Adding layer dependency meta-perl
>> INFO: Adding layer dependency meta-security-isafw
>> INFO: Adding layer dependency meta-filesystems
>> INFO: Adding layer dependency meta-gnome
>> INFO: Adding layer dependency meta-networking
>> INFO: Adding layer dependency meta-flatpak
>> Traceback (most recent call last):
>>    File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/yocto-compat-layer.py", line 203, in <module>
>>      ret =  main()
>>    File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/yocto-compat-layer.py", line 151, in main
>>      if not add_layer_dependencies(bblayersconf, layer, dep_layers, logger) or \
>>    File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/lib/compatlayer/__init__.py", line 195, in add_layer_dependencies
>>      output = check_command('Getting existing layers failed.', 'bitbake-layers show-layers').decode('utf-8')
>>    File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/lib/compatlayer/__init__.py", line 228, in check_command
>>      raise RuntimeError(msg)
>> RuntimeError: Getting existing layers failed.
>> Command: bitbake-layers show-layers
>> Output:
>> ERROR: Layer 'security' depends on layer 'xfce-layer', but this layer is not enabled in your configuration
Thats not good. Thanks for the feedback.

>>
>> The reason is the way how meta-security declares its dependencies:
>>
>> LAYERDEPENDS_security = "core openembedded-layer perl-layer"
>> LAYERDEPENDS_security += "${@bb.utils.contains("DISTRO_FEATURES", "x11", "gnome-layer xfce-layer", "", d)}"
>> LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "tpm", "tpm-layer", "",d)}"
>> LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "tpm2", "tpm-layer", "",d)}"
>> LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "vtpm", "tpm-layer meta-filesystems", "",d)}"
>>
>> I haven't checked how exactly yocto-compat-layer.py deals with this, but
>> the end result is that it doesn't add meta-xfce when adding
>> meta-security. Then when "bitbake-layers show-layers" is called, a
>> distro (poky) has been chosen where x11 is in DISTRO_FEATURES, causing
>> the dependency check to fail.
> The meta-freescale method of checking if a layer is present and adding features
> is how they resolve this issue.  Instead of making it a dependency, they simply
> check if it's already present and then add to it.
>
> I agree the above is less then optimal way of doing things, and will prevent the
> layer index from getting a clear list of what may be required.
>
> Note, there is also LAYERRECOMMENDS_... in this case, the system could recommend
> gnome-layer, xfce-layer, tpm-layer, meta-filesystems, and then use the
> meta-freescale approach to only enabling thsoe features if the recommended
> layers are present.
Thanks. I will take a look at that.

>
>> This highlights a conceptual issue with the dynamic
>> LAYERDEPENDS_security above: which distro or machine config is active?
>> Tools like yocto-compat-layer.py or the layer index have to do their
>> work before that is defined.
>>
>> This is also problematic with multiconfig. The same bblayers.conf might
>> be used with one config where x11 is off and another where it is on.
>>
>> I suspect LAYERDEPENDS simply can't be dynamic. It either has to include
>> everything that a layer might depend on (even when the additional
>> content doesn't end up being used), or it just depends on the essential
>> parts and then enables additional recipes only when the optional layers
>> they depend on are present. BBFILES_DYNAMIC can be used for that, or the
>> recipes themselves check BBCOLLECTIONS in anonymous python code and
>> raise a SkipRecipe exception when their pre-condition isn't satisfied.
>>
>> Copying Paul (layer index) and Mark (oe-yocto-compat.py, setup tool) for
>> their input.
> I would certainly caution against dynamic layerdepends/layerrecommends and
> instead focus on listing everything between what is required and what you want
> to add in addition -- then using the meta-freescale approach of only extending
> (the recommends) when present.
This appears to be more beneficial in a  bbappends than a recipe needing 
a package from another layer.

>
> The setup program and layer index can handle this situaton.  The
> yocto-compat-layer checking may have to be augmented to deal with recommends as
> well, but that should be fairly minor.
>
> --Mark
Thanks for the detailed explanation. I will look into the and submit 
changes soon.
-armin

Re: dynamic layer dependencies in meta-security

[Patrick Ohly]

On Thu, 2017-07-20 at 14:39 -0700, akuster808 wrote:
> > I would certainly caution against dynamic layerdepends/layerrecommends and
> > instead focus on listing everything between what is required and what you want
> > to add in addition -- then using the meta-freescale approach of only extending
> > (the recommends) when present.

FWIW, the meta-freescale approach has been superseeded by
BBFILES_DYNAMIC.

> This appears to be more beneficial in a  bbappends than a recipe needing 
> a package from another layer.

BBFILES_DYNAMIC can be used for both .bb and .bbappend files.

In refkit, I ended up just using it for .bbappend files:
https://github.com/intel/intel-iot-refkit/blob/master/meta-refkit-core/conf/layer.conf#L7

        # All our .bbappends for other layers are in a separate
        # "bbappends/<layer>" hierarchy. We activate only those
        # bbappends for which the layer they apply to is actually
        # present.
        #
        # Sorted by layer path to keep related layers together.
        BBFILES_DYNAMIC += " \
        clang-layer:${LAYERDIR}/bbappends/meta-clang/*/*/*.bbappend \
        flatpak:${LAYERDIR}/bbappends/meta-flatpak/*/*/*.bbappend \
        
For recipes, I wanted to have a bit more flexibility:
- let developers decide from where they get the dependency - the
  well-known layer or a copy elsewhere
- let recipes decide which features they enable based on which
  dependencies are available
        
Also, "bitbake foo" should result in an error about "dependency bar
required for foo not found", not a blunt "nothing provides foo". With
BBFILES_DYNAMIC for .bb files one only gets the latter.

Here's how it is defined what's available:
https://github.com/intel/intel-iot-refkit/blob/master/meta-refkit-core/conf/layer.conf#L51
        
                # There are multiple different ways for providing some of the
                # dependencies. Here we assume that the dependencies are available if
                # the layers that the refkit distro takes them from are present.
                HAVE_META_OE = "${@ bb.utils.contains('BBFILE_COLLECTIONS', 'openembedded-layer', 'True', 'False', d) }"
                HAVE_ATOP ??= "${HAVE_META_OE}"
            HAVE_CRYPTSETUP ??= "${HAVE_META_OE}"
            ...

And here's how it is used:
https://github.com/intel/intel-iot-refkit/blob/master/meta-refkit-core/recipes-images/images/initramfs-framework-refkit-dm-verity.bb#L22

        python () {
            if not oe.types.boolean(d.getVar('HAVE_CRYPTSETUP') or '0'):
                raise bb.parse.SkipRecipe('cryptsetup dependency not available')
        }


-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: dynamic layer dependencies in meta-security

[Mark Hatle]

On 7/20/17 11:37 PM, Patrick Ohly wrote:
> On Thu, 2017-07-20 at 14:39 -0700, akuster808 wrote:
>>> I would certainly caution against dynamic layerdepends/layerrecommends and
>>> instead focus on listing everything between what is required and what you want
>>> to add in addition -- then using the meta-freescale approach of only extending
>>> (the recommends) when present.
> 
> FWIW, the meta-freescale approach has been superseeded by
> BBFILES_DYNAMIC.

BBFILES_DYNAMIC requires master.  You need to use the meta-freescale approach
for any of the existing released trees.

You are correct it is the right approach for master and future, but we should
fix existing trees as well, since it does affect the output of the layer index
for people looking at existing releases.

>> This appears to be more beneficial in a  bbappends than a recipe needing 
>> a package from another layer.
> 
> BBFILES_DYNAMIC can be used for both .bb and .bbappend files.
> 
> In refkit, I ended up just using it for .bbappend files:
> https://github.com/intel/intel-iot-refkit/blob/master/meta-refkit-core/conf/layer.conf#L7
> 
>         # All our .bbappends for other layers are in a separate
>         # "bbappends/<layer>" hierarchy. We activate only those
>         # bbappends for which the layer they apply to is actually
>         # present.
>         #
>         # Sorted by layer path to keep related layers together.
>         BBFILES_DYNAMIC += " \
>         clang-layer:${LAYERDIR}/bbappends/meta-clang/*/*/*.bbappend \
>         flatpak:${LAYERDIR}/bbappends/meta-flatpak/*/*/*.bbappend \

Even when using this, there should be a LAYERRECOMMENDS for clang-layer and
flatpak IMHO.  Otherwise the layer index and other automated tools will have no
way to determine that the user should have those items.

> For recipes, I wanted to have a bit more flexibility:
> - let developers decide from where they get the dependency - the
>   well-known layer or a copy elsewhere
> - let recipes decide which features they enable based on which
>   dependencies are available
>         
> Also, "bitbake foo" should result in an error about "dependency bar
> required for foo not found", not a blunt "nothing provides foo". With
> BBFILES_DYNAMIC for .bb files one only gets the latter.

Problem of course is that if we bring in a .bb that requires another layer (in
certain ways) then the parse can fail, or you have to start 'protecting' certain
things (like inherits/requires/etc) so the parse can complete.  This becomes a
big problem very quickly.

> Here's how it is defined what's available:
> https://github.com/intel/intel-iot-refkit/blob/master/meta-refkit-core/conf/layer.conf#L51
>         
>                 # There are multiple different ways for providing some of the
>                 # dependencies. Here we assume that the dependencies are available if
>                 # the layers that the refkit distro takes them from are present.
>                 HAVE_META_OE = "${@ bb.utils.contains('BBFILE_COLLECTIONS', 'openembedded-layer', 'True', 'False', d) }"
>                 HAVE_ATOP ??= "${HAVE_META_OE}"
>             HAVE_CRYPTSETUP ??= "${HAVE_META_OE}"
>             ...

I have mixed feeling on this approach... I certainly understand it, and get why
it's needed.  (I've used it myself as well...)  but in the end you are
essentially defining a distribution flag (variable?) that is being set by the
layer.conf based on the existence of another layer.  This would be very
confusing to someone trying to understand why two different builds are getting
different behaviors.

The best of the 'bad situation' would be to implement (in the recipes) a
PKGCONFIG where the default uses these magic variables to define the defaults.
Then the user can have a chance at debugging the issue by seeing that the
PKGCONFIG (default) is changing from one build to another.  Using the variables
directly during the build is very tempting, but I would caution against it.

> And here's how it is used:
> https://github.com/intel/intel-iot-refkit/blob/master/meta-refkit-core/recipes-images/images/initramfs-framework-refkit-dm-verity.bb#L22
> 
>         python () {
>             if not oe.types.boolean(d.getVar('HAVE_CRYPTSETUP') or '0'):
>                 raise bb.parse.SkipRecipe('cryptsetup dependency not available')
>         }

This is certainly nicer then the other way.  It may be possible in bitbake to
implement this type of work directly.  I.e. in the BBFILES_DYNAMIC processing.
If the layer isn't there, we could still process the glob and report the
skipRecipe.  (I'm sure I'm making it sound easier then it would be to implement.)

--Mark