[yocto-docs PATCH v3 0/2] Document how to provide configuration from layer.conf

[yocto-docs PATCH v3 0/2] Document how to provide configuration from layer.conf

[Antonin Godard]

This series documents how to provide configurations from layer.conf, be
it unconditionally or conditionally. I wanted to reference a file in
bitbake, so I added the :bitbake_git: new term for that.

This is a attempt at closing
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12688.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
---
Changes in v3:
- Replace attention block by warning block.
- Create an anchor for the conditional confs section, I was getting a
  sphinx error.
- Rephrase the PREFERRED_PROVIDER sentence.
- More precision on PREFERRED_PROVIDER assignments, with other conf
  files..
- Link to v2: https://lore.kernel.org/r/20241112-layer-confs-v2-0-22a09f1a0a8e@bootlin.com

Changes in v2:
- Create an `.. attention::` block to tell users _not_ to set
  unconditional confs from layer.conf.
- Fix typos reported by Quentin (thanks!)
- Add "Reviewed-by: Quentin Schulz <quentin.schulz@cherry.de>" on first
  commit (thanks!).
- Simplify "your layer is part of BBLAYERS" 🡒 "your layer is part of the
  build".
- Fix code blocks spacing (three spaces).
- Link to v1: https://lore.kernel.org/r/20241030-layer-confs-v1-0-0f7dcf460e27@bootlin.com

---
Antonin Godard (2):
      conf.py: add a bitbake_git extlink
      dev-manual: document how to provide confs from layer.conf

 documentation/conf.py               |  1 +
 documentation/dev-manual/layers.rst | 90 +++++++++++++++++++++++++++++++++++++
 2 files changed, 91 insertions(+)
---
base-commit: 13caec1386708d8609dff5f42956d2329a074f37
change-id: 20241030-layer-confs-54c26ece97d3

Best regards,
-- 
Antonin Godard <antonin.godard@bootlin.com>

[yocto-docs PATCH v3 1/2] conf.py: add a bitbake_git extlink

[Antonin Godard]

To make references to the bitbake repo, add an extlink for it and use it
in the docs with ":bitbake_git:`lib/bb/utils.py </tree/lib/bb/utils.py>`".

Reviewed-by: Quentin Schulz <quentin.schulz@cherry.de>
Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
---
 documentation/conf.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/documentation/conf.py b/documentation/conf.py
index c7dd4e03fc0e99176a0fa2f74fda58efaf0af9b2..b0e7c91eb5d8e4e855c0ca46351586cdf237fcb3 100644
--- a/documentation/conf.py
+++ b/documentation/conf.py
@@ -90,6 +90,7 @@ rst_prolog = """
 
 # external links and substitutions
 extlinks = {
+    'bitbake_git': ('https://git.openembedded.org/bitbake%s', None),
     'cve_mitre': ('https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-%s', 'CVE-%s'),
     'cve_nist': ('https://nvd.nist.gov/vuln/detail/CVE-%s', 'CVE-%s'),
     'yocto_home': ('https://www.yoctoproject.org%s', None),

-- 
2.46.1

[yocto-docs PATCH v3 2/2] dev-manual: document how to provide confs from layer.conf

[Antonin Godard]

Add a section on providing global level configuration from the
layer.conf file. Since this file is parsed at an earlier stage in the
parsing process, it's not possible to combine bb.utils.contains and
{DISTRO,MACHINE}_FEATURES to conditionally set some configurations.

This patch documents:

- First that this file can be used for providing such configuration.
- Then demonstrate how to conditionally provide them, using a technique
  that is currently used in meta-virtualization
  (https://git.yoctoproject.org/meta-virtualization/tree/conf/layer.conf#n50).

Fixes [YOCTO #12688].

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
---
 documentation/dev-manual/layers.rst | 90 +++++++++++++++++++++++++++++++++++++
 1 file changed, 90 insertions(+)

diff --git a/documentation/dev-manual/layers.rst b/documentation/dev-manual/layers.rst
index 91889bd0ae391c7b6e62068778ae5c180c840762..ac18321b04d08bceae63672cf6da26660646fc59 100644
--- a/documentation/dev-manual/layers.rst
+++ b/documentation/dev-manual/layers.rst
@@ -644,6 +644,96 @@ variable and append the layer's root name::
    order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
    might address this.
 
+Providing Global-level Configurations With Your Layer
+-----------------------------------------------------
+
+When creating a layer, you may need to define configurations that should take
+effect globally in your build environment when the layer is part of the build.
+The ``layer.conf`` file is a :term:`configuration file` that affects the build
+system globally, so it is a candidate for this use-case.
+
+.. warning::
+
+   Providing unconditional global level configuration from the ``layer.conf``
+   file is *not* a good practice, and should be avoided. For this reason, the
+   section :ref:`ref-conditional-layer-confs` below shows how the ``layer.conf``
+   file can be used to provide configurations only if a certain condition is
+   met.
+
+For example, if your layer provides a Linux kernel recipe named
+``linux-custom``, you may want to make :term:`PREFERRED_PROVIDER_virtual/kernel
+<PREFERRED_PROVIDER>` point to ``linux-custom``::
+
+  PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
+
+This can be defined in the ``layer.conf`` file. If your layer is at the last
+position in the :term:`BBLAYERS` list, it will take precedence over previous
+``PREFERRED_PROVIDER_virtual/kernel`` assignments (unless it is set from a
+:term:`configuration file` that is parsed later, such as machine or distro
+configuration files).
+
+.. _ref-conditional-layer-confs:
+
+Conditionally Provide Global-level Configurations With Your Layer
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In some cases, your layer may provide global configurations only if some
+features it provides are enabled. Since the ``layer.conf`` file is parsed at an
+earlier stage in the parsing process, the :term:`DISTRO_FEATURES` and
+:term:`MACHINE_FEATURES` variables are not yet available to ``layer.conf``, and
+declaring conditional assignments based on these variables is not possible. The
+following technique shows a way to bypass this limitation by using the
+:term:`USER_CLASSES` variable and a conditional ``include`` command.
+
+In the following steps, let's assume our layer is named ``meta-mylayer`` and
+that this layer defines a custom :ref:`distro feature <ref-features-distro>`
+named ``mylayer-kernel``. We will set the :term:`PREFERRED_PROVIDER` variable
+for the kernel only if our feature ``mylayer-kernel`` is part of the
+:term:`DISTRO_FEATURES`:
+
+#. Create an include file in the directory
+   ``meta-mylayer/conf/distro/include/``, for example a file named
+   ``mylayer-kernel-provider.inc`` that sets the kernel provider to
+   ``linux-custom``::
+
+      PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
+
+#. Provide a path to this include file in your ``layer.conf``::
+
+      META_MYLAYER_KERNEL_PROVIDER_PATH = "${LAYERDIR}/conf/distro/include/mylayer-kernel-provider.inc"
+
+#. Create a new class in ``meta-mylayer/classes-global/``, for example a class
+   ``meta-mylayer-cfg.bbclass``. Make it conditionally include the file
+   ``mylayer-kernel-provider.inc`` defined above, using the variable
+   ``META_MYLAYER_KERNEL_PROVIDER_PATH`` defined in ``layer.conf``::
+
+      require ${@bb.utils.contains('DISTRO_FEATURES', 'mylayer-kernel', '${META_MYLAYER_KERNEL_PROVIDER_PATH}', '', d)}
+
+   For details on the ``bb.utils.contains`` function, see its definition in
+   :bitbake_git:`lib/bb/utils.py </tree/lib/bb/utils.py>`.
+
+   .. note::
+
+      The ``require`` command is designed to not fail if the function
+      ``bb.utils.contains`` returns an empty string.
+
+#. Back to your ``layer.conf`` file, add the class ``meta-mylayer-cfg`` class to
+   the :term:`USER_CLASSES` variable::
+
+      USER_CLASSES:append = " meta-mylayer-cfg"
+
+   This will add the class ``meta-mylayer-cfg`` to the list of classes to
+   globally inherit. Since the ``include`` command is conditional in
+   ``meta-mylayer-cfg.bbclass``, even though inherited the class will have no
+   effect until the feature ``mylayer-kernel`` is enabled through
+   :term:`DISTRO_FEATURES`.
+
+This technique can also be used for :ref:`Machine features
+<ref-features-machine>` by following the same steps. Though not mandatory, it is
+recommended to put include files for :term:`DISTRO_FEATURES` in your layer's
+``conf/distro/include`` and the ones for :term:`MACHINE_FEATURES` in your
+layer's ``conf/machine/include``.
+
 Managing Layers
 ===============
 

-- 
2.46.1

Re: [yocto-docs PATCH v3 2/2] dev-manual: document how to provide confs from layer.conf

[Quentin Schulz]

Hi Antonin,

On 11/13/24 3:30 PM, Antonin Godard wrote:
> Add a section on providing global level configuration from the
> layer.conf file. Since this file is parsed at an earlier stage in the
> parsing process, it's not possible to combine bb.utils.contains and
> {DISTRO,MACHINE}_FEATURES to conditionally set some configurations.
> 
> This patch documents:
> 
> - First that this file can be used for providing such configuration.
> - Then demonstrate how to conditionally provide them, using a technique
>    that is currently used in meta-virtualization
>    (https://git.yoctoproject.org/meta-virtualization/tree/conf/layer.conf#n50).
> 
> Fixes [YOCTO #12688].
> 
> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
> ---
>   documentation/dev-manual/layers.rst | 90 +++++++++++++++++++++++++++++++++++++
>   1 file changed, 90 insertions(+)
> 
> diff --git a/documentation/dev-manual/layers.rst b/documentation/dev-manual/layers.rst
> index 91889bd0ae391c7b6e62068778ae5c180c840762..ac18321b04d08bceae63672cf6da26660646fc59 100644
> --- a/documentation/dev-manual/layers.rst
> +++ b/documentation/dev-manual/layers.rst
> @@ -644,6 +644,96 @@ variable and append the layer's root name::
>      order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
>      might address this.
>   
> +Providing Global-level Configurations With Your Layer
> +-----------------------------------------------------
> +
> +When creating a layer, you may need to define configurations that should take
> +effect globally in your build environment when the layer is part of the build.
> +The ``layer.conf`` file is a :term:`configuration file` that affects the build
> +system globally, so it is a candidate for this use-case.
> +
> +.. warning::
> +
> +   Providing unconditional global level configuration from the ``layer.conf``
> +   file is *not* a good practice, and should be avoided. For this reason, the
> +   section :ref:`ref-conditional-layer-confs` below shows how the ``layer.conf``
> +   file can be used to provide configurations only if a certain condition is
> +   met.
> +
> +For example, if your layer provides a Linux kernel recipe named
> +``linux-custom``, you may want to make :term:`PREFERRED_PROVIDER_virtual/kernel
> +<PREFERRED_PROVIDER>` point to ``linux-custom``::
> +
> +  PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
> +
> +This can be defined in the ``layer.conf`` file. If your layer is at the last
> +position in the :term:`BBLAYERS` list, it will take precedence over previous
> +``PREFERRED_PROVIDER_virtual/kernel`` assignments (unless it is set from a

s/unless it/unless one/

The issue is we have two "it" in that sentence, one for representing the 
layer and the other the PREFERRED_PROVIDER_virtual/kernel assignment. 
What's a bit confusing is that "it" should be understood as *an* 
assignment whereas the word is in its plural form before.

> +:term:`configuration file` that is parsed later, such as machine or distro
> +configuration files).
> +
> +.. _ref-conditional-layer-confs:
> +
> +Conditionally Provide Global-level Configurations With Your Layer
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +In some cases, your layer may provide global configurations only if some
> +features it provides are enabled. Since the ``layer.conf`` file is parsed at an
> +earlier stage in the parsing process, the :term:`DISTRO_FEATURES` and
> +:term:`MACHINE_FEATURES` variables are not yet available to ``layer.conf``, and
> +declaring conditional assignments based on these variables is not possible. The
> +following technique shows a way to bypass this limitation by using the
> +:term:`USER_CLASSES` variable and a conditional ``include`` command.
> +
> +In the following steps, let's assume our layer is named ``meta-mylayer`` and
> +that this layer defines a custom :ref:`distro feature <ref-features-distro>`
> +named ``mylayer-kernel``. We will set the :term:`PREFERRED_PROVIDER` variable
> +for the kernel only if our feature ``mylayer-kernel`` is part of the
> +:term:`DISTRO_FEATURES`:
> +
> +#. Create an include file in the directory
> +   ``meta-mylayer/conf/distro/include/``, for example a file named
> +   ``mylayer-kernel-provider.inc`` that sets the kernel provider to
> +   ``linux-custom``::
> +
> +      PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
> +
> +#. Provide a path to this include file in your ``layer.conf``::
> +
> +      META_MYLAYER_KERNEL_PROVIDER_PATH = "${LAYERDIR}/conf/distro/include/mylayer-kernel-provider.inc"
> +
> +#. Create a new class in ``meta-mylayer/classes-global/``, for example a class
> +   ``meta-mylayer-cfg.bbclass``. Make it conditionally include the file
> +   ``mylayer-kernel-provider.inc`` defined above, using the variable
> +   ``META_MYLAYER_KERNEL_PROVIDER_PATH`` defined in ``layer.conf``::
> +
> +      require ${@bb.utils.contains('DISTRO_FEATURES', 'mylayer-kernel', '${META_MYLAYER_KERNEL_PROVIDER_PATH}', '', d)}
> +
> +   For details on the ``bb.utils.contains`` function, see its definition in
> +   :bitbake_git:`lib/bb/utils.py </tree/lib/bb/utils.py>`.
> +
> +   .. note::
> +
> +      The ``require`` command is designed to not fail if the function
> +      ``bb.utils.contains`` returns an empty string.
> +
> +#. Back to your ``layer.conf`` file, add the class ``meta-mylayer-cfg`` class to
> +   the :term:`USER_CLASSES` variable::
> +
> +      USER_CLASSES:append = " meta-mylayer-cfg"
> +
> +   This will add the class ``meta-mylayer-cfg`` to the list of classes to
> +   globally inherit. Since the ``include`` command is conditional in

s/include/require/

We changed include for require in v2 I believe, we probably forgot to 
update that one as well :)

> +   ``meta-mylayer-cfg.bbclass``, even though inherited the class will have no
> +   effect until the feature ``mylayer-kernel`` is enabled through

s/until/unless/ ?

With the s/include/require/:

Reviewed-by: Quentin Schulz <quentin.schulz@cherry.de>

Thanks!
Quentin

Re: [yocto-docs PATCH v3 2/2] dev-manual: document how to provide confs from layer.conf

[Antonin Godard]

Hi Quentin,

On Wed Nov 13, 2024 at 3:36 PM CET, Quentin Schulz wrote:
> Hi Antonin,
>
> On 11/13/24 3:30 PM, Antonin Godard wrote:
>> Add a section on providing global level configuration from the
>> layer.conf file. Since this file is parsed at an earlier stage in the
>> parsing process, it's not possible to combine bb.utils.contains and
>> {DISTRO,MACHINE}_FEATURES to conditionally set some configurations.
>> 
>> This patch documents:
>> 
>> - First that this file can be used for providing such configuration.
>> - Then demonstrate how to conditionally provide them, using a technique
>>    that is currently used in meta-virtualization
>>    (https://git.yoctoproject.org/meta-virtualization/tree/conf/layer.conf#n50).
>> 
>> Fixes [YOCTO #12688].
>> 
>> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
>> ---
>>   documentation/dev-manual/layers.rst | 90 +++++++++++++++++++++++++++++++++++++
>>   1 file changed, 90 insertions(+)
>> 
>> diff --git a/documentation/dev-manual/layers.rst b/documentation/dev-manual/layers.rst
>> index 91889bd0ae391c7b6e62068778ae5c180c840762..ac18321b04d08bceae63672cf6da26660646fc59 100644
>> --- a/documentation/dev-manual/layers.rst
>> +++ b/documentation/dev-manual/layers.rst
>> @@ -644,6 +644,96 @@ variable and append the layer's root name::
>>      order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
>>      might address this.
>>   
>> +Providing Global-level Configurations With Your Layer
>> +-----------------------------------------------------
>> +
>> +When creating a layer, you may need to define configurations that should take
>> +effect globally in your build environment when the layer is part of the build.
>> +The ``layer.conf`` file is a :term:`configuration file` that affects the build
>> +system globally, so it is a candidate for this use-case.
>> +
>> +.. warning::
>> +
>> +   Providing unconditional global level configuration from the ``layer.conf``
>> +   file is *not* a good practice, and should be avoided. For this reason, the
>> +   section :ref:`ref-conditional-layer-confs` below shows how the ``layer.conf``
>> +   file can be used to provide configurations only if a certain condition is
>> +   met.
>> +
>> +For example, if your layer provides a Linux kernel recipe named
>> +``linux-custom``, you may want to make :term:`PREFERRED_PROVIDER_virtual/kernel
>> +<PREFERRED_PROVIDER>` point to ``linux-custom``::
>> +
>> +  PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
>> +
>> +This can be defined in the ``layer.conf`` file. If your layer is at the last
>> +position in the :term:`BBLAYERS` list, it will take precedence over previous
>> +``PREFERRED_PROVIDER_virtual/kernel`` assignments (unless it is set from a
>
> s/unless it/unless one/
>
> The issue is we have two "it" in that sentence, one for representing the 
> layer and the other the PREFERRED_PROVIDER_virtual/kernel assignment. 
> What's a bit confusing is that "it" should be understood as *an* 
> assignment whereas the word is in its plural form before.

+1, thanks!

>> +:term:`configuration file` that is parsed later, such as machine or distro
>> +configuration files).
>> +
>> +.. _ref-conditional-layer-confs:
>> +
>> +Conditionally Provide Global-level Configurations With Your Layer
>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> +
>> +In some cases, your layer may provide global configurations only if some
>> +features it provides are enabled. Since the ``layer.conf`` file is parsed at an
>> +earlier stage in the parsing process, the :term:`DISTRO_FEATURES` and
>> +:term:`MACHINE_FEATURES` variables are not yet available to ``layer.conf``, and
>> +declaring conditional assignments based on these variables is not possible. The
>> +following technique shows a way to bypass this limitation by using the
>> +:term:`USER_CLASSES` variable and a conditional ``include`` command.
>> +
>> +In the following steps, let's assume our layer is named ``meta-mylayer`` and
>> +that this layer defines a custom :ref:`distro feature <ref-features-distro>`
>> +named ``mylayer-kernel``. We will set the :term:`PREFERRED_PROVIDER` variable
>> +for the kernel only if our feature ``mylayer-kernel`` is part of the
>> +:term:`DISTRO_FEATURES`:
>> +
>> +#. Create an include file in the directory
>> +   ``meta-mylayer/conf/distro/include/``, for example a file named
>> +   ``mylayer-kernel-provider.inc`` that sets the kernel provider to
>> +   ``linux-custom``::
>> +
>> +      PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
>> +
>> +#. Provide a path to this include file in your ``layer.conf``::
>> +
>> +      META_MYLAYER_KERNEL_PROVIDER_PATH = "${LAYERDIR}/conf/distro/include/mylayer-kernel-provider.inc"
>> +
>> +#. Create a new class in ``meta-mylayer/classes-global/``, for example a class
>> +   ``meta-mylayer-cfg.bbclass``. Make it conditionally include the file
>> +   ``mylayer-kernel-provider.inc`` defined above, using the variable
>> +   ``META_MYLAYER_KERNEL_PROVIDER_PATH`` defined in ``layer.conf``::
>> +
>> +      require ${@bb.utils.contains('DISTRO_FEATURES', 'mylayer-kernel', '${META_MYLAYER_KERNEL_PROVIDER_PATH}', '', d)}
>> +
>> +   For details on the ``bb.utils.contains`` function, see its definition in
>> +   :bitbake_git:`lib/bb/utils.py </tree/lib/bb/utils.py>`.
>> +
>> +   .. note::
>> +
>> +      The ``require`` command is designed to not fail if the function
>> +      ``bb.utils.contains`` returns an empty string.
>> +
>> +#. Back to your ``layer.conf`` file, add the class ``meta-mylayer-cfg`` class to
>> +   the :term:`USER_CLASSES` variable::
>> +
>> +      USER_CLASSES:append = " meta-mylayer-cfg"
>> +
>> +   This will add the class ``meta-mylayer-cfg`` to the list of classes to
>> +   globally inherit. Since the ``include`` command is conditional in
>
> s/include/require/
>
> We changed include for require in v2 I believe, we probably forgot to 
> update that one as well :)

Yes, I forgot to update them, I will send a v3.

>> +   ``meta-mylayer-cfg.bbclass``, even though inherited the class will have no
>> +   effect until the feature ``mylayer-kernel`` is enabled through
>
> s/until/unless/ ?

Yes, thanks!

> With the s/include/require/:
>
> Reviewed-by: Quentin Schulz <quentin.schulz@cherry.de>
>
> Thanks!
> Quentin

Thanks a lot!
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com