[PATCH 0/2] Document how to provide configuration from layer.conf

[PATCH 0/2] Document how to provide configuration from layer.conf

[Antonin Godard]

This series documents how to provide configurations from layer.conf, be
it unconditionally or conditionally. I wanted to reference a file in
bitbake, so I added the :bitbake_git: new term for that.

This is a attempt at closing
https://bugzilla.yoctoproject.org/show_bug.cgi?id=12688.

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
---
Antonin Godard (2):
      conf.py: add a bitbake_git extlink
      dev-manual: document how to provide confs from layer.conf

 documentation/conf.py               |  1 +
 documentation/dev-manual/layers.rst | 78 ++++++++++++++++++++++++++++++++++++-
 2 files changed, 78 insertions(+), 1 deletion(-)
---
base-commit: 4c81e2856c268d7d0221ebb6b8156ad65b9e83ca
change-id: 20241030-layer-confs-54c26ece97d3

Best regards,
-- 
Antonin Godard <antonin.godard@bootlin.com>

[PATCH 1/2] conf.py: add a bitbake_git extlink

[Antonin Godard]

To make references to the bitbake repo, add an extlink for it and use it
in the docs with ":bitbake_git:`lib/bb/utils.py </tree/lib/bb/utils.py>`".

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
---
 documentation/conf.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/documentation/conf.py b/documentation/conf.py
index 35c5c14535ba67aae3554819b55f0fb52a9d0027..d45aab45241fdce0e7eb3938e7562168752d7c93 100644
--- a/documentation/conf.py
+++ b/documentation/conf.py
@@ -90,6 +90,7 @@ rst_prolog = """
 
 # external links and substitutions
 extlinks = {
+    'bitbake_git': ('https://git.openembedded.org/bitbake%s', None),
     'cve': ('https://nvd.nist.gov/vuln/detail/CVE-%s', 'CVE-%s'),
     'cve_mitre': ('https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-%s', 'CVE-%s'),
     'yocto_home': ('https://www.yoctoproject.org%s', None),

-- 
2.46.1

Re: [docs] [PATCH 1/2] conf.py: add a bitbake_git extlink

[Quentin Schulz]

Hi Antonin,

On 10/30/24 10:23 AM, Antonin Godard via lists.yoctoproject.org wrote:
> To make references to the bitbake repo, add an extlink for it and use it
> in the docs with ":bitbake_git:`lib/bb/utils.py </tree/lib/bb/utils.py>`".
> 
> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>

Reviewed-by: Quentin Schulz <quentin.schulz@cherry.de>

I believe we could also migrate a few oe_git to bitbake_git now across 
the docs?

git grep 'oe_git.*bitbake' should help you identify those.

Thanks!
Quentin

Re: [docs] [PATCH 1/2] conf.py: add a bitbake_git extlink

[Antonin Godard]

Hi Quentin,

On Wed Oct 30, 2024 at 11:21 AM CET, Quentin Schulz wrote:
> Hi Antonin,
>
> On 10/30/24 10:23 AM, Antonin Godard via lists.yoctoproject.org wrote:
>> To make references to the bitbake repo, add an extlink for it and use it
>> in the docs with ":bitbake_git:`lib/bb/utils.py </tree/lib/bb/utils.py>`".
>>
>> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
>
> Reviewed-by: Quentin Schulz <quentin.schulz@cherry.de>
>
> I believe we could also migrate a few oe_git to bitbake_git now across
> the docs?
>
> git grep 'oe_git.*bitbake' should help you identify those.

Surely, I think there is also a lot of `poky_git` usage while it should be
`oe_git`… I have a note to fix that in my todo. :)

> Thanks!
> Quentin

Thank you,
Antonin

--
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[PATCH 2/2] dev-manual: document how to provide confs from layer.conf

[Antonin Godard]

Add a section on providing global level configuration from the
layer.conf file. Since this file is parsed at an earlier stage in the
parsing process, it's not possible to combine bb.utils.contains and
{DISTRO,MACHINE}_FEATURES to conditionally some configurations.

This patch documents:

- First that this file can be used for providing such configuration.
- Then demonstrates how to conditionally provide them, using a technique
  that is currently used in meta-virtualization
  (https://git.yoctoproject.org/meta-virtualization/tree/conf/layer.conf#n50).

Fixes [YOCTO #12688].

Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
---
 documentation/dev-manual/layers.rst | 78 ++++++++++++++++++++++++++++++++++++-
 1 file changed, 77 insertions(+), 1 deletion(-)

diff --git a/documentation/dev-manual/layers.rst b/documentation/dev-manual/layers.rst
index 91889bd0ae391c7b6e62068778ae5c180c840762..94d959fde67e911435d99ce8dd85b3d591573c82 100644
--- a/documentation/dev-manual/layers.rst
+++ b/documentation/dev-manual/layers.rst
@@ -644,6 +644,83 @@ variable and append the layer's root name::
    order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
    might address this.
 
+Providing Global-level Configurations With Your Layer
+-----------------------------------------------------
+
+When creating a layer, you may need to define configurations that should take
+effect globally in your build environment when it is part of the
+:term:`BBLAYERS` variable. The ``layer.conf`` file is a :term:`configuration
+file` that affects the build system globally, so it is a candidate for this
+use-case.
+
+For example, if your layer provides an alternative to the linux kernel named
+``linux-custom``, you may want to define the :term:`PREFERRED_PROVIDER` for the
+``linux-custom`` recipe::
+
+  PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
+
+This can be defined in the ``layer.conf`` file. If your layer has a higher
+:term:`BBFILE_PRIORITY` value, it will take precedence over previous definitions
+of the same ``PREFERRED_PROVIDER_virtual/kernel`` assignments.
+
+Conditionally Provide Global-level Configurations With Your Layer
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In some cases, your layer may provide global configurations only if some
+features it provides are enabled. Since the ``layer.conf`` file is parsed at an
+earlier stage in the parsing process, the :term:`DISTRO_FEATURES` and
+:term:`MACHINE_FEATURES` variables are not yet available to ``layer.conf``, and
+declaring conditional assignments based on these variables is not possible. The
+following technique shows a way to leverage this limitation by using the
+:term:`USER_CLASSES` variable and a conditional ``include`` command.
+
+In the following steps, let's assume our layer is named ``meta-mylayer`` and
+that this layer defines a custom :ref:`distro feature <ref-features-distro>`
+named ``mylayer-kernel``. We will set the :term:`PREFERRED_PROVIDER` variable
+for the kernel only if our feature ``mylayer-kernel`` is part of the
+:term:`DISTRO_FEATURES`:
+
+#. Create an include file in the directory
+   ``meta-mylayer/conf/distro/include/``, for example a file named
+   ``mylayer-kernel-provider.inc`` that sets the kernel provider to
+   ``linux-custom``::
+
+     PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
+
+#. Provide a path to this include file in your ``layer.conf``::
+
+     META_MYLAYER_KERNEL_PROVIDER_PATH = "${LAYERDIR}/conf/distro/include/mylayer-kernel-provider.inc"
+
+#. Create a new class in ``meta-mylayer/classes-global/``, for example a class
+   ``meta-mylayer-cfg.bbclass``. It Conditionally includes the file
+   ``mylayer-kernel-provider.inc`` defined above, using the variable
+   ``META_MYLAYER_KERNEL_PROVIDER_PATH`` defined in ``layer.conf``::
+
+     include ${@bb.utils.contains('DISTRO_FEATURES', 'mylayer-kernel', '${META_MYLAYER_KERNEL_PROVIDER_PATH}', '', d)}
+
+   For details on the ``bb.utils.contains`` function, see its definition in
+   :bitbake_git:`lib/bb/utils.py </tree/lib/bb/utils.py>`.
+
+   .. note::
+
+      The ``include`` command is designed to not fail if the function
+      ``bb.utils.contains`` returns an empty string.
+
+#. Back to your ``layer.conf`` file, add the class ``meta-mylayer-cfg`` class to
+   the :term:`USER_CLASSES` variable::
+
+     USER_CLASSES:append = " meta-mylayer-cfg"
+
+   This will add the class ``meta-mylayer-cfg`` to the list of classes to
+   globally inherit. Since the ``include`` command is conditional in
+   ``meta-mylayer-cfg.bbclass``, even though inherited the class will have no
+   effect until the feature ``mylayer-kernel`` is enabled through
+   :term:`DISTRO_FEATURES`.
+
+This technique can also be used for :ref:`Machine features
+<ref-features-machine>` by following the same steps, the only difference being
+to place the include file in ``meta-mylayer/conf/machine/include/`` instead.
+
 Managing Layers
 ===============
 
@@ -916,4 +993,3 @@ above:
 .. note::
    Both the ``create-layers-setup`` and the ``setup-layers`` provided several additional options
    that customize their behavior - you are welcome to study them via ``--help`` command line parameter.
-

-- 
2.46.1

Re: [docs] [PATCH 2/2] dev-manual: document how to provide confs from layer.conf

[Quentin Schulz]

Hi Antonin,

On 10/30/24 10:23 AM, Antonin Godard via lists.yoctoproject.org wrote:
> Add a section on providing global level configuration from the
> layer.conf file. Since this file is parsed at an earlier stage in the
> parsing process, it's not possible to combine bb.utils.contains and
> {DISTRO,MACHINE}_FEATURES to conditionally some configurations.
> 
> This patch documents:
> 
> - First that this file can be used for providing such configuration.
> - Then demonstrates how to conditionally provide them, using a technique
>    that is currently used in meta-virtualization
>    (https://git.yoctoproject.org/meta-virtualization/tree/conf/layer.conf#n50).
> 
> Fixes [YOCTO #12688].
> 
> Signed-off-by: Antonin Godard <antonin.godard@bootlin.com>
> ---
>   documentation/dev-manual/layers.rst | 78 ++++++++++++++++++++++++++++++++++++-
>   1 file changed, 77 insertions(+), 1 deletion(-)
> 
> diff --git a/documentation/dev-manual/layers.rst b/documentation/dev-manual/layers.rst
> index 91889bd0ae391c7b6e62068778ae5c180c840762..94d959fde67e911435d99ce8dd85b3d591573c82 100644
> --- a/documentation/dev-manual/layers.rst
> +++ b/documentation/dev-manual/layers.rst
> @@ -644,6 +644,83 @@ variable and append the layer's root name::
>      order of ``.conf`` or ``.bbclass`` files. Future versions of BitBake
>      might address this.
>   
> +Providing Global-level Configurations With Your Layer
> +-----------------------------------------------------
> +
> +When creating a layer, you may need to define configurations that should take
> +effect globally in your build environment when it is part of the
> +:term:`BBLAYERS` variable. The ``layer.conf`` file is a :term:`configuration

"when it is part of the BBLAYERS variable" is a complicated way of 
saying "when it is used"/"when it is part of the build setup". Maybe we 
can find something more explicit?

I would also suggest adding "in the new layer" after ``layer.conf`` to 
explicit the location of layer.conf (maybe we should even add it's in 
conf/?). We have many different possible locations for configuration 
files so I feel like we should tell people where exactly to expect 
files. (e.g. local.conf, distro conf file, machine conf file, 
(multiconfig?), bblayers.conf, are all expected in different locations).

> +file` that affects the build system globally, so it is a candidate for this
> +use-case.
> +
> +For example, if your layer provides an alternative to the linux kernel named

s/provides an alternative to the Linux kernel/provides a Linux kernel 
recipe/

s/linux/Linux/

> +``linux-custom``, you may want to define the :term:`PREFERRED_PROVIDER` for the
> +``linux-custom`` recipe::
> +
> +  PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
> +
> +This can be defined in the ``layer.conf`` file. If your layer has a higher
> +:term:`BBFILE_PRIORITY` value, it will take precedence over previous definitions
> +of the same ``PREFERRED_PROVIDER_virtual/kernel`` assignments.
> +

I seem to recall this is the perfect example of what we do NOT want in a 
layer.conf though (might be bad recollection though :) ). I actually do 
not even know what'd be a valid reason to have variables directly 
defined in layer.conf (I see you're providing an example in the next 
section, but do we really need this section to provide an example of 
something the user shouldn't be doing?).

I would add a very big note/warning at the end of this section saying 
that this is almost always wrong as it is mostly expected that adding a 
layer should only minimally change the build output. And that for this 
very example, it should be in a machine or distro configuration file 
instead.

I am not sure that BBFILE_PRIORITY is applicable for this as what 
matters is the parsing order of all layer.conf files from entries in 
BBLAYERS. I believe it's just parsed in the order it appears in 
BBLAYERS? Last one has priority (because we use = and not ?=). If that 
is actually working, we should then expand 
https://docs.yoctoproject.org/ref-manual/variables.html#term-BBFILE_PRIORITY 
as well because it clearly only defines priority for recipes.

> +Conditionally Provide Global-level Configurations With Your Layer
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +In some cases, your layer may provide global configurations only if some
> +features it provides are enabled. Since the ``layer.conf`` file is parsed at an
> +earlier stage in the parsing process, the :term:`DISTRO_FEATURES` and
> +:term:`MACHINE_FEATURES` variables are not yet available to ``layer.conf``, and
> +declaring conditional assignments based on these variables is not possible. The
> +following technique shows a way to leverage this limitation by using the
> +:term:`USER_CLASSES` variable and a conditional ``include`` command.
> +
> +In the following steps, let's assume our layer is named ``meta-mylayer`` and
> +that this layer defines a custom :ref:`distro feature <ref-features-distro>`
> +named ``mylayer-kernel``. We will set the :term:`PREFERRED_PROVIDER` variable
> +for the kernel only if our feature ``mylayer-kernel`` is part of the
> +:term:`DISTRO_FEATURES`:
> +
> +#. Create an include file in the directory
> +   ``meta-mylayer/conf/distro/include/``, for example a file named
> +   ``mylayer-kernel-provider.inc`` that sets the kernel provider to
> +   ``linux-custom``::
> +
> +     PREFERRED_PROVIDER_virtual/kernel = "linux-custom"

Random thought but maybe we should have align this with three 
whitespaces starting from the first character in the list. Here it is 
only one and difficult to notice at first sight. Actually, it's two 
whitespaces, I think that proves my point :D

> +
> +#. Provide a path to this include file in your ``layer.conf``::
> +
> +     META_MYLAYER_KERNEL_PROVIDER_PATH = "${LAYERDIR}/conf/distro/include/mylayer-kernel-provider.inc"
> +
> +#. Create a new class in ``meta-mylayer/classes-global/``, for example a class
> +   ``meta-mylayer-cfg.bbclass``. It Conditionally includes the file

s/Conditionally/conditionally/

Maybe say "Make it conditionally include the file" instead, as the way 
it's phrased could mean that because it's named this way or/and in that 
special directory that it magically includes the file, instead of 
telling the user what to put in that file.

> +   ``mylayer-kernel-provider.inc`` defined above, using the variable
> +   ``META_MYLAYER_KERNEL_PROVIDER_PATH`` defined in ``layer.conf``::
> +
> +     include ${@bb.utils.contains('DISTRO_FEATURES', 'mylayer-kernel', '${META_MYLAYER_KERNEL_PROVIDER_PATH}', '', d)}
> +
> +   For details on the ``bb.utils.contains`` function, see its definition in
> +   :bitbake_git:`lib/bb/utils.py </tree/lib/bb/utils.py>`.
> +
> +   .. note::
> +
> +      The ``include`` command is designed to not fail if the function
> +      ``bb.utils.contains`` returns an empty string.

I believe `require` too and we very likely want to use require here to 
make sure the path doesn't have a typo :)

Also, I'm wondering if this isn't actually "breaking" the sstate cache 
as this META_MYLAYER_KERNEL_PROVIDER_PATH would be different on each 
machine so people wouldn't be able to reuse the sstate cache.

Building with the exact same setup - and a shared SSTATE_DIR - in two 
different directories on the same machine should answer that question :)

> +
> +#. Back to your ``layer.conf`` file, add the class ``meta-mylayer-cfg`` class to
> +   the :term:`USER_CLASSES` variable::
> +
> +     USER_CLASSES:append = " meta-mylayer-cfg"
> +
> +   This will add the class ``meta-mylayer-cfg`` to the list of classes to
> +   globally inherit. Since the ``include`` command is conditional in
> +   ``meta-mylayer-cfg.bbclass``, even though inherited the class will have no
> +   effect until the feature ``mylayer-kernel`` is enabled through
> +   :term:`DISTRO_FEATURES`.
> +
> +This technique can also be used for :ref:`Machine features
> +<ref-features-machine>` by following the same steps, the only difference being
> +to place the include file in ``meta-mylayer/conf/machine/include/`` instead.
> +

I don't think the path needs to be specific?

>   Managing Layers
>   ===============
>   
> @@ -916,4 +993,3 @@ above:
>   .. note::
>      Both the ``create-layers-setup`` and the ``setup-layers`` provided several additional options
>      that customize their behavior - you are welcome to study them via ``--help`` command line parameter.
> -

Not sure this line deletion makes sense in this patch?

Cheers,
Quentin

Re: [docs] [PATCH 2/2] dev-manual: document how to provide confs from layer.conf

[Antonin Godard]

Hi Quentin,

On Wed Oct 30, 2024 at 11:18 AM CET, Quentin Schulz wrote:
[...]
>> +Providing Global-level Configurations With Your Layer
>> +-----------------------------------------------------
>> +
>> +When creating a layer, you may need to define configurations that should take
>> +effect globally in your build environment when it is part of the
>> +:term:`BBLAYERS` variable. The ``layer.conf`` file is a :term:`configuration
>
> "when it is part of the BBLAYERS variable" is a complicated way of 
> saying "when it is used"/"when it is part of the build setup". Maybe we 
> can find something more explicit?
>
> I would also suggest adding "in the new layer" after ``layer.conf`` to 
> explicit the location of layer.conf (maybe we should even add it's in 
> conf/?). We have many different possible locations for configuration 
> files so I feel like we should tell people where exactly to expect 
> files. (e.g. local.conf, distro conf file, machine conf file, 
> (multiconfig?), bblayers.conf, are all expected in different locations).
>
>> +file` that affects the build system globally, so it is a candidate for this
>> +use-case.
>> +
>> +For example, if your layer provides an alternative to the linux kernel named
>
> s/provides an alternative to the Linux kernel/provides a Linux kernel 
> recipe/
>
> s/linux/Linux/

+1, thanks.

>> +``linux-custom``, you may want to define the :term:`PREFERRED_PROVIDER` for the
>> +``linux-custom`` recipe::
>> +
>> +  PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
>> +
>> +This can be defined in the ``layer.conf`` file. If your layer has a higher
>> +:term:`BBFILE_PRIORITY` value, it will take precedence over previous definitions
>> +of the same ``PREFERRED_PROVIDER_virtual/kernel`` assignments.
>> +
>
> I seem to recall this is the perfect example of what we do NOT want in a 
> layer.conf though (might be bad recollection though :) ). I actually do 
> not even know what'd be a valid reason to have variables directly 
> defined in layer.conf (I see you're providing an example in the next 
> section, but do we really need this section to provide an example of 
> something the user shouldn't be doing?).
>
> I would add a very big note/warning at the end of this section saying 
> that this is almost always wrong as it is mostly expected that adding a 
> layer should only minimally change the build output. And that for this 
> very example, it should be in a machine or distro configuration file 
> instead.

You're right, it's not a good practice. Initially, I wanted to document only the
conditional case below, but it felt strange introducing it without a note on the
role of layers.conf & setting things from there. What about the following
changes instead:

* Remove the first section that shows how to set configurations unconditionally.
* Instead, introduce the next section by saying something like "Since it's not
  good practice to unconditionally set global configurations from the layer.conf
  file, we can use the following technique...".
* Simply rename the section "Providing Global-level Configurations With Your
  Layer" instead of "Conditionally Provide Global-level Configurations With Your
  Layer". This way we only show the "good" way to make changes from layer.conf.

> I am not sure that BBFILE_PRIORITY is applicable for this as what 
> matters is the parsing order of all layer.conf files from entries in 
> BBLAYERS. I believe it's just parsed in the order it appears in 
> BBLAYERS? Last one has priority (because we use = and not ?=). If that 
> is actually working, we should then expand 
> https://docs.yoctoproject.org/ref-manual/variables.html#term-BBFILE_PRIORITY 
> as well because it clearly only defines priority for recipes.

My bad you're right, it's only the order in BBLAYERS that matters, I'll remove
this sentence.

>> +Conditionally Provide Global-level Configurations With Your Layer
>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> +
>> +In some cases, your layer may provide global configurations only if some
>> +features it provides are enabled. Since the ``layer.conf`` file is parsed at an
>> +earlier stage in the parsing process, the :term:`DISTRO_FEATURES` and
>> +:term:`MACHINE_FEATURES` variables are not yet available to ``layer.conf``, and
>> +declaring conditional assignments based on these variables is not possible. The
>> +following technique shows a way to leverage this limitation by using the
>> +:term:`USER_CLASSES` variable and a conditional ``include`` command.
>> +
>> +In the following steps, let's assume our layer is named ``meta-mylayer`` and
>> +that this layer defines a custom :ref:`distro feature <ref-features-distro>`
>> +named ``mylayer-kernel``. We will set the :term:`PREFERRED_PROVIDER` variable
>> +for the kernel only if our feature ``mylayer-kernel`` is part of the
>> +:term:`DISTRO_FEATURES`:
>> +
>> +#. Create an include file in the directory
>> +   ``meta-mylayer/conf/distro/include/``, for example a file named
>> +   ``mylayer-kernel-provider.inc`` that sets the kernel provider to
>> +   ``linux-custom``::
>> +
>> +     PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
>
> Random thought but maybe we should have align this with three 
> whitespaces starting from the first character in the list. Here it is 
> only one and difficult to notice at first sight. Actually, it's two 
> whitespaces, I think that proves my point :D

Yes, it should be 3 actually, missed that!

>> +
>> +#. Provide a path to this include file in your ``layer.conf``::
>> +
>> +     META_MYLAYER_KERNEL_PROVIDER_PATH = "${LAYERDIR}/conf/distro/include/mylayer-kernel-provider.inc"
>> +
>> +#. Create a new class in ``meta-mylayer/classes-global/``, for example a class
>> +   ``meta-mylayer-cfg.bbclass``. It Conditionally includes the file
>
> s/Conditionally/conditionally/
>
> Maybe say "Make it conditionally include the file" instead, as the way 
> it's phrased could mean that because it's named this way or/and in that 
> special directory that it magically includes the file, instead of 
> telling the user what to put in that file.

+1, better that way :)

>> +   ``mylayer-kernel-provider.inc`` defined above, using the variable
>> +   ``META_MYLAYER_KERNEL_PROVIDER_PATH`` defined in ``layer.conf``::
>> +
>> +     include ${@bb.utils.contains('DISTRO_FEATURES', 'mylayer-kernel', '${META_MYLAYER_KERNEL_PROVIDER_PATH}', '', d)}
>> +
>> +   For details on the ``bb.utils.contains`` function, see its definition in
>> +   :bitbake_git:`lib/bb/utils.py </tree/lib/bb/utils.py>`.
>> +
>> +   .. note::
>> +
>> +      The ``include`` command is designed to not fail if the function
>> +      ``bb.utils.contains`` returns an empty string.
>
> I believe `require` too and we very likely want to use require here to 
> make sure the path doesn't have a typo :)

Correct, I thought `require` would've failed if contains returned an empty
string (and that it was for that reason they used include in meta-virt) but
apparently not, the parser is smart enough. :) Will change, thanks

> Also, I'm wondering if this isn't actually "breaking" the sstate cache 
> as this META_MYLAYER_KERNEL_PROVIDER_PATH would be different on each 
> machine so people wouldn't be able to reuse the sstate cache.
>
> Building with the exact same setup - and a shared SSTATE_DIR - in two 
> different directories on the same machine should answer that question :)

I've tested it and no, it doesn't affect the sstate-cache.
I've done the following:

1) Build core-image-minimal with virtual/kernel set to linux-yocto-rt (set with
   a custom layer + technique detailed above, through a distro feature).
2) Same directory, rm -r tmp/, rebuild -> setscene tasks are executed as
   expected, so no rebuild, just the rootfs being created.
3) Different directory (to make LAYERDIR change for my custom layer), rm -r
   tmp/, rebuild -> same result as 2).

>> +
>> +#. Back to your ``layer.conf`` file, add the class ``meta-mylayer-cfg`` class to
>> +   the :term:`USER_CLASSES` variable::
>> +
>> +     USER_CLASSES:append = " meta-mylayer-cfg"
>> +
>> +   This will add the class ``meta-mylayer-cfg`` to the list of classes to
>> +   globally inherit. Since the ``include`` command is conditional in
>> +   ``meta-mylayer-cfg.bbclass``, even though inherited the class will have no
>> +   effect until the feature ``mylayer-kernel`` is enabled through
>> +   :term:`DISTRO_FEATURES`.
>> +
>> +This technique can also be used for :ref:`Machine features
>> +<ref-features-machine>` by following the same steps, the only difference being
>> +to place the include file in ``meta-mylayer/conf/machine/include/`` instead.
>> +
>
> I don't think the path needs to be specific?

Not really, but I'd rather advise here to place it in a conventionally used
directories (since we generally use this directory for machine include files)
rather than giving a random example. What do you think?

>>   Managing Layers
>>   ===============
>>   
>> @@ -916,4 +993,3 @@ above:
>>   .. note::
>>      Both the ``create-layers-setup`` and the ``setup-layers`` provided several additional options
>>      that customize their behavior - you are welcome to study them via ``--help`` command line parameter.
>> -
>
> Not sure this line deletion makes sense in this patch?

Oops, will remove.

> Cheers,
> Quentin

Cheers,
Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH 2/2] dev-manual: document how to provide confs from layer.conf

[Quentin Schulz]

Hi Antonin,

On 10/30/24 3:32 PM, Antonin Godard wrote:
[...]
>>> +``linux-custom``, you may want to define the :term:`PREFERRED_PROVIDER` for the
>>> +``linux-custom`` recipe::
>>> +
>>> +  PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
>>> +
>>> +This can be defined in the ``layer.conf`` file. If your layer has a higher
>>> +:term:`BBFILE_PRIORITY` value, it will take precedence over previous definitions
>>> +of the same ``PREFERRED_PROVIDER_virtual/kernel`` assignments.
>>> +
>>
>> I seem to recall this is the perfect example of what we do NOT want in a
>> layer.conf though (might be bad recollection though :) ). I actually do
>> not even know what'd be a valid reason to have variables directly
>> defined in layer.conf (I see you're providing an example in the next
>> section, but do we really need this section to provide an example of
>> something the user shouldn't be doing?).
>>
>> I would add a very big note/warning at the end of this section saying
>> that this is almost always wrong as it is mostly expected that adding a
>> layer should only minimally change the build output. And that for this
>> very example, it should be in a machine or distro configuration file
>> instead.
> 
> You're right, it's not a good practice. Initially, I wanted to document only the
> conditional case below, but it felt strange introducing it without a note on the
> role of layers.conf & setting things from there. What about the following
> changes instead:
> 
> * Remove the first section that shows how to set configurations unconditionally.
> * Instead, introduce the next section by saying something like "Since it's not
>    good practice to unconditionally set global configurations from the layer.conf
>    file, we can use the following technique...".
> * Simply rename the section "Providing Global-level Configurations With Your
>    Layer" instead of "Conditionally Provide Global-level Configurations With Your
>    Layer". This way we only show the "good" way to make changes from layer.conf.
> 

Not sure to understand if we would have one or two sections there.

I think it's a good idea to mention what the user shouldn't do and why. 
Starting the section with something like "please do not use any of the 
examples in this section as they are bad practice. The reasons will be 
explained at the end of the section. We still add this section as it's a 
more accessible introduction to the mechanism that will be used in the 
good practice explained in the next section."

>> I am not sure that BBFILE_PRIORITY is applicable for this as what
>> matters is the parsing order of all layer.conf files from entries in
>> BBLAYERS. I believe it's just parsed in the order it appears in
>> BBLAYERS? Last one has priority (because we use = and not ?=). If that
>> is actually working, we should then expand
>> https://docs.yoctoproject.org/ref-manual/variables.html#term-BBFILE_PRIORITY
>> as well because it clearly only defines priority for recipes.
> 
> My bad you're right, it's only the order in BBLAYERS that matters, I'll remove
> this sentence.
> 

Not necessarily! It's a good thing to mention as people may wonder why 
it works on one build machine and not another, and people then figure 
out that they including the layers in a different order.

>>> +Conditionally Provide Global-level Configurations With Your Layer
>>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>> +
>>> +In some cases, your layer may provide global configurations only if some
>>> +features it provides are enabled. Since the ``layer.conf`` file is parsed at an
>>> +earlier stage in the parsing process, the :term:`DISTRO_FEATURES` and
>>> +:term:`MACHINE_FEATURES` variables are not yet available to ``layer.conf``, and
>>> +declaring conditional assignments based on these variables is not possible. The
>>> +following technique shows a way to leverage this limitation by using the
>>> +:term:`USER_CLASSES` variable and a conditional ``include`` command.
>>> +
>>> +In the following steps, let's assume our layer is named ``meta-mylayer`` and
>>> +that this layer defines a custom :ref:`distro feature <ref-features-distro>`
>>> +named ``mylayer-kernel``. We will set the :term:`PREFERRED_PROVIDER` variable
>>> +for the kernel only if our feature ``mylayer-kernel`` is part of the
>>> +:term:`DISTRO_FEATURES`:
>>> +
>>> +#. Create an include file in the directory
>>> +   ``meta-mylayer/conf/distro/include/``, for example a file named
>>> +   ``mylayer-kernel-provider.inc`` that sets the kernel provider to
>>> +   ``linux-custom``::
>>> +
>>> +     PREFERRED_PROVIDER_virtual/kernel = "linux-custom"
>>
>> Random thought but maybe we should have align this with three
>> whitespaces starting from the first character in the list. Here it is
>> only one and difficult to notice at first sight. Actually, it's two
>> whitespaces, I think that proves my point :D
> 
> Yes, it should be 3 actually, missed that!
> 

I think what makes it difficult to read is that the last line before the 
codeblock is starting with two tick quotes. Let's see if three 
whitespaces help, but in any case being consistent would be a good 
thing. And it seems we aren't completely consistent, c.f.
$ mysql -u root -p
$ cd /var/www/toaster/poky/
$ source oe-init-build-env
/etc/apache2/conf-available/toaster.conf
/etc/httpd/conf.d/toaster.conf
/etc/apache2/conf.d/toaster.conf
Alias /static /var/www/toaster/static_files
...
in documentation/toaster-manual/setup-and-use.rst

[...]
>>> +
>>> +#. Back to your ``layer.conf`` file, add the class ``meta-mylayer-cfg`` class to
>>> +   the :term:`USER_CLASSES` variable::
>>> +
>>> +     USER_CLASSES:append = " meta-mylayer-cfg"
>>> +
>>> +   This will add the class ``meta-mylayer-cfg`` to the list of classes to
>>> +   globally inherit. Since the ``include`` command is conditional in
>>> +   ``meta-mylayer-cfg.bbclass``, even though inherited the class will have no
>>> +   effect until the feature ``mylayer-kernel`` is enabled through
>>> +   :term:`DISTRO_FEATURES`.
>>> +
>>> +This technique can also be used for :ref:`Machine features
>>> +<ref-features-machine>` by following the same steps, the only difference being
>>> +to place the include file in ``meta-mylayer/conf/machine/include/`` instead.
>>> +
>>
>> I don't think the path needs to be specific?
> 
> Not really, but I'd rather advise here to place it in a conventionally used
> directories (since we generally use this directory for machine include files)
> rather than giving a random example. What do you think?
> 

I would remove the mention to "the only difference being to place the 
include in <> instead" and instead add the following:

Though not mandatory, it is recommended to put include files for 
DISTRO_FEATURES in your layer's conf/distro/include and the ones for 
MACHINE_FEATURES in your layer's conf/machine/include.

Which provides the necessary information I believe? "You can do whatever 
you want, but it's best practice to do <that>".

Cheers,
Quentin