Re: [PATCH 2/2] Add documentation on fragments

["Antonin Godard" <antonin.godard@bootlin.com>]

On Thu Sep 25, 2025 at 1:07 PM CEST, Alexander Kanavin wrote:
> On Thu, 25 Sept 2025 at 09:29, Antonin Godard
> <antonin.godard@bootlin.com> wrote:
>>
>> - terms.rst: Provide the definitions of a Configuration Fragment and a
>>   Built-in Fragment.
>> - ref-manual: Add a quick reference guide on bitbake-config-build, and
>>   list the available fragments in OE-Core.
>>   Document the underlying variables related to fragments in the
>>   glossary.
>> - dev-manual: give instructions on how to create new custom fragments.
>
> Thanks for writing these up. It's mostly fine, but I do have a few comments.
>
>> +Using Configuration Fragments In Your Build
>> +*******************************************
>
> What follows is not about using fragments (e.g. enabling/disabling
> them), it's about making new fragments. I think the section should be
> titled accordingly.

Right, this turned out to be a document different from my initial draft and
forgot to update the title. Thanks for spotting this.

>> +The Yocto Project supports enabling and disable some configuration variables
>> +through :term:`Configuration Fragments <Configuration Fragment>`. This mechanism
>> +allows modifying the top-level configuration of a build from the command-line
>> +using the :oe_git:`bitbake-config-build </bitbake/tree/bin/bitbake-config-build>`
>> +tool. This section will describe how to create new fragments for your builds.
>
> I think here and elsewhere (where you need to explain what fragments
> are) it's good to introduce fragments like this:
> 'Fragments define top level build configuration features that can be
> independently enabled and disabled using standard tooling. Such
> features are made of one or several build configuration statements
> that are either contained in a fragment file, or are set indirectly
> using the built-in fragment mechanism. '.

Ok, I felt that I needed a good catch phrase on fragments to put in different
places - I'll use that one.

>> +There are two kinds of configuration fragments:
>> +
>> +-  :term:`Configuration Fragments <Configuration Fragment>` which a stored
>> +   in a file. These fragment include a summary and a description.
>
> Spelling issues. They are also in some other places, I won't nit-pick
> on those. Let's get the content correct first.
>
> 'Standard :term:`Configuration Fragments <Configuration Fragment>`
> which are defined in a file. These fragments include a summary and a
> description, followed by configuration statements.

Thanks for your feedback on this. I had a struggle whether we should name
fragments in file "standard" or "regular", or just "fragments". I think standard
is good.

>> +-  :term:`Built-in Fragments <Built-in Fragment>` which are used to set a
>> +   variable value.
>
> ... which can be used to assign a value to a single variable and do
> not require a separate definition file. They are especially useful
> when a list of possible values is very long (or infinite).
>
>> +Creating new :term:`Built-in Fragments <Built-in Fragment>` is not supported.
>
> I think it is, but that requires putting an additional addfragment
> statement somewhere, or some way to append to OE_FRAGMENTS_BUILTIN
> *before* the oe-core's addfragments is processed.

Yes exactly, I did have some struggle yesterday so in the meantime wrote this.
But I know it can be supported.

>> +You should instead create :term:`Configuration Fragment` files as documented
>> +below to customize a build to your need.
>
> I would try to describe adding built-in fragments as well.

I tried but failed to have something consistent yesterday.

Setting it from a distro configuration file, or any configuration file that
comes after the addfragments call in bitbake.conf does not work in my
experience. Moving the addfragments call after the distro file include is not
possible as the the distro/ builtin fragment sets the DISTRO (chicken and egg
problem :)).

There are multiple solutions:

- Telling users to define their custom builtin fragments in local.conf,
  auto.conf, site.conf, or any file that comes before the addfragments call.
  This is not desirable because this means we don't encourage to share
  configurations.

- Telling users to defines their builtin fragments in layer.conf. This works,
  but this means layers can "pollute" the existing builtin fragments by
  overriding them when added to bblayers.conf. Other built-in fragments that have
  different names (not machine/ or distro/) is fine I guess, because
  just declaring a fragment as available does not affect the build.

  Maybe this could be reasonable, but we enforce layer maintainers to _not_
  override core built-in fragments (yocto-check-layer test, maybe?)

- One idea I had was to introduce an `addbuiltinfragments FRAGMENTS_BUILTIN`
  directive. This way we handle standard and built-in fragments separately.
  This turns out to be difficult as well, as standard and built-in fragments
  are mixed in together in OE_FRAGMENTS.

I feel like our best option is option 2 here. What do you think? Other ideas
maybe?

>> +Creating A Configuration Fragment File
>> +======================================
>> +
>> +By default, all configuration fragments are located within the
>> +``conf/fragments`` directory of a :term:`layer`. This location is defined by the
>> +:term:`OE_FRAGMENTS_PREFIX` variable.
>
> ... which, in turn, is used as a parameter in a addfragments statement
> in meta/conf/bitbake.conf
>
>> +You can create one or more :term:`configuration fragment` files in your
>> +:term:`layer` in this directory. Let's take the following example, where
>> +``custom-fragment.conf`` is our custom fragment file::
>> +
>> +   meta-custom
>> +   ├── conf
>> +   │   ├── fragments
>> +   │   │   └── custom-fragment.conf
>> +   │   └── layer.conf
>> +   ...
>> +
>> +For our ``custom-fragment.conf`` file, the following variables **must** be set
>> +for our fragment to be considered a valid fragment by the :term:`OpenEmbedded
>> +Build System`:
>> +
>> +-  :term:`BB_CONF_FRAGMENT_SUMMARY`: a one-line summary of this fragment.
>> +
>> +-  :term:`BB_CONF_FRAGMENT_DESCRIPTION`: a description of this fragment.
>
> These are again defined through a parameter to addfragments, so it
> would be good to reference that.

My initial approach was to document this as if a user did not need to have
custom addfragments calls and all that was required was to place fragment files
in the good spot in their layer, and modify OE_FRAGMENTS_BUILTIN as needed
(turns the second use case is not that easy in the end).

For example, we don't make the location of machine configurations configurable
AFAIK - why would changing the location of fragments be needed for users of OE?

But I can add references to this command, I just don't think we should emphasize
too much on it.

>> +After creating these variables, our custom fragment should look like the
>> +following:
>> +
>> +.. code-block::
>> +   :caption: custom-fragment.conf
>> +
>> +   BB_CONF_FRAGMENT_SUMMARY = "My custom fragment summary"
>> +   BB_CONF_FRAGMENT_DESCRIPTION = "My custom fragment description, \
>> +   which can be multiple lines."
>
> "This fragment sets a limit of 4 bitbake threads and 4 parsing threads."
> "This fragment is useful to constrain resource consumption when the
> Yocto default \
>  is causing an overload of host machine's memory and cpu resources."

Thanks :) I could have made the extra effort here!

>> +***********************
>> +Configuration Fragments
>
> Using configuration fragments.
>
>
>> +***********************
>> +
>> +:term:`Configuration Fragments <Configuration Fragment>` are used to provide a
>> +snippet of global configuration that can be enabled or disabled for a build.
>> +This document provides a quick reference of the :oe_git:`bitbake-config-build
>> +</bitbake/tree/bin/bitbake-config-build>` tool and lists the
>> +:term:`Configuration Fragments <Configuration Fragment>` and :term:`Built-in
>> +Fragments <Built-in Fragment>` available in the :term:`OpenEmbedded Build
>> +System` core repositories.
>
> See above how the fragments could be introduced.
>
>> +When a fragment is enabled with :ref:`ref-bitbake-config-build-enable-fragment`,
>> +its name is automatically appended to the :term:`OE_FRAGMENTS` variable in
>> +:ref:`structure-build-conf-auto.conf`.
>
> It is also possible (but not recommended) to manually add to this
> variable from local.conf.

Will mention this

>> +   :term:`Built-in Fragment`
>> +      A built-in fragment is a specific kind of :term:`Configuration Fragment`
>> +      that affects the value of a single variable globally. Like
>> +      a normal :term:`Configuration Fragment`, Built-in Fragments can be enabled
>> +      or disabled using the :oe_git:`bitbake-config-build </bitbake/tree/bin/bitbake-config-build>`
>> +      command-line utility.
>> +
>> +      When declared, a built-in fragment follows the following naming
>> +      convention::
>> +
>> +         <fragment>:<variable name>
>> +
>> +      Where:
>> +
>> +      -  ``<fragment>`` is the name of the built-in fragment.
>> +      -  ``<variable name>`` is the name of the variable to be modified by this
>> +         fragment.
>> +
>> +      For example::
>> +
>> +         machine:MACHINE
>> +
>> +      Will setup the ``machine`` Built-in Fragment for modifying the value of
>> +      the :term:`MACHINE` variable.
>> +
>> +      Setting the :term:`MACHINE` variable through this fragment must follow
>> +      this syntax::
>> +
>> +         machine/qemux86-64
>> +
>> +      This sets the value of :term:`MACHINE` to ``qemux86-64``.
>> +
>> +      For more details on fragments, see:
>> +
>> +      -  The :doc:`/ref-manual/fragments` section of the Yocto Project Reference
>> +         Manual for a list of fragments the :term:`OpenEmbedded Build System`
>> +         supports, and a quick reference guide on how to manage fragments.
>> +
>> +      -  The :doc:`/dev-manual/using-fragments` section of the Yocto Project
>> +         Development Tasks Manual for details on how to create new fragments
>> +         in your build.
>
> Two more points need to be made:
>
> - built-in fragments can be used to assign a value to a single
> variable and do not require a separate definition file. They are
> especially useful when a list of possible values is very long (or
> infinite), as that avoids making large amounts of fragment files, all
> very similar to each other.
>
> - the definition of available built-in fragments is in meta/conf/bitbake.conf:
> OE_FRAGMENTS_BUILTIN ?= "machine:MACHINE distro:DISTRO"
>
> In turn, what variable contains the definition is specified in the
> addfragments directive in the same file:
> addfragments ${OE_FRAGMENTS_PREFIX} OE_FRAGMENTS
> OE_FRAGMENTS_METADATA_VARS OE_FRAGMENTS_BUILTIN

Will add that

>>     :term:`Classes`
>>        Files that provide for logic encapsulation and inheritance so that
>>        commonly used patterns can be defined once and then easily used in
>> @@ -154,6 +196,37 @@ universal, the list includes them just in case:
>>        only used when building for that target (e.g. the
>>        :file:`machine/beaglebone.conf` configuration file defines variables for
>>        the Texas Instruments ARM Cortex-A8 development board).
>> +      :term:`Configuration Fragments <Configuration Fragment>` such as
>> +      :ref:`ref-fragments-core-yocto-sstate-mirror-cdn` define snippets of
>> +      configuration that can be enabled from the command-line.
>> +
>> +   :term:`Configuration Fragment`
>> +      A configuration fragment is a :term:`Configuration File` that contains
>> +      variable assignments affecting the build at a global-level when the
>
> And directives, such as 'require ...'. I'd instead say 'configuration
> statements, such as variable assignments'.

Indeed, thanks.

>> +      fragment is enabled. By default, configuration fragments are located in
>> +      the :file:`conf/fragments/` directory of a :term:`Layer`.
>> +
>> +      A fragment :term:`configuration file` must contain a summary
>> +      (:term:`BB_CONF_FRAGMENT_SUMMARY`) and a description
>> +      (:term:`BB_CONF_FRAGMENT_DESCRIPTION`) explaining the purpose of the
>> +      fragment.
>
> In oe-core, the location of fragments and what variables are required
> in a fragment is specified via meta/conf/bitbake.conf:
>
> OE_FRAGMENTS_PREFIX ?= "conf/fragments"
> OE_FRAGMENTS_METADATA_VARS ?= "BB_CONF_FRAGMENT_SUMMARY
> BB_CONF_FRAGMENT_DESCRIPTION"
> addfragments ${OE_FRAGMENTS_PREFIX} OE_FRAGMENTS
> OE_FRAGMENTS_METADATA_VARS OE_FRAGMENTS_BUILTIN
>

I usually don't include code snippets from OE-Core as it is harder to maintain,
but I can reference the variable and location, this should be similar without
the maintenance burden.

> Thanks!
>
> Alex

Antonin

-- 
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com