[PATCH 1/5] ref-manual: correct the location of default configuration template

[PATCH 1/5] ref-manual: correct the location of default configuration template

[Alexander Kanavin]

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 documentation/ref-manual/structure.rst | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/documentation/ref-manual/structure.rst b/documentation/ref-manual/structure.rst
index bdcffc194..6c6f52e79 100644
--- a/documentation/ref-manual/structure.rst
+++ b/documentation/ref-manual/structure.rst
@@ -189,7 +189,7 @@ Directory named ``mybuilds/`` that is outside of the :term:`Source Directory`::
    $ source oe-init-build-env ~/mybuilds
 
 The OpenEmbedded build system uses the template configuration files, which
-are found by default in the ``meta-poky/conf/`` directory in the Source
+are found by default in the ``meta-poky/conf/templates/default`` directory in the Source
 Directory. See the
 ":ref:`dev-manual/common-tasks:creating a custom template configuration directory`"
 section in the Yocto Project Development Tasks Manual for more
@@ -261,15 +261,15 @@ OpenEmbedded build system creates it from ``local.conf.sample`` when you
 :ref:`structure-core-script`.
 
 The source ``local.conf.sample`` file used depends on the
-:term:`TEMPLATECONF` script variable, which defaults to ``meta-poky/conf/``
+:term:`TEMPLATECONF` script variable, which defaults to ``meta-poky/conf/templates/default``
 when you are building from the Yocto Project development environment,
-and to ``meta/conf/`` when you are building from the OpenEmbedded-Core
+and to ``meta/conf/templates/default`` when you are building from the OpenEmbedded-Core
 environment. Because the script variable points to the source of the
 ``local.conf.sample`` file, this implies that you can configure your
 build environment from any layer by setting the variable in the
 top-level build environment setup script as follows::
 
-   TEMPLATECONF=your_layer/conf
+   TEMPLATECONF=your_layer/conf/templates/your_template_name
 
 Once the build process gets the sample
 file, it uses ``sed`` to substitute final
@@ -281,7 +281,7 @@ file, it uses ``sed`` to substitute final
    You can see how the :term:`TEMPLATECONF` variable is used by looking at the
    ``scripts/oe-setup-builddir`` script in the :term:`Source Directory`.
    You can find the Yocto Project version of the ``local.conf.sample`` file in
-   the ``meta-poky/conf`` directory.
+   the ``meta-poky/conf/templates/default`` directory.
 
 .. _structure-build-conf-bblayers.conf:
 
@@ -301,14 +301,14 @@ you ``source`` the top-level build environment setup script (i.e.
 
 As with the ``local.conf`` file, the source ``bblayers.conf.sample``
 file used depends on the :term:`TEMPLATECONF` script variable, which
-defaults to ``meta-poky/conf/`` when you are building from the Yocto
-Project development environment, and to ``meta/conf/`` when you are
+defaults to ``meta-poky/conf/templates/default`` when you are building from the Yocto
+Project development environment, and to ``meta/conf/templates/default`` when you are
 building from the OpenEmbedded-Core environment. Because the script
 variable points to the source of the ``bblayers.conf.sample`` file, this
 implies that you can base your build from any layer by setting the
 variable in the top-level build environment setup script as follows::
 
-   TEMPLATECONF=your_layer/conf
+   TEMPLATECONF=your_layer/conf/templates/your_template_name
 
 Once the build process gets the sample file, it uses ``sed`` to substitute final
 ``${``\ :term:`OEROOT`\ ``}`` values for all ``##OEROOT##`` values.
@@ -317,7 +317,7 @@ Once the build process gets the sample file, it uses ``sed`` to substitute final
 
    You can see how the :term:`TEMPLATECONF` variable is defined by the ``scripts/oe-setup-builddir``
    script in the :term:`Source Directory`. You can find the Yocto Project
-   version of the ``bblayers.conf.sample`` file in the ``meta-poky/conf/``
+   version of the ``bblayers.conf.sample`` file in the ``meta-poky/conf/templates/default``
    directory.
 
 .. _structure-build-conf-sanity_info:
-- 
2.30.2

[PATCH 2/5] common-tasks.rst: drop the output of 'bitbake-layers --help'

[Alexander Kanavin]

This is really not needed as the sub-commands are described in
greater detail just below, and is prone to become outdated.

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 documentation/dev-manual/common-tasks.rst | 33 -----------------------
 1 file changed, 33 deletions(-)

diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
index bdc528b72..c2afec946 100644
--- a/documentation/dev-manual/common-tasks.rst
+++ b/documentation/dev-manual/common-tasks.rst
@@ -653,39 +653,6 @@ For help on the BitBake layer management tool, use the following
 command::
 
    $ bitbake-layers --help
-   NOTE: Starting bitbake server...
-   usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ...
-
-   BitBake layers utility
-
-   optional arguments:
-     -d, --debug           Enable debug output
-     -q, --quiet           Print only errors
-     -F, --force           Force add without recipe parse verification
-     --color COLOR         Colorize output (where COLOR is auto, always, never)
-     -h, --help            show this help message and exit
-
-   subcommands:
-     <subcommand>
-       layerindex-fetch    Fetches a layer from a layer index along with its
-                           dependent layers, and adds them to conf/bblayers.conf.
-       layerindex-show-depends
-                           Find layer dependencies from layer index.
-       add-layer           Add one or more layers to bblayers.conf.
-       remove-layer        Remove one or more layers from bblayers.conf.
-       flatten             flatten layer configuration into a separate output
-                           directory.
-       show-layers         show current configured layers.
-       show-overlayed      list overlayed recipes (where the same recipe exists
-                           in another layer)
-       show-recipes        list available recipes, showing the layer they are
-                           provided by
-       show-appends        list bbappend files and recipe files they apply to
-       show-cross-depends  Show dependencies between recipes that cross layer
-                           boundaries.
-       create-layer        Create a basic layer
-
-   Use bitbake-layers <subcommand> --help to get help on a specific command
 
 The following list describes the available commands:
 
-- 
2.30.2

[PATCH 3/5] common-tasks.rst: describe the newly added layer setup and template config commands

[Alexander Kanavin]

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 documentation/dev-manual/common-tasks.rst | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
index c2afec946..4d19bd921 100644
--- a/documentation/dev-manual/common-tasks.rst
+++ b/documentation/dev-manual/common-tasks.rst
@@ -728,6 +728,17 @@ The following list describes the available commands:
 
 -  ``create-layer``: Creates a basic layer.
 
+-  ``save-build-conf``: Saves the currently active build configuration
+   (conf/local.conf, conf/bblayers.conf) as a template into a layer.
+   This template can later be used for setting up builds via TEMPLATECONF.
+   For information about saving and usinc configuration templates, see
+   ":ref:`dev-manual/common-tasks:creating a custom template configuration directory`".
+
+-  ``create-layers-setup``: Writes out a configuration file and/or a script that
+   can replicate the directory structure and revisions of the layers in a current build.
+   For more information, see ":ref:`dev-manual/common-tasks:saving and restoring the layers setup`".
+
+
 Creating a General Layer Using the ``bitbake-layers`` Script
 ------------------------------------------------------------
 
-- 
2.30.2

[PATCH 4/5] common-tasks.rst: describe the layer setup tooling

[Alexander Kanavin]

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 documentation/dev-manual/common-tasks.rst | 54 +++++++++++++++++++++++
 1 file changed, 54 insertions(+)

diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
index 4d19bd921..6c2df9aa6 100644
--- a/documentation/dev-manual/common-tasks.rst
+++ b/documentation/dev-manual/common-tasks.rst
@@ -858,6 +858,60 @@ enables the build system to locate the layer during the build.
    During a build, the OpenEmbedded build system looks in the layers
    from the top of the list down to the bottom in that order.
 
+Saving and restoring the layers setup
+-------------------------------------
+
+Once you have a working build with the correct set of layers, it is beneficial
+to capture the layer setup - what they are, which repositories they come from
+and which SCM revisions they're at - into a configuration file, so that this
+setup can be easily replicated later, perhaps on a different machine. Here's
+how to do this:
+
+   $ bitbake-layers create-layers-setup /srv/work/alex/meta-alex/
+   NOTE: Starting bitbake server...
+   NOTE: Created /srv/work/alex/meta-alex/setup-layers.json
+   NOTE: Created /srv/work/alex/meta-alex/setup-layers
+
+The tool needs a single argument which tells where to place the output, consisting
+of a json formatted layer configuration, and a 'setup-layers' script that can use that configuration
+to restore the layers in a different location, or on a different host machine. The argument
+can point to a custom layer (which is then deemed a 'bootstrap' layer that needs to be
+checked out first), or into a completely independent location.
+
+The replication of the layers is performed by running the setup-layer script provided
+above:
+
+1. Clone the bootstrap layer or some other repository to obtain
+the json config and the setup script that can use it.
+
+2. Run the script directly with no options:
+(note: this will work to update an existing checkout as well)
+
+   alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers
+   Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override.
+
+   Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master
+   Running 'git init -q /srv/work/alex/my-build/meta-intel'
+   Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel
+   Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel
+   Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel
+
+   Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers
+   Running 'git init -q /srv/work/alex/my-build/poky'
+   Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky
+   Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky
+   Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky
+   Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky
+   Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky
+
+.. note::
+   The script is self-suffucient and requires only python3
+   and git on the host machne where it will run.
+
+.. note::
+   Both the 'create-layers-setup' and the 'setup-layers' provided several additional options
+   that customize their behavior - you are welcome to study them via '--help' command line parameter.
+
 Customizing Images
 ==================
 
-- 
2.30.2

[PATCH 5/5] common-tasks.rst: rewrite the section about configuration templates

[Alexander Kanavin]

This now includes a description about how to create a template
with the newly added tooling, and drops the description
of .templateconf as a way to point to a custom template
(which is not correct: .templateconf is used only to point to a default
template in poky or core when TEMPLATECONF is not specified).

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
---
 documentation/dev-manual/common-tasks.rst | 92 +++++++++--------------
 1 file changed, 36 insertions(+), 56 deletions(-)

diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
index 6c2df9aa6..eab5648fd 100644
--- a/documentation/dev-manual/common-tasks.rst
+++ b/documentation/dev-manual/common-tasks.rst
@@ -6463,71 +6463,51 @@ Creating a Custom Template Configuration Directory
 ==================================================
 
 If you are producing your own customized version of the build system for
-use by other users, you might want to customize the message shown by the
-setup script or you might want to change the template configuration
-files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a
-new build directory.
+use by other users, you might want to provide a custom build configuration
+that includes all the necessary settings and layers (i.e. ``local.conf`` and
+``bblayers.conf`` that are created in a new build directory) and a custom
+message that is shown when setting up the build. This can be done by
+creating one or more template configuration directories in your
+custom distribution layer.
 
-The OpenEmbedded build system uses the environment variable
-:term:`TEMPLATECONF` to locate the directory from which it gathers
-configuration information that ultimately ends up in the
-:term:`Build Directory` ``conf`` directory.
-By default, :term:`TEMPLATECONF` is set as follows in the ``poky``
-repository::
+This can be done by using ``bitbake-layers save-build-conf``:
 
-   TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
-
-This is the
-directory used by the build system to find templates from which to build
-some key configuration files. If you look at this directory, you will
-see the ``bblayers.conf.sample``, ``local.conf.sample``, and
-``conf-notes.txt`` files. The build system uses these files to form the
-respective ``bblayers.conf`` file, ``local.conf`` file, and display the
-list of BitBake targets when running the setup script.
-
-To override these default configuration files with configurations you
-want used within every new Build Directory, simply set the
-:term:`TEMPLATECONF` variable to your directory. The :term:`TEMPLATECONF`
-variable is set in the ``.templateconf`` file, which is in the top-level
-:term:`Source Directory` folder
-(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your
-directory.
+   $ bitbake-layers save-build-conf ../../meta-alex/ test-1
+   NOTE: Starting bitbake server...
+   NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1
+   Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt
+   You can try out the configuration with
+   TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
 
-Best practices dictate that you should keep your template configuration
-directory in your custom distribution layer. For example, suppose you
-have a layer named ``meta-mylayer`` located in your home directory and
-you want your template configuration directory named ``myconf``.
-Changing the ``.templateconf`` as follows causes the OpenEmbedded build
-system to look in your directory and base its configuration files on the
-``*.sample`` configuration files it finds. The final configuration files
-(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in
-your Build Directory, but they are based on your ``*.sample`` files.
-::
+The above command takes the config files from the currently active build directory under ``conf\``,
+replaces site-specific paths in bblayers.conf with ##OECORE##-relative paths, and copies
+the config files into a specified layer under a specified template name.
 
-   TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf}
+To use those saved templates as a starting point for a build, users should point
+to one of them with :term:`TEMPLATECONF` environment variable:
 
-Aside from the ``*.sample`` configuration files, the ``conf-notes.txt``
-also resides in the default ``meta-poky/conf`` directory. The script
-that sets up the build environment (i.e.
-:ref:`structure-core-script`) uses this file to
-display BitBake targets as part of the script output. Customizing this
-``conf-notes.txt`` file is a good way to make sure your list of custom
-targets appears as part of the script's output.
+   TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
 
-Here is the default list of targets displayed as a result of running
-either of the setup scripts::
+The OpenEmbedded build system uses the environment variable
+:term:`TEMPLATECONF` to locate the directory from which it gathers
+configuration information that ultimately ends up in the
+:term:`Build Directory` ``conf`` directory.
 
-   You can now run 'bitbake <target>'
+If :term:`TEMPLATECONF` is not set, the default value is obtained
+from ```.templateconf`` file that is read from the same directory as
+`oe-init-build-env` script. For ``poky`` reference distribution this
+would be::
 
-   Common targets are:
-       core-image-minimal
-       core-image-sato
-       meta-toolchain
-       meta-ide-support
+   TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf/templates/default}
 
-Changing the listed common targets is as easy as editing your version of
-``conf-notes.txt`` in your custom template configuration directory and
-making sure you have :term:`TEMPLATECONF` set to your directory.
+If you look at a configuration template directory, you will
+see the ``bblayers.conf.sample``, ``local.conf.sample``, and
+``conf-notes.txt`` files. The build system uses these files to form the
+respective ``bblayers.conf`` file, ``local.conf`` file, and show
+users a note about the build they're setting up
+when running the ``oe-init-build-env`` setup script. These can be
+edited further if needed to improve or change the build configurations
+available to the users.
 
 Conserving Disk Space
 =====================
-- 
2.30.2

RE: [docs] [PATCH 5/5] common-tasks.rst: rewrite the section about configuration templates

[Peter Kjellerstedt]

> -----Original Message-----
> From: docs@lists.yoctoproject.org <docs@lists.yoctoproject.org> On Behalf
> Of Alexander Kanavin
> Sent: den 11 september 2022 18:26
> To: docs@lists.yoctoproject.org
> Cc: Alexander Kanavin <alex@linutronix.de>
> Subject: [docs] [PATCH 5/5] common-tasks.rst: rewrite the section about
> configuration templates
> 
> This now includes a description about how to create a template
> with the newly added tooling, and drops the description
> of .templateconf as a way to point to a custom template
> (which is not correct: .templateconf is used only to point to a default
> template in poky or core when TEMPLATECONF is not specified).

I do not agree with this sentiment. If you create and distribute a setup 
similar to Poky (which we do), why would we not change .templateconf to 
point at our templates? This has been the documented method and the one 
at least we use for our distribution. Where have you gotten the notion 
that this is only to be used by OE-Core & Poky? It does not make sense.

//Peter

> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>  documentation/dev-manual/common-tasks.rst | 92 +++++++++--------------
>  1 file changed, 36 insertions(+), 56 deletions(-)
> 
> diff --git a/documentation/dev-manual/common-tasks.rst
> b/documentation/dev-manual/common-tasks.rst
> index 6c2df9aa6..eab5648fd 100644
> --- a/documentation/dev-manual/common-tasks.rst
> +++ b/documentation/dev-manual/common-tasks.rst
> @@ -6463,71 +6463,51 @@ Creating a Custom Template Configuration Directory
>  ==================================================
> 
>  If you are producing your own customized version of the build system for
> -use by other users, you might want to customize the message shown by the
> -setup script or you might want to change the template configuration
> -files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a
> -new build directory.
> +use by other users, you might want to provide a custom build
> configuration
> +that includes all the necessary settings and layers (i.e. ``local.conf``
> and
> +``bblayers.conf`` that are created in a new build directory) and a custom
> +message that is shown when setting up the build. This can be done by
> +creating one or more template configuration directories in your
> +custom distribution layer.
> 
> -The OpenEmbedded build system uses the environment variable
> -:term:`TEMPLATECONF` to locate the directory from which it gathers
> -configuration information that ultimately ends up in the
> -:term:`Build Directory` ``conf`` directory.
> -By default, :term:`TEMPLATECONF` is set as follows in the ``poky``
> -repository::
> +This can be done by using ``bitbake-layers save-build-conf``:
> 
> -   TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
> -
> -This is the
> -directory used by the build system to find templates from which to build
> -some key configuration files. If you look at this directory, you will
> -see the ``bblayers.conf.sample``, ``local.conf.sample``, and
> -``conf-notes.txt`` files. The build system uses these files to form the
> -respective ``bblayers.conf`` file, ``local.conf`` file, and display the
> -list of BitBake targets when running the setup script.
> -
> -To override these default configuration files with configurations you
> -want used within every new Build Directory, simply set the
> -:term:`TEMPLATECONF` variable to your directory. The :term:`TEMPLATECONF`
> -variable is set in the ``.templateconf`` file, which is in the top-level
> -:term:`Source Directory` folder
> -(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your
> -directory.
> +   $ bitbake-layers save-build-conf ../../meta-alex/ test-1
> +   NOTE: Starting bitbake server...
> +   NOTE: Configuration template placed into /srv/work/alex/meta-
> alex/conf/templates/test-1
> +   Please review the files in there, and particularly provide a
> configuration description in /srv/work/alex/meta-alex/conf/templates/test-
> 1/conf-notes.txt
> +   You can try out the configuration with
> +   TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 .
> /srv/work/alex/poky/oe-init-build-env build-try-test-1
> 
> -Best practices dictate that you should keep your template configuration
> -directory in your custom distribution layer. For example, suppose you
> -have a layer named ``meta-mylayer`` located in your home directory and
> -you want your template configuration directory named ``myconf``.
> -Changing the ``.templateconf`` as follows causes the OpenEmbedded build
> -system to look in your directory and base its configuration files on the
> -``*.sample`` configuration files it finds. The final configuration files
> -(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in
> -your Build Directory, but they are based on your ``*.sample`` files.
> -::
> +The above command takes the config files from the currently active build
> directory under ``conf\``,
> +replaces site-specific paths in bblayers.conf with ##OECORE##-relative
> paths, and copies
> +the config files into a specified layer under a specified template name.
> 
> -   TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf}
> +To use those saved templates as a starting point for a build, users
> should point
> +to one of them with :term:`TEMPLATECONF` environment variable:
> 
> -Aside from the ``*.sample`` configuration files, the ``conf-notes.txt``
> -also resides in the default ``meta-poky/conf`` directory. The script
> -that sets up the build environment (i.e.
> -:ref:`structure-core-script`) uses this file to
> -display BitBake targets as part of the script output. Customizing this
> -``conf-notes.txt`` file is a good way to make sure your list of custom
> -targets appears as part of the script's output.
> +   TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 .
> /srv/work/alex/poky/oe-init-build-env build-try-test-1
> 
> -Here is the default list of targets displayed as a result of running
> -either of the setup scripts::
> +The OpenEmbedded build system uses the environment variable
> +:term:`TEMPLATECONF` to locate the directory from which it gathers
> +configuration information that ultimately ends up in the
> +:term:`Build Directory` ``conf`` directory.
> 
> -   You can now run 'bitbake <target>'
> +If :term:`TEMPLATECONF` is not set, the default value is obtained
> +from ```.templateconf`` file that is read from the same directory as
> +`oe-init-build-env` script. For ``poky`` reference distribution this
> +would be::
> 
> -   Common targets are:
> -       core-image-minimal
> -       core-image-sato
> -       meta-toolchain
> -       meta-ide-support
> +   TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf/templates/default}
> 
> -Changing the listed common targets is as easy as editing your version of
> -``conf-notes.txt`` in your custom template configuration directory and
> -making sure you have :term:`TEMPLATECONF` set to your directory.
> +If you look at a configuration template directory, you will
> +see the ``bblayers.conf.sample``, ``local.conf.sample``, and
> +``conf-notes.txt`` files. The build system uses these files to form the
> +respective ``bblayers.conf`` file, ``local.conf`` file, and show
> +users a note about the build they're setting up
> +when running the ``oe-init-build-env`` setup script. These can be
> +edited further if needed to improve or change the build configurations
> +available to the users.
> 
>  Conserving Disk Space
>  =====================
> --
> 2.30.2

Re: [docs] [PATCH 5/5] common-tasks.rst: rewrite the section about configuration templates

[Alexander Kanavin]

On Tue, 13 Sept 2022 at 19:38, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> I do not agree with this sentiment. If you create and distribute a setup
> similar to Poky (which we do), why would we not change .templateconf to
> point at our templates? This has been the documented method and the one
> at least we use for our distribution. Where have you gotten the notion
> that this is only to be used by OE-Core & Poky? It does not make sense.

First we need to find out, where the documentation section that sets
the idea is coming from. It was written by Scott R. in 2014:
https://git.yoctoproject.org/yocto-docs/commit/?id=c07fb7082fe08387bbc546b2a23620dedc7127b8
from
https://bugzilla.yoctoproject.org/show_bug.cgi?id=5895

I believe Scott misunderstood how the whole thing is supposed to work,
and never road tested what he had written. You are free to continue to
do the .templateconf rewrite, but I do not want to keep it in the
docs. It just isn't actually supported, unless you *also* provide a
description and tools for 'creating a distribution setup similar to
poky' and for doing the actual rewrite, and find someone else except
you who's using it (I believe you do this with custom unpublished
scripts?). Besides, the only feature it gets you is not having to
specify TEMPLATECONF when you have a default choice for it, which is
useful, but should be made possible with the setup everyone *actually*
uses, which is cloning poky, then adding layers next to it.

Alex

RE: [docs] [PATCH 5/5] common-tasks.rst: rewrite the section about configuration templates

[Peter Kjellerstedt]

> -----Original Message-----
> From: Alexander Kanavin <alex.kanavin@gmail.com>
> Sent: den 13 september 2022 19:27
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> Cc: docs@lists.yoctoproject.org; Alexander Kanavin <alex@linutronix.de>
> Subject: Re: [docs] [PATCH 5/5] common-tasks.rst: rewrite the section
> about configuration templates
> 
> On Tue, 13 Sept 2022 at 19:38, Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
> > I do not agree with this sentiment. If you create and distribute a setup
> > similar to Poky (which we do), why would we not change .templateconf to
> > point at our templates? This has been the documented method and the one
> > at least we use for our distribution. Where have you gotten the notion
> > that this is only to be used by OE-Core & Poky? It does not make sense.
> 
> First we need to find out, where the documentation section that sets
> the idea is coming from. It was written by Scott R. in 2014:
> https://git.yoctoproject.org/yocto-docs/commit/?id=c07fb7082fe08387bbc546b2a23620dedc7127b8
> from
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=5895
> 
> I believe Scott misunderstood how the whole thing is supposed to work,
> and never road tested what he had written.

And I think he documented it exactly as it was intended to be used. See 
Paul Eggleton's comment:

  One way to do it is to set TEMPLATECONF (an environment variable, not a 
  bitbake variable) to point to a directory where the custom conf-notes.txt 
  can be found. In a repository this can be set within a .templateconf file - 
  hopefully we document that somewhere as well."

> You are free to continue to
> do the .templateconf rewrite, but I do not want to keep it in the
> docs.

Well, I would prefer to keep it in the docs since it is a documented 
feature and (at least was) the recommended way to do things. Maybe this 
has changed over the years, but if it is removed from the docs there is 
nothing stopping someone from removing it. And just because it does not 
match how you use Yocto, doesn't mean it isn't useful to others.

> It just isn't actually supported, unless you *also* provide a
> description and tools for 'creating a distribution setup similar to
> poky' and for doing the actual rewrite, and find someone else except
> you who's using it (I believe you do this with custom unpublished
> scripts?).

Nowadays we do, since we gave up on the limitations of static 
configurations in the layers. Our repo manifests decide which layers 
are fetched and those are the layers that will be added to the 
bblayers.conf.sample file.

> Besides, the only feature it gets you is not having to
> specify TEMPLATECONF when you have a default choice for it, which is
> useful, but should be made possible with the setup everyone *actually*
> uses, which is cloning poky, then adding layers next to it.

Well, that's what we do too using our repo manifests. It's just that repo 
has the support to copy files into the directories it sets up, which we 
use to copy in a .templateconf that overrides the one from poky. Anyone 
can do that if they have a defined set of layers they use.

> 
> Alex

//Peter

Re: [docs] [PATCH 5/5] common-tasks.rst: rewrite the section about configuration templates

[Alexander Kanavin]

On Wed, 14 Sept 2022 at 01:59, Peter Kjellerstedt
<peter.kjellerstedt@axis.com> wrote:
> And I think he documented it exactly as it was intended to be used. See
> Paul Eggleton's comment:
>
>   One way to do it is to set TEMPLATECONF (an environment variable, not a
>   bitbake variable) to point to a directory where the custom conf-notes.txt
>   can be found. In a repository this can be set within a .templateconf file -
>   hopefully we document that somewhere as well."

Paul didn't think it through either when he wrote this. Specifically,
.templateconf must be directly next to oe-init-build-env, which means
it cannot simply be 'in a repository'.

You can't have it both ways: demand that this is kept in the docs, but
without explaining how to actually make it work (using standard
tooling). Also, your setup
is unnecessarily complicated to begin with: just skip the template
assembly altogether, and write directly to build/conf. Eventually
you'll be able to do it in a structured, supported
way with fragments, which is what we should *actually* be talking about.

Alex

Re: [docs] [PATCH 3/5] common-tasks.rst: describe the newly added layer setup and template config commands

[Quentin Schulz]

Hi Alex,

On 9/11/22 19:26, Alexander Kanavin wrote:
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>   documentation/dev-manual/common-tasks.rst | 11 +++++++++++
>   1 file changed, 11 insertions(+)
> 
> diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
> index c2afec946..4d19bd921 100644
> --- a/documentation/dev-manual/common-tasks.rst
> +++ b/documentation/dev-manual/common-tasks.rst
> @@ -728,6 +728,17 @@ The following list describes the available commands:
>   
>   -  ``create-layer``: Creates a basic layer.
>   
> +-  ``save-build-conf``: Saves the currently active build configuration
> +   (conf/local.conf, conf/bblayers.conf) as a template into a layer.

Double tick-quote the paths please, e.g.:
``conf/local.conf``

> +   This template can later be used for setting up builds via TEMPLATECONF.

:term:`TEMPLATECONF`

> +   For information about saving and usinc configuration templates, see

s/usinc/using/

> +   ":ref:`dev-manual/common-tasks:creating a custom template configuration directory`".
> +
> +-  ``create-layers-setup``: Writes out a configuration file and/or a script that
> +   can replicate the directory structure and revisions of the layers in a current build.
> +   For more information, see ":ref:`dev-manual/common-tasks:saving and restoring the layers setup`".
> +

Should we order alphabetically the list of available commands?

Cheers,
Quentin

Re: [docs] [PATCH 4/5] common-tasks.rst: describe the layer setup tooling

[Quentin Schulz]

Hi Alex,

On 9/11/22 19:26, Alexander Kanavin wrote:
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>   documentation/dev-manual/common-tasks.rst | 54 +++++++++++++++++++++++
>   1 file changed, 54 insertions(+)
> 
> diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
> index 4d19bd921..6c2df9aa6 100644
> --- a/documentation/dev-manual/common-tasks.rst
> +++ b/documentation/dev-manual/common-tasks.rst
> @@ -858,6 +858,60 @@ enables the build system to locate the layer during the build.
>      During a build, the OpenEmbedded build system looks in the layers
>      from the top of the list down to the bottom in that order.
>   
> +Saving and restoring the layers setup
> +-------------------------------------
> +
> +Once you have a working build with the correct set of layers, it is beneficial
> +to capture the layer setup - what they are, which repositories they come from
> +and which SCM revisions they're at - into a configuration file, so that this
> +setup can be easily replicated later, perhaps on a different machine. Here's
> +how to do this:
> +
> +   $ bitbake-layers create-layers-setup /srv/work/alex/meta-alex/
> +   NOTE: Starting bitbake server...
> +   NOTE: Created /srv/work/alex/meta-alex/setup-layers.json
> +   NOTE: Created /srv/work/alex/meta-alex/setup-layers
> +
> +The tool needs a single argument which tells where to place the output, consisting
> +of a json formatted layer configuration, and a 'setup-layers' script that can use that configuration
> +to restore the layers in a different location, or on a different host machine. The argument
> +can point to a custom layer (which is then deemed a 'bootstrap' layer that needs to be
> +checked out first), or into a completely independent location.
> +
> +The replication of the layers is performed by running the setup-layer script provided
> +above:
> +
> +1. Clone the bootstrap layer or some other repository to obtain
> +the json config and the setup script that can use it.
> +
> +2. Run the script directly with no options:
> +(note: this will work to update an existing checkout as well)
> +

This breaks the expected sphinx syntax.

Please move the note after the code block and add re-indent the "code" 
blocks with three more spaces (otherwise it thinks it's part of the 
bullet point text instead of a code block instead the bullet point text).

Also, add a second colon after "options:" above to start the code block.

> +   alex@Zen2:/srv/work/alex/my-build$ meta-alex/setup-layers
> +   Note: not checking out source meta-alex, use --force-bootstraplayer-checkout to override.
> +
> +   Setting up source meta-intel, revision 15.0-hardknott-3.3-310-g0a96edae, branch master
> +   Running 'git init -q /srv/work/alex/my-build/meta-intel'
> +   Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/meta-intel' in /srv/work/alex/my-build/meta-intel
> +   Running 'git fetch -q origin || true' in /srv/work/alex/my-build/meta-intel
> +   Running 'git checkout -q 0a96edae609a3f48befac36af82cf1eed6786b4a' in /srv/work/alex/my-build/meta-intel
> +
> +   Setting up source poky, revision 4.1_M1-372-g55483d28f2, branch akanavin/setup-layers
> +   Running 'git init -q /srv/work/alex/my-build/poky'
> +   Running 'git remote remove origin > /dev/null 2>&1; git remote add origin git://git.yoctoproject.org/poky' in /srv/work/alex/my-build/poky
> +   Running 'git fetch -q origin || true' in /srv/work/alex/my-build/poky
> +   Running 'git remote remove poky-contrib > /dev/null 2>&1; git remote add poky-contrib ssh://git@push.yoctoproject.org/poky-contrib' in /srv/work/alex/my-build/poky
> +   Running 'git fetch -q poky-contrib || true' in /srv/work/alex/my-build/poky
> +   Running 'git checkout -q 11db0390b02acac1324e0f827beb0e2e3d0d1d63' in /srv/work/alex/my-build/poky
> +

Add your note for existing checkout here.

> +.. note::
> +   The script is self-suffucient and requires only python3

s/self-suffucient/self-sufficient/

> +   and git on the host machne where it will run.
> +

s/machne/machine/

Replace "host machne where it will run" with "build machine"?

> +.. note::
> +   Both the 'create-layers-setup' and the 'setup-layers' provided several additional options
> +   that customize their behavior - you are welcome to study them via '--help' command line parameter.
> +

Replace single quotes by double-tick quotes to highlight.

Cheers,
Quentin

Re: [docs] [PATCH 4/5] common-tasks.rst: describe the layer setup tooling

[Quentin Schulz]

On 9/11/22 19:26, Alexander Kanavin wrote:
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>   documentation/dev-manual/common-tasks.rst | 54 +++++++++++++++++++++++
>   1 file changed, 54 insertions(+)
> 
> diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
> index 4d19bd921..6c2df9aa6 100644
> --- a/documentation/dev-manual/common-tasks.rst
> +++ b/documentation/dev-manual/common-tasks.rst
> @@ -858,6 +858,60 @@ enables the build system to locate the layer during the build.
>      During a build, the OpenEmbedded build system looks in the layers
>      from the top of the list down to the bottom in that order.
>   
> +Saving and restoring the layers setup
> +-------------------------------------
> +
> +Once you have a working build with the correct set of layers, it is beneficial
> +to capture the layer setup - what they are, which repositories they come from
> +and which SCM revisions they're at - into a configuration file, so that this
> +setup can be easily replicated later, perhaps on a different machine. Here's
> +how to do this:
> +

Double colon here to start a code-block.

> +   $ bitbake-layers create-layers-setup /srv/work/alex/meta-alex/
> +   NOTE: Starting bitbake server...
> +   NOTE: Created /srv/work/alex/meta-alex/setup-layers.json
> +   NOTE: Created /srv/work/alex/meta-alex/setup-layers
> +
> +The tool needs a single argument which tells where to place the output, consisting
> +of a json formatted layer configuration, and a 'setup-layers' script that can use that configuration

Replace single quotes with double-tick quotes to highlight.

> +to restore the layers in a different location, or on a different host machine. The argument
> +can point to a custom layer (which is then deemed a 'bootstrap' layer that needs to be

s/'bootstrap'/"bootstrap"/ ?

> +checked out first), or into a completely independent location.
> +
> +The replication of the layers is performed by running the setup-layer script provided
> +above:
> +

``setup-layer`` script

Sorry for the second mail, missed a few things.

Cheers,
Quentin

Re: [docs] [PATCH 5/5] common-tasks.rst: rewrite the section about configuration templates

[Quentin Schulz]

Hi Alex,

On 9/11/22 19:26, Alexander Kanavin wrote:
> This now includes a description about how to create a template
> with the newly added tooling, and drops the description
> of .templateconf as a way to point to a custom template
> (which is not correct: .templateconf is used only to point to a default
> template in poky or core when TEMPLATECONF is not specified).
> 
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>   documentation/dev-manual/common-tasks.rst | 92 +++++++++--------------
>   1 file changed, 36 insertions(+), 56 deletions(-)
> 
> diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
> index 6c2df9aa6..eab5648fd 100644
> --- a/documentation/dev-manual/common-tasks.rst
> +++ b/documentation/dev-manual/common-tasks.rst
> @@ -6463,71 +6463,51 @@ Creating a Custom Template Configuration Directory
>   ==================================================
>   
>   If you are producing your own customized version of the build system for
> -use by other users, you might want to customize the message shown by the
> -setup script or you might want to change the template configuration
> -files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a
> -new build directory.
> +use by other users, you might want to provide a custom build configuration
> +that includes all the necessary settings and layers (i.e. ``local.conf`` and
> +``bblayers.conf`` that are created in a new build directory) and a custom
> +message that is shown when setting up the build. This can be done by
> +creating one or more template configuration directories in your
> +custom distribution layer.
>   
> -The OpenEmbedded build system uses the environment variable
> -:term:`TEMPLATECONF` to locate the directory from which it gathers
> -configuration information that ultimately ends up in the
> -:term:`Build Directory` ``conf`` directory.
> -By default, :term:`TEMPLATECONF` is set as follows in the ``poky``
> -repository::
> +This can be done by using ``bitbake-layers save-build-conf``:
>   

Double colon to end a sentence to start a code-block missing here.

> -   TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
> -
> -This is the
> -directory used by the build system to find templates from which to build
> -some key configuration files. If you look at this directory, you will
> -see the ``bblayers.conf.sample``, ``local.conf.sample``, and
> -``conf-notes.txt`` files. The build system uses these files to form the
> -respective ``bblayers.conf`` file, ``local.conf`` file, and display the
> -list of BitBake targets when running the setup script.
> -
> -To override these default configuration files with configurations you
> -want used within every new Build Directory, simply set the
> -:term:`TEMPLATECONF` variable to your directory. The :term:`TEMPLATECONF`
> -variable is set in the ``.templateconf`` file, which is in the top-level
> -:term:`Source Directory` folder
> -(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your
> -directory.
> +   $ bitbake-layers save-build-conf ../../meta-alex/ test-1
> +   NOTE: Starting bitbake server...
> +   NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1
> +   Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt
> +   You can try out the configuration with
> +   TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
>   
> -Best practices dictate that you should keep your template configuration
> -directory in your custom distribution layer. For example, suppose you
> -have a layer named ``meta-mylayer`` located in your home directory and
> -you want your template configuration directory named ``myconf``.
> -Changing the ``.templateconf`` as follows causes the OpenEmbedded build
> -system to look in your directory and base its configuration files on the
> -``*.sample`` configuration files it finds. The final configuration files
> -(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in
> -your Build Directory, but they are based on your ``*.sample`` files.
> -::
> +The above command takes the config files from the currently active build directory under ``conf\``,

Spurious backslash.

> +replaces site-specific paths in bblayers.conf with ##OECORE##-relative paths, and copies

Please double-tick quote bblayers.conf and ##OECORE##.

> +the config files into a specified layer under a specified template name.
>   
> -   TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf}
> +To use those saved templates as a starting point for a build, users should point
> +to one of them with :term:`TEMPLATECONF` environment variable:
>   

Double colon to end a sentence to start a code-block missing here.

> -Aside from the ``*.sample`` configuration files, the ``conf-notes.txt``
> -also resides in the default ``meta-poky/conf`` directory. The script
> -that sets up the build environment (i.e.
> -:ref:`structure-core-script`) uses this file to
> -display BitBake targets as part of the script output. Customizing this
> -``conf-notes.txt`` file is a good way to make sure your list of custom
> -targets appears as part of the script's output.
> +   TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
>   
> -Here is the default list of targets displayed as a result of running
> -either of the setup scripts::
> +The OpenEmbedded build system uses the environment variable
> +:term:`TEMPLATECONF` to locate the directory from which it gathers
> +configuration information that ultimately ends up in the
> +:term:`Build Directory` ``conf`` directory.
>   
> -   You can now run 'bitbake <target>'
> +If :term:`TEMPLATECONF` is not set, the default value is obtained
> +from ```.templateconf`` file that is read from the same directory as

Spurious tick quote.

> +`oe-init-build-env` script. For ``poky`` reference distribution this

Double-tick quote oe-init-build-env.

Cheers,
Quentin

Re: [docs] [PATCH 1/5] ref-manual: correct the location of default configuration template

[Michael Opdenacker]

Hi Alex,

On 9/11/22 18:26, Alexander Kanavin wrote:
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>   documentation/ref-manual/structure.rst | 18 +++++++++---------
>   1 file changed, 9 insertions(+), 9 deletions(-)
>
> diff --git a/documentation/ref-manual/structure.rst b/documentation/ref-manual/structure.rst
> index bdcffc194..6c6f52e79 100644
> --- a/documentation/ref-manual/structure.rst
> +++ b/documentation/ref-manual/structure.rst
> @@ -189,7 +189,7 @@ Directory named ``mybuilds/`` that is outside of the :term:`Source Directory`::
>      $ source oe-init-build-env ~/mybuilds
>   
>   The OpenEmbedded build system uses the template configuration files, which
> -are found by default in the ``meta-poky/conf/`` directory in the Source
> +are found by default in the ``meta-poky/conf/templates/default`` directory in the Source
>   Directory. See the
>   ":ref:`dev-manual/common-tasks:creating a custom template configuration directory`"
>   section in the Yocto Project Development Tasks Manual for more
> @@ -261,15 +261,15 @@ OpenEmbedded build system creates it from ``local.conf.sample`` when you
>   :ref:`structure-core-script`.
>   
>   The source ``local.conf.sample`` file used depends on the
> -:term:`TEMPLATECONF` script variable, which defaults to ``meta-poky/conf/``
> +:term:`TEMPLATECONF` script variable, which defaults to ``meta-poky/conf/templates/default``
>   when you are building from the Yocto Project development environment,
> -and to ``meta/conf/`` when you are building from the OpenEmbedded-Core
> +and to ``meta/conf/templates/default`` when you are building from the OpenEmbedded-Core
>   environment. Because the script variable points to the source of the
>   ``local.conf.sample`` file, this implies that you can configure your
>   build environment from any layer by setting the variable in the
>   top-level build environment setup script as follows::
>   
> -   TEMPLATECONF=your_layer/conf
> +   TEMPLATECONF=your_layer/conf/templates/your_template_name
>   
>   Once the build process gets the sample
>   file, it uses ``sed`` to substitute final
> @@ -281,7 +281,7 @@ file, it uses ``sed`` to substitute final
>      You can see how the :term:`TEMPLATECONF` variable is used by looking at the
>      ``scripts/oe-setup-builddir`` script in the :term:`Source Directory`.
>      You can find the Yocto Project version of the ``local.conf.sample`` file in
> -   the ``meta-poky/conf`` directory.
> +   the ``meta-poky/conf/templates/default`` directory.
>   
>   .. _structure-build-conf-bblayers.conf:
>   
> @@ -301,14 +301,14 @@ you ``source`` the top-level build environment setup script (i.e.
>   
>   As with the ``local.conf`` file, the source ``bblayers.conf.sample``
>   file used depends on the :term:`TEMPLATECONF` script variable, which
> -defaults to ``meta-poky/conf/`` when you are building from the Yocto
> -Project development environment, and to ``meta/conf/`` when you are
> +defaults to ``meta-poky/conf/templates/default`` when you are building from the Yocto
> +Project development environment, and to ``meta/conf/templates/default`` when you are
>   building from the OpenEmbedded-Core environment. Because the script
>   variable points to the source of the ``bblayers.conf.sample`` file, this
>   implies that you can base your build from any layer by setting the
>   variable in the top-level build environment setup script as follows::
>   
> -   TEMPLATECONF=your_layer/conf
> +   TEMPLATECONF=your_layer/conf/templates/your_template_name
>   
>   Once the build process gets the sample file, it uses ``sed`` to substitute final
>   ``${``\ :term:`OEROOT`\ ``}`` values for all ``##OEROOT##`` values.
> @@ -317,7 +317,7 @@ Once the build process gets the sample file, it uses ``sed`` to substitute final
>   
>      You can see how the :term:`TEMPLATECONF` variable is defined by the ``scripts/oe-setup-builddir``
>      script in the :term:`Source Directory`. You can find the Yocto Project
> -   version of the ``bblayers.conf.sample`` file in the ``meta-poky/conf/``
> +   version of the ``bblayers.conf.sample`` file in the ``meta-poky/conf/templates/default``
>      directory.
>   
>   .. _structure-build-conf-sanity_info:


Thanks for the patch!
Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
... and merged into master-next.

Cheers
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH 2/5] common-tasks.rst: drop the output of 'bitbake-layers --help'

[Michael Opdenacker]

On 9/11/22 18:26, Alexander Kanavin wrote:
> This is really not needed as the sub-commands are described in
> greater detail just below, and is prone to become outdated.
>
> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> ---
>   documentation/dev-manual/common-tasks.rst | 33 -----------------------
>   1 file changed, 33 deletions(-)
>
> diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
> index bdc528b72..c2afec946 100644
> --- a/documentation/dev-manual/common-tasks.rst
> +++ b/documentation/dev-manual/common-tasks.rst
> @@ -653,39 +653,6 @@ For help on the BitBake layer management tool, use the following
>   command::
>   
>      $ bitbake-layers --help
> -   NOTE: Starting bitbake server...
> -   usage: bitbake-layers [-d] [-q] [-F] [--color COLOR] [-h] <subcommand> ...
> -
> -   BitBake layers utility
> -
> -   optional arguments:
> -     -d, --debug           Enable debug output
> -     -q, --quiet           Print only errors
> -     -F, --force           Force add without recipe parse verification
> -     --color COLOR         Colorize output (where COLOR is auto, always, never)
> -     -h, --help            show this help message and exit
> -
> -   subcommands:
> -     <subcommand>
> -       layerindex-fetch    Fetches a layer from a layer index along with its
> -                           dependent layers, and adds them to conf/bblayers.conf.
> -       layerindex-show-depends
> -                           Find layer dependencies from layer index.
> -       add-layer           Add one or more layers to bblayers.conf.
> -       remove-layer        Remove one or more layers from bblayers.conf.
> -       flatten             flatten layer configuration into a separate output
> -                           directory.
> -       show-layers         show current configured layers.
> -       show-overlayed      list overlayed recipes (where the same recipe exists
> -                           in another layer)
> -       show-recipes        list available recipes, showing the layer they are
> -                           provided by
> -       show-appends        list bbappend files and recipe files they apply to
> -       show-cross-depends  Show dependencies between recipes that cross layer
> -                           boundaries.
> -       create-layer        Create a basic layer
> -
> -   Use bitbake-layers <subcommand> --help to get help on a specific command
>   
>   The following list describes the available commands:


Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>
... and merged into "master-next".

Thanks!
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH 5/5] common-tasks.rst: rewrite the section about configuration templates

[Michael Opdenacker]

Greetings,

On 14.09.22 06:50, Alexander Kanavin wrote:
> On Wed, 14 Sept 2022 at 01:59, Peter Kjellerstedt
> <peter.kjellerstedt@axis.com> wrote:
>> And I think he documented it exactly as it was intended to be used. See
>> Paul Eggleton's comment:
>>
>>    One way to do it is to set TEMPLATECONF (an environment variable, not a
>>    bitbake variable) to point to a directory where the custom conf-notes.txt
>>    can be found. In a repository this can be set within a .templateconf file -
>>    hopefully we document that somewhere as well."
> Paul didn't think it through either when he wrote this. Specifically,
> .templateconf must be directly next to oe-init-build-env, which means
> it cannot simply be 'in a repository'.
>
> You can't have it both ways: demand that this is kept in the docs, but
> without explaining how to actually make it work (using standard
> tooling). Also, your setup
> is unnecessarily complicated to begin with: just skip the template
> assembly altogether, and write directly to build/conf. Eventually
> you'll be able to do it in a structured, supported
> way with fragments, which is what we should *actually* be talking about.


It's obvious we don't have a consensus yet here.
Can anyone else jump in and contribute to this discussion?

The other patches in this series have been merged into master-next.

Thanks in advance
Michael.

-- 
Michael Opdenacker, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH 5/5] common-tasks.rst: rewrite the section about configuration templates

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 1814 bytes --]

It’s not a need-for-consensus situation. It’s correcting something that
should have never been added in the first place.

Alex

On Tue 20. Sep 2022 at 21.46, Michael Opdenacker <
michael.opdenacker@bootlin.com> wrote:

> Greetings,
>
> On 14.09.22 06:50, Alexander Kanavin wrote:
> > On Wed, 14 Sept 2022 at 01:59, Peter Kjellerstedt
> > <peter.kjellerstedt@axis.com> wrote:
> >> And I think he documented it exactly as it was intended to be used. See
> >> Paul Eggleton's comment:
> >>
> >>    One way to do it is to set TEMPLATECONF (an environment variable,
> not a
> >>    bitbake variable) to point to a directory where the custom
> conf-notes.txt
> >>    can be found. In a repository this can be set within a .templateconf
> file -
> >>    hopefully we document that somewhere as well."
> > Paul didn't think it through either when he wrote this. Specifically,
> > .templateconf must be directly next to oe-init-build-env, which means
> > it cannot simply be 'in a repository'.
> >
> > You can't have it both ways: demand that this is kept in the docs, but
> > without explaining how to actually make it work (using standard
> > tooling). Also, your setup
> > is unnecessarily complicated to begin with: just skip the template
> > assembly altogether, and write directly to build/conf. Eventually
> > you'll be able to do it in a structured, supported
> > way with fragments, which is what we should *actually* be talking about.
>
>
> It's obvious we don't have a consensus yet here.
> Can anyone else jump in and contribute to this discussion?
>
> The other patches in this series have been merged into master-next.
>
> Thanks in advance
> Michael.
>
> --
> Michael Opdenacker, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>
>

[-- Attachment #2: Type: text/html, Size: 2531 bytes --]

Re: [docs] [PATCH 5/5] common-tasks.rst: rewrite the section about configuration templates

[Alexander Kanavin]

To clarify, the .templateconf stuff can be brought back, but it must
come with specific explanation about how to use it if it comes from
anything else than poky or oe-core. Which is not possible with current
tooling in core. But Peter is welcome to write that section.

Alex

On Tue, 20 Sept 2022 at 22:54, Alexander Kanavin via
lists.yoctoproject.org <alex.kanavin=gmail.com@lists.yoctoproject.org>
wrote:
>
> It’s not a need-for-consensus situation. It’s correcting something that should have never been added in the first place.
>
> Alex
>
> On Tue 20. Sep 2022 at 21.46, Michael Opdenacker <michael.opdenacker@bootlin.com> wrote:
>>
>> Greetings,
>>
>> On 14.09.22 06:50, Alexander Kanavin wrote:
>> > On Wed, 14 Sept 2022 at 01:59, Peter Kjellerstedt
>> > <peter.kjellerstedt@axis.com> wrote:
>> >> And I think he documented it exactly as it was intended to be used. See
>> >> Paul Eggleton's comment:
>> >>
>> >>    One way to do it is to set TEMPLATECONF (an environment variable, not a
>> >>    bitbake variable) to point to a directory where the custom conf-notes.txt
>> >>    can be found. In a repository this can be set within a .templateconf file -
>> >>    hopefully we document that somewhere as well."
>> > Paul didn't think it through either when he wrote this. Specifically,
>> > .templateconf must be directly next to oe-init-build-env, which means
>> > it cannot simply be 'in a repository'.
>> >
>> > You can't have it both ways: demand that this is kept in the docs, but
>> > without explaining how to actually make it work (using standard
>> > tooling). Also, your setup
>> > is unnecessarily complicated to begin with: just skip the template
>> > assembly altogether, and write directly to build/conf. Eventually
>> > you'll be able to do it in a structured, supported
>> > way with fragments, which is what we should *actually* be talking about.
>>
>>
>> It's obvious we don't have a consensus yet here.
>> Can anyone else jump in and contribute to this discussion?
>>
>> The other patches in this series have been merged into master-next.
>>
>> Thanks in advance
>> Michael.
>>
>> --
>> Michael Opdenacker, Bootlin
>> Embedded Linux and Kernel engineering
>> https://bootlin.com
>>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#3191): https://lists.yoctoproject.org/g/docs/message/3191
> Mute This Topic: https://lists.yoctoproject.org/mt/93615912/1686489
> Group Owner: docs+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/docs/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [docs] [PATCH 3/5] common-tasks.rst: describe the newly added layer setup and template config commands

[Alexander Kanavin]

I'm not sure alphabetical order makes sense. There is a certain
logical order in describing the commands that follows a typical
'setting up a build and configuration' workflow, and that is perhaps
better. In any case, that should be a separate commit, as reordering
things should always be kept separate from modifying them.

Thanks for all the fixups!

Alex

On Tue, 20 Sept 2022 at 22:28, Michael Opdenacker
<michael.opdenacker@bootlin.com> wrote:
>
> Quentin, Alex,
>
> Thanks for the patch and review.
>
> On 19.09.22 17:23, Quentin Schulz via lists.yoctoproject.org wrote:
> > Hi Alex,
> >
> > On 9/11/22 19:26, Alexander Kanavin wrote:
> >> Signed-off-by: Alexander Kanavin <alex@linutronix.de>
> >> ---
> >>   documentation/dev-manual/common-tasks.rst | 11 +++++++++++
> >>   1 file changed, 11 insertions(+)
> >>
> >> diff --git a/documentation/dev-manual/common-tasks.rst
> >> b/documentation/dev-manual/common-tasks.rst
> >> index c2afec946..4d19bd921 100644
> >> --- a/documentation/dev-manual/common-tasks.rst
> >> +++ b/documentation/dev-manual/common-tasks.rst
> >> @@ -728,6 +728,17 @@ The following list describes the available
> >> commands:
> >>     -  ``create-layer``: Creates a basic layer.
> >>   +-  ``save-build-conf``: Saves the currently active build
> >> configuration
> >> +   (conf/local.conf, conf/bblayers.conf) as a template into a layer.
> >
> > Double tick-quote the paths please, e.g.:
> > ``conf/local.conf``
> >
> >> +   This template can later be used for setting up builds via
> >> TEMPLATECONF.
> >
> > :term:`TEMPLATECONF`
> >
> >> +   For information about saving and usinc configuration templates, see
> >
> > s/usinc/using/
> >
> >> + ":ref:`dev-manual/common-tasks:creating a custom template
> >> configuration directory`".
> >> +
> >> +-  ``create-layers-setup``: Writes out a configuration file and/or a
> >> script that
> >> +   can replicate the directory structure and revisions of the layers
> >> in a current build.
> >> +   For more information, see ":ref:`dev-manual/common-tasks:saving
> >> and restoring the layers setup`".
> >> +
>
>
> I took your review into account, Quentin.
>
> >
> > Should we order alphabetically the list of available commands?
>
>
> Maybe, this should be another change, as the commands are not sorted
> alphabetically in the existing text. At least, I modified Alex' changes
> to insert them in a way that is consistent with the order in the
> "--help" output.
>
> I agree though that sorting commands alphabetically would help, in the
> documentation, but in the code too. Would the latter make sense?
>
> Anyway, I merged this change into master-next. I had to apply the 4/5
> patch first, otherwise, this one was making a reference to a chapter
> that didn't exist yet.
>
> Thanks again!
> Michael.
>
> --
> Michael Opdenacker, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>

[PATCH v2] dev-manual: common-tasks.rst: rewrite the section about configuration templates

[michael.opdenacker]

From: Michael Opdenacker <michael.opdenacker@bootlin.com>

From: Alexander Kanavin <alex.kanavin@gmail.com>

This now includes a description about how to create a template
with the newly added tooling, and drops the description
of .templateconf as a way to point to a custom template
(which is not correct: .templateconf is used only to point to a default
template in poky or core when TEMPLATECONF is not specified).

Signed-off-by: Alexander Kanavin <alex@linutronix.de>
Reviewed-by: Michael Opdenacker <michael.opdenacker@bootlin.com>

---
Changes in V2:
- Misc syntax fixes from the reviews from Quentin Schulz and Michael Opdenacker
---
 documentation/dev-manual/common-tasks.rst | 92 +++++++++--------------
 1 file changed, 36 insertions(+), 56 deletions(-)

diff --git a/documentation/dev-manual/common-tasks.rst b/documentation/dev-manual/common-tasks.rst
index 837824be38..cfbd0bef02 100644
--- a/documentation/dev-manual/common-tasks.rst
+++ b/documentation/dev-manual/common-tasks.rst
@@ -6466,71 +6466,51 @@ Creating a Custom Template Configuration Directory
 ==================================================
 
 If you are producing your own customized version of the build system for
-use by other users, you might want to customize the message shown by the
-setup script or you might want to change the template configuration
-files (i.e. ``local.conf`` and ``bblayers.conf``) that are created in a
-new build directory.
+use by other users, you might want to provide a custom build configuration
+that includes all the necessary settings and layers (i.e. ``local.conf`` and
+``bblayers.conf`` that are created in a new build directory) and a custom
+message that is shown when setting up the build. This can be done by
+creating one or more template configuration directories in your
+custom distribution layer.
 
-The OpenEmbedded build system uses the environment variable
-:term:`TEMPLATECONF` to locate the directory from which it gathers
-configuration information that ultimately ends up in the
-:term:`Build Directory` ``conf`` directory.
-By default, :term:`TEMPLATECONF` is set as follows in the ``poky``
-repository::
+This can be done by using ``bitbake-layers save-build-conf``::
 
-   TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf}
-
-This is the
-directory used by the build system to find templates from which to build
-some key configuration files. If you look at this directory, you will
-see the ``bblayers.conf.sample``, ``local.conf.sample``, and
-``conf-notes.txt`` files. The build system uses these files to form the
-respective ``bblayers.conf`` file, ``local.conf`` file, and display the
-list of BitBake targets when running the setup script.
-
-To override these default configuration files with configurations you
-want used within every new Build Directory, simply set the
-:term:`TEMPLATECONF` variable to your directory. The :term:`TEMPLATECONF`
-variable is set in the ``.templateconf`` file, which is in the top-level
-:term:`Source Directory` folder
-(e.g. ``poky``). Edit the ``.templateconf`` so that it can locate your
-directory.
+   $ bitbake-layers save-build-conf ../../meta-alex/ test-1
+   NOTE: Starting bitbake server...
+   NOTE: Configuration template placed into /srv/work/alex/meta-alex/conf/templates/test-1
+   Please review the files in there, and particularly provide a configuration description in /srv/work/alex/meta-alex/conf/templates/test-1/conf-notes.txt
+   You can try out the configuration with
+   TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
 
-Best practices dictate that you should keep your template configuration
-directory in your custom distribution layer. For example, suppose you
-have a layer named ``meta-mylayer`` located in your home directory and
-you want your template configuration directory named ``myconf``.
-Changing the ``.templateconf`` as follows causes the OpenEmbedded build
-system to look in your directory and base its configuration files on the
-``*.sample`` configuration files it finds. The final configuration files
-(i.e. ``local.conf`` and ``bblayers.conf`` ultimately still end up in
-your Build Directory, but they are based on your ``*.sample`` files.
-::
+The above command takes the config files from the currently active build directory under ``conf``,
+replaces site-specific paths in ``bblayers.conf`` with ``##OECORE##``-relative paths, and copies
+the config files into a specified layer under a specified template name.
 
-   TEMPLATECONF=${TEMPLATECONF:-meta-mylayer/myconf}
+To use those saved templates as a starting point for a build, users should point
+to one of them with :term:`TEMPLATECONF` environment variable::
 
-Aside from the ``*.sample`` configuration files, the ``conf-notes.txt``
-also resides in the default ``meta-poky/conf`` directory. The script
-that sets up the build environment (i.e.
-:ref:`structure-core-script`) uses this file to
-display BitBake targets as part of the script output. Customizing this
-``conf-notes.txt`` file is a good way to make sure your list of custom
-targets appears as part of the script's output.
+   TEMPLATECONF=/srv/work/alex/meta-alex/conf/templates/test-1 . /srv/work/alex/poky/oe-init-build-env build-try-test-1
 
-Here is the default list of targets displayed as a result of running
-either of the setup scripts::
+The OpenEmbedded build system uses the environment variable
+:term:`TEMPLATECONF` to locate the directory from which it gathers
+configuration information that ultimately ends up in the
+:term:`Build Directory` ``conf`` directory.
 
-   You can now run 'bitbake <target>'
+If :term:`TEMPLATECONF` is not set, the default value is obtained
+from ``.templateconf`` file that is read from the same directory as
+``oe-init-build-env`` script. For the Poky reference distribution this
+would be::
 
-   Common targets are:
-       core-image-minimal
-       core-image-sato
-       meta-toolchain
-       meta-ide-support
+   TEMPLATECONF=${TEMPLATECONF:-meta-poky/conf/templates/default}
 
-Changing the listed common targets is as easy as editing your version of
-``conf-notes.txt`` in your custom template configuration directory and
-making sure you have :term:`TEMPLATECONF` set to your directory.
+If you look at a configuration template directory, you will
+see the ``bblayers.conf.sample``, ``local.conf.sample``, and
+``conf-notes.txt`` files. The build system uses these files to form the
+respective ``bblayers.conf`` file, ``local.conf`` file, and show
+users a note about the build they're setting up
+when running the ``oe-init-build-env`` setup script. These can be
+edited further if needed to improve or change the build configurations
+available to the users.
 
 Conserving Disk Space
 =====================
-- 
2.34.1