[PATCH v4 0/3] Add bblock documentation

[PATCH v4 0/3] Add bblock documentation

[Julien Stephan]

Hello,

The main purpose of this series is to add documentation for bblock.

bblock uses internally the 3 following variables: SIGGEN_LOCKEDSIGS,
SIGGEN_LOCKEDSIGS_TASKSIG_CHECK and SIGGEN_LOCKEDSIGS_TYPE which are not
documentated, so adding them to the glossary.

I also added "sstate" into the Yocto vocabulary to avoir error when running
Vale.

This series was built and checked with:

  $ make html
  $ make stylecheck VALEDOCS=dev-manual/bblock.rst
  $ make sphinx-lint SPHINXLINTDOCS=dev-manual/bblock.rst

I tried my best to make Vale happy, but I still have the following
warnings:

 dev-manual/bblock.rst
 1:1  warning     Try to keep the Coleman–Liau Index grade (10.29) below 9.                           Readability.ColemanLiau
 1:1  warning     Try to keep the Flesch reading ease score (57.51) above 70.                         Readability.FleschReadingEase
 1:1  warning     Try to keep the Flesch–Kincaid grade level (8.28) below 8.                          Readability.FleschKincaid
 1:1  warning     Try to keep the SMOG grade (11.42) below 10.                                        Readability.SMOG
 1:1  warning     Try to keep the LIX score (38.20) below 35.                                         Readability.LIX
 3:1  suggestion  Use sentence-style capitalization in 'Locking and Unlocking Recipes Using ******'.  RedHat.Headings
 29:49  suggestion  Consider wrapping this Pascal or Camel case term ('TipsAndTricks') in backticks.    RedHat.PascalCamelCase
 39:60  suggestion  Use simple language. Consider using 'many' rather than 'multiple'.                  RedHat.SimpleWords
 39:60  warning     'multiple' is too wordy.                                                            write-good.TooWordy
 57:62  warning     'multiple' is too wordy.                                                            write-good.TooWordy
 76:43  error       Use 'BitBake' instead of 'bitbake'.                                                 Vale.Terms
 76:58  error       Use 'OpenEmbedded' instead of 'openembedded'.                                       Vale.Terms
 76:71  warning     Use either 'or' or 'and' in 'core/tree'                                             RedHat.Slash
 76:81  warning     Use either 'or' or 'and' in 'meta/conf'                                             RedHat.Slash
 76:91  error       Use 'BitBake' instead of 'bitbake'.                                                 Vale.Terms

About the Readability, I am not sure I can do better ^^ But anyone with
better english skills than me can help here!

About the capitalization: I decided to ignore it to keep consistency with
all other headings of the reference manual.

Suggestion and errors/warnings on lines 29 and 76 are false positive, we may need
to add some rule to ignore these check on :oe_git: directives
Also ignore the warning about 'multiple' as 'many' does not fit well in
this case.

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
Changes in v4:
- use "the sstate cache" everywhere for consistency
- fix some typo
- add comment to explain that bblock.conf does not exist by default
- Link to v3: https://lore.kernel.org/r/20241105-add-bblock-documentation-v3-0-b870ded39a2d@baylibre.com

Changes in v3:
- Fix missing typos reported by Ulrich Olmann in v1
- Link to v2: https://lore.kernel.org/r/20241104-add-bblock-documentation-v2-0-0704e4d59929@baylibre.com

Changes in v2:
- Fixed various typos reported by Ulrich Ölmann and Antonin Godard
- Splited the series in two: the Readme/Makefile changes related to Vale
  and SPHINXLINT and bblock documentation
- Added a bblock.conf file in `documentation/ref-manual/structure.rst`
- Link to v1: https://lore.kernel.org/r/20241031-add-bblock-documentation-v1-0-32b89093bbda@baylibre.com

---
Julien Stephan (3):
      styles: vocabularies: Yocto: add sstate
      ref-manual: variables: add SIGGEN_LOCKEDSIGS* variables
      dev-manual: add bblock documentation

 documentation/dev-manual/bblock.rst                | 129 +++++++++++++++++++++
 documentation/dev-manual/index.rst                 |   1 +
 documentation/ref-manual/structure.rst             |   9 ++
 documentation/ref-manual/variables.rst             |  47 ++++++++
 .../styles/config/vocabularies/Yocto/accept.txt    |   1 +
 5 files changed, 187 insertions(+)
---
base-commit: 7b40c7b73a360a1ec383c4c9f00c3e126208320b
change-id: 20241031-add-bblock-documentation-4a5f8cf9c6d2

Best regards,
-- 
Julien Stephan <jstephan@baylibre.com>

[PATCH v4 1/3] styles: vocabularies: Yocto: add sstate

[Julien Stephan]

Add sstate as an accepted word to avoid errors when runnign make
stylecheck.

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 documentation/styles/config/vocabularies/Yocto/accept.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/documentation/styles/config/vocabularies/Yocto/accept.txt b/documentation/styles/config/vocabularies/Yocto/accept.txt
index ca622ba41274c892f1fc1e16d8332a02f07a3531..7fedda8ae23c22f3660292faa0a1c3df23da353e 100644
--- a/documentation/styles/config/vocabularies/Yocto/accept.txt
+++ b/documentation/styles/config/vocabularies/Yocto/accept.txt
@@ -2,4 +2,5 @@ BitBake
 BSP
 crosstap
 OpenEmbedded
+sstate
 Yocto

-- 
2.47.0

[PATCH v4 2/3] ref-manual: variables: add SIGGEN_LOCKEDSIGS* variables

[Julien Stephan]

Variables SIGGEN_LOCKEDSIGS, SIGGEN_LOCKEDSIGS_TASKSIG_CHECK and
SIGGEN_LOCKEDSIGS_TYPES are used to lock specific tasks to specific
signatures. They are used by bitbake -S <lockedsigs> and bblock, so add
documentation for them.

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 documentation/ref-manual/variables.rst | 44 ++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index 07ed143ac5606617c720301ac6621355a341f90a..76fc50ecbfb841125860e7dac42d4d4c102942cf 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -7880,6 +7880,50 @@ system and gives an overview of their function and contents.
          might break at runtime if the interface of the recipe was changed
          after the other had been built.
 
+   :term:`SIGGEN_LOCKEDSIGS`
+     The list of locked tasks, with the form::
+
+       SIGGEN_LOCKEDSIGS += "<package>:<task>:<signature>"
+
+     If ``<signature>`` exists for the specified ``<task>`` and ``<package>``
+     in the sstate cache, BitBake will use the cached output instead of
+     rebuilding the ``<task>``. If it does not exist, BitBake will build the
+     ``<task>`` and the sstate cache will be used next time.
+
+     Example::
+
+       SIGGEN_LOCKEDSIGS += "bc:do_compile:09772aa4532512baf96d433484f27234d4b7c11dd9cda0d6f56fa1b7ce6f25f0"
+
+     You can obtain the signature of all the tasks for the recipe ``bc`` using::
+
+       bitbake -S none bc
+
+     Then you can look at files in ``build/stamps/<arch>/bc`` and search for the
+     file: ``<PV>.do_compile.sigdata.09772aa4532512baf96d433484f27234d4b7c11dd9cda0d6f56fa1b7ce6f25f0``.
+
+   :term:`SIGGEN_LOCKEDSIGS_TASKSIG_CHECK`
+     Specifies the debug level of task signature check. 3 levels are supported:
+
+     * ``info``: displays a "Note" message to remind the user that a task is locked
+       and the current signature matches the locked one.
+     * ``warn``: displays a "Warning" message if a task is locked and the current
+       signature does not match the locked one.
+     * ``error``: same as warn but displays an "Error" message and aborts.
+
+   :term:`SIGGEN_LOCKEDSIGS_TYPES`
+     Allowed overrides for :term:`SIGGEN_LOCKEDSIGS`. This is mainly used
+     for architecture specific locks. A common value for
+     :term:`SIGGEN_LOCKEDSIGS_TYPES` is ``${PACKAGE_ARCHS}``::
+
+       SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"
+
+       SIGGEN_LOCKEDSIGS_core2-64 += "bc:do_compile:09772aa4532512baf96d433484f27234d4b7c11dd9cda0d6f56fa1b7ce6f25f0"
+       SIGGEN_LOCKEDSIGS_cortexa57 += "bc:do_compile:12178eb6d55ef602a8fe638e49862fd247e07b228f0f08967697b655bfe4bb61"
+
+     Here, the ``do_compile`` task from ``bc`` will be locked only for
+     ``core2-64`` and ``cortexa57`` but not for other architectures such as
+     ``mips32r2``.
+
    :term:`SITEINFO_BITS`
       Specifies the number of bits for the target system CPU. The value
       should be either "32" or "64".

-- 
2.47.0

[PATCH v4 3/3] dev-manual: add bblock documentation

[Julien Stephan]

bblock is a helper tool to lock/unlock tasks and recipes to specific
signatures. Add a documentation page for it.

Signed-off-by: Julien Stephan <jstephan@baylibre.com>
---
 documentation/dev-manual/bblock.rst    | 129 +++++++++++++++++++++++++++++++++
 documentation/dev-manual/index.rst     |   1 +
 documentation/ref-manual/structure.rst |   9 +++
 documentation/ref-manual/variables.rst |   3 +
 4 files changed, 142 insertions(+)

diff --git a/documentation/dev-manual/bblock.rst b/documentation/dev-manual/bblock.rst
new file mode 100644
index 0000000000000000000000000000000000000000..68292903bab039ad66936a5061887a0ed569dfb4
--- /dev/null
+++ b/documentation/dev-manual/bblock.rst
@@ -0,0 +1,129 @@
+.. SPDX-License-Identifier: CC-BY-SA-2.0-UK
+
+Locking and Unlocking Recipes Using ``bblock``
+**********************************************
+
+By design, the OpenEmbedded build system builds everything from scratch
+unless BitBake determines that specific tasks do not require rebuilding.
+At startup, it computes a signature for all tasks, based on the task's input.
+Then, it compares these signatures with the ones from the sstate cache (if they
+exist). Any changes cause the task to rerun.
+
+During development, changes might trigger BitBake to rebuild certain
+recipes, even when we know they do not require rebuilding at that stage.
+For example, modifying a recipe can lead to rebuilding its native
+counterpart, which might prove unnecessary. Editing the ``python3`` recipe,
+for instance, can prompt BitBake to rebuild ``python3-native`` along with any
+recipes that depend on it.
+
+To prevent this, use ``bblock`` to lock specific tasks or recipes to
+specific signatures, forcing BitBake to use the sstate cache for them.
+
+.. attention::
+
+   Use ``bblock`` only during the development phase.
+
+   Forcing BitBake to use the sstate cache, regardless of input changes, means
+   the recipe metadata no longer directly reflect the output. Use this feature
+   with caution. If you do not understand why signatures change, see the section
+   on :yocto_wiki:`understanding what changed </TipsAndTricks/Understanding_what_changed_(diffsigs_etc)>`.
+
+
+Locking tasks and recipes
+-------------------------
+
+To lock a recipe, use::
+
+   $ bblock recipe
+
+You can also use a space-separated list of recipes to lock multiple recipes::
+
+   $ bblock recipe1 recipe2
+
+Locking a recipe means locking all tasks of the recipe. If you need to
+lock only particular tasks, use the `-t` option with a comma-separated
+list of tasks::
+
+  $ bblock -t task1,task2 recipe
+
+
+Unlocking tasks and recipes
+---------------------------
+
+To unlock a recipe, use the ``-r`` option::
+
+   $ bblock -r recipe
+
+You can also use a space-separated list of recipes to unlock multiple recipes::
+
+   $ bblock -r recipe1 recipe2
+
+Unlocking a recipe means unlocking all tasks of the recipe. If you need to
+unlock only particular tasks use the ``-t`` option with a comma-separated
+list of tasks::
+
+  $ bblock -r -t task1,task2 recipe
+
+To unlock all recipes, do not specify any recipe::
+
+  $ bblock -r
+
+
+Configuration file
+------------------
+
+``bblock`` will dump the signatures in the ``build/conf/bblock.conf`` file,
+included by default in :oe_git:`meta/conf/bitbake.conf </openembedded-core/tree/meta/conf/bitbake.conf>`.
+
+To dump the file, use the ``-d`` option::
+
+  $ bblock -d
+
+
+Locking mechanism
+-----------------
+
+``bblock`` computes the signature(s) of the task(s) and sets the 3 following
+variables: :term:`SIGGEN_LOCKEDSIGS`, :term:`SIGGEN_LOCKEDSIGS_TYPES`
+and :term:`SIGGEN_LOCKEDSIGS_TASKSIG_CHECK`.
+
+In particular, ``bblock`` sets::
+
+  SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"
+  SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"
+
+  SIGGEN_LOCKEDSIGS_<package_arch> += "<recipe>:<task>:<signature>"
+
+This produces architecture specific locks and reminds user that some tasks
+have locked signatures.
+
+Example
+-------
+
+When working on the ``python3`` recipe, we can lock ``python3-native`` with
+the following::
+
+  $ bblock python3-native
+  $ bblock -d
+  # Generated by bblock
+  SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "info"
+  SIGGEN_LOCKEDSIGS_TYPES += "${PACKAGE_ARCHS}"
+
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_patch:865859c27e603ba42025b7bb766c3cd4c0f477e4962cfd39128c0619d695fce7"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_populate_sysroot:f8fa5d3194cef638416000252b959e86d0a19f6b7898e1f56b643c588cdd8605"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_prepare_recipe_sysroot:fe295ac505d9d1143313424b201c6f3f2a0a90da40a13a905b86b874705f226a"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_fetch:1b6e4728fee631bc7a8a7006855c5b8182a8224579e32e3d0a2db77c26459f25"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_unpack:2ad74d6f865ef75c35c0e6bbe3f9a90923a6b2c62c18a3ddef514ea31fbc588f"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_deploy_source_date_epoch:15f89b8483c1ad7507480f337619bb98c26e231227785eb3543db163593e7b42"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_configure:7960c13d23270fdb12b3a7c426ce1da0d2f5c7cf5e5d3f5bdce5fa330eb7d482"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_compile:012e1d4a63f1a78fc2143bd90d704dbcf5865c5257d6272aa7540ec1cd3063d9"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_install:d3401cc2afa4c996beb154beaad3e45fa0272b9c56fb86e9db14ec3544c68f9d"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_build:fa88bb7afb9046c0417c24a3fa98a058653805a8b00eda2c2d7fea68fc42f882"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_collect_spdx_deps:cc9c53ba7c495567e9a38ec4801830c425c0d1f895aa2fc66930a2edd510d9b4"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_create_spdx:766a1d09368438b7b5a1a8e2a8f823b2b731db44b57e67d8b3196de91966f9c5"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_create_package_spdx:46f80faeab25575e9977ba3bf14c819489c3d489432ae5145255635108c21020"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_recipe_qa:cb960cdb074e7944e894958db58f3dc2a0436ecf87c247feb3e095e214fec0e4"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_populate_lic:15657441621ee83f15c2e650e7edbb036870b56f55e72e046c6142da3c5783fd"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_create_manifest:24f0abbec221d27bbb2909b6e846288b12cab419f1faf9f5006ed80423d37e28"
+  SIGGEN_LOCKEDSIGS_x86_64 += "python3-native:do_addto_recipe_sysroot:bcb6a1905f113128de3f88d702b706befd6a786267c045ee82532759a7c214d7"
+
diff --git a/documentation/dev-manual/index.rst b/documentation/dev-manual/index.rst
index 9ccf60f7012ea344612298e77ebabed84ad6de3e..7afd0d820e94d56cb2145962b9b1a976c7951936 100644
--- a/documentation/dev-manual/index.rst
+++ b/documentation/dev-manual/index.rst
@@ -48,5 +48,6 @@ Yocto Project Development Tasks Manual
    error-reporting-tool
    wayland
    qemu
+   bblock
 
 .. include:: /boilerplate.rst
diff --git a/documentation/ref-manual/structure.rst b/documentation/ref-manual/structure.rst
index bbeda78c791d430faa55356a3209f5eeb1402837..616c3c4c9bf375bb60ab34587b2c70bdb42d280b 100644
--- a/documentation/ref-manual/structure.rst
+++ b/documentation/ref-manual/structure.rst
@@ -335,6 +335,15 @@ Once the build process gets the sample file, it uses ``sed`` to substitute final
    version of the ``bblayers.conf.sample`` file in the ``meta-poky/conf/templates/default``
    directory.
 
+.. _structure-build-conf-bblock.conf:
+
+``build/conf/bblock.conf``
+--------------------------
+
+This configuration file is generated by :doc:`bblock </dev-manual/bblock>` and
+contains the signatures locked by ``bblock``. By default, it does not exist
+and will be created upon the first invocation of ``bblock``.
+
 .. _structure-build-downloads:
 
 ``build/downloads/``
diff --git a/documentation/ref-manual/variables.rst b/documentation/ref-manual/variables.rst
index 76fc50ecbfb841125860e7dac42d4d4c102942cf..831b7c54c89bdeb64b8ce3bbf512a510b94dfc02 100644
--- a/documentation/ref-manual/variables.rst
+++ b/documentation/ref-manual/variables.rst
@@ -7901,6 +7901,9 @@ system and gives an overview of their function and contents.
      Then you can look at files in ``build/stamps/<arch>/bc`` and search for the
      file: ``<PV>.do_compile.sigdata.09772aa4532512baf96d433484f27234d4b7c11dd9cda0d6f56fa1b7ce6f25f0``.
 
+     Alternatively, you can also use :doc:`bblock </dev-manual/bblock>` to
+     generate this line for you.
+
    :term:`SIGGEN_LOCKEDSIGS_TASKSIG_CHECK`
      Specifies the debug level of task signature check. 3 levels are supported:
 

-- 
2.47.0

Re: [docs] [PATCH v4 2/3] ref-manual: variables: add SIGGEN_LOCKEDSIGS* variables

[Antonin Godard]

Hi Julien,

On Wed Nov 6, 2024 at 10:02 AM CET, Julien Stephan wrote:
[...]
> +     Then you can look at files in ``build/stamps/<arch>/bc`` and search for the

Didn't notice before, but it should be ``tmp/stamps/<arch>/bc``, right?

> +     file: ``<PV>.do_compile.sigdata.09772aa4532512baf96d433484f27234d4b7c11dd9cda0d6f56fa1b7ce6f25f0``.
> +

Also, how about s/search for the file/look for files like:/.
The current phrasing makes it look like this is only valid for `do_compile`.

Otherwise, this looks good to me, thanks!

Antonin

--
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [docs] [PATCH v4 2/3] ref-manual: variables: add SIGGEN_LOCKEDSIGS* variables

[Julien Stephan]

Le mer. 6 nov. 2024 à 11:30, Antonin Godard
<antonin.godard@bootlin.com> a écrit :
>
> Hi Julien,
>
> On Wed Nov 6, 2024 at 10:02 AM CET, Julien Stephan wrote:
> [...]
> > +     Then you can look at files in ``build/stamps/<arch>/bc`` and search for the
>
> Didn't notice before, but it should be ``tmp/stamps/<arch>/bc``, right?

right, I forgot the ``tmp``. Should I use
``build/tmp/stamps/<arch>/bc``  or ``tmp/stamps/<arch>/bc`` ?
I want confirmation to avoid another round ;)

>
> > +     file: ``<PV>.do_compile.sigdata.09772aa4532512baf96d433484f27234d4b7c11dd9cda0d6f56fa1b7ce6f25f0``.
> > +
>
> Also, how about s/search for the file/look for files like:/.
> The current phrasing makes it look like this is only valid for `do_compile`.

I'll fix this also in the new version.

Thank you

Cheers
Julien
>
> Otherwise, this looks good to me, thanks!
>
> Antonin
>
> --
> Antonin Godard, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>

Re: [docs] [PATCH v4 2/3] ref-manual: variables: add SIGGEN_LOCKEDSIGS* variables

[Antonin Godard]

On Wed Nov 6, 2024 at 2:34 PM CET, Julien Stephan wrote:
> Le mer. 6 nov. 2024 à 11:30, Antonin Godard
> <antonin.godard@bootlin.com> a écrit :
>>
>> Hi Julien,
>>
>> On Wed Nov 6, 2024 at 10:02 AM CET, Julien Stephan wrote:
>> [...]
>> > +     Then you can look at files in ``build/stamps/<arch>/bc`` and search for the
>>
>> Didn't notice before, but it should be ``tmp/stamps/<arch>/bc``, right?
>
> right, I forgot the ``tmp``. Should I use
> ``build/tmp/stamps/<arch>/bc``  or ``tmp/stamps/<arch>/bc`` ?
> I want confirmation to avoid another round ;)

Seems like a lot of places in the docs refer to build/tmp/work/..., so yes,
please include `build/` at the beginning.

Thank you,
Antonin

--
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com