[PATCH 0/3] yocto-check-layer: Faster BSP checks over many machines

[PATCH 0/3] yocto-check-layer: Faster BSP checks over many machines

[Otavio Salvador]

Checking a BSP layer over many machines is dominated by world signature
dumps: two per machine, plus four for the common tests. In CI the
machines are usually split into several runs, which repeat the common
tests and their dumps in each.

This series lets such runs select their tests and stops repeating a dump
whose result is already known:

- --run-tests, as in oe-test, so the common tests run once and the BSP
  tests in each group of machines;
- no initial (pre-layer) dump when test_signatures is not selected;
- reuse of a successful dump when the same one is asked for again,
  which test_world and test_machine_world do after test_signatures and
  test_machine_signatures. Failed dumps are not reused, so the world
  tests still report them.

A full run over N machines goes from 2N+4 dumps to N+3; a split run
does 3 dumps for the common tests and N for the BSP ones. Tested on
meta-freescale with two machines: 5 dumps instead of 8 with identical
results, and a recipe breaking world on one machine still fails
test_machine_world for that machine only.

This came out of the check-layer CI work for meta-freescale, which
currently does the same through a wrapper around yocto-check-layer.

Otavio Salvador (3):
  yocto-check-layer: Add --run-tests to select the tests to run
  yocto-check-layer: Skip initial signatures unless test_signatures runs
  checklayer: Reuse signatures of an identical successful world dump

 scripts/lib/checklayer/__init__.py | 13 ++++++++++++-
 scripts/yocto-check-layer          | 12 ++++++++----
 2 files changed, 20 insertions(+), 5 deletions(-)

-- 
2.54.0

[PATCH 1/3] yocto-check-layer: Add --run-tests to select the tests to run

[Otavio Salvador]

The common and distro tests ignore --machines, so a BSP layer checked in
several runs, one per group of machines, repeats them in each. Add
--run-tests, taking <module>[.<class>[.<name>]] as oe-test does, so the
common tests can run once and only the BSP ones in the others. A name
matching no test is an error rather than a run that checked nothing.

Tested on meta-freescale with "--run-tests common distro" and
"--run-tests bsp --machines imx8mm-ddr4-evk imx8mn-ddr4-evk".

Suggested-by: Luciano Dittgen <luciano.dittgen@ossystems.com.br>
AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
---
 scripts/yocto-check-layer | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/scripts/yocto-check-layer b/scripts/yocto-check-layer
index 8c8b4dbcad..e747451366 100755
--- a/scripts/yocto-check-layer
+++ b/scripts/yocto-check-layer
@@ -32,14 +32,14 @@ CASES_PATHS = [os.path.join(os.path.abspath(os.path.dirname(__file__)),
                 'lib', 'checklayer', 'cases')]
 logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
 
-def test_layer(td, layer, test_software_layer_signatures):
+def test_layer(td, layer, test_software_layer_signatures, run_tests):
     from checklayer.context import CheckLayerTestContext
     logger.info("Starting to analyze: %s" % layer['name'])
     logger.info('Distro: %s' % td['bbvars']["DISTRO"])
     logger.info("----------------------------------------------------------------------")
 
     tc = CheckLayerTestContext(td=td, logger=logger, layer=layer, test_software_layer_signatures=test_software_layer_signatures)
-    tc.loadTests(CASES_PATHS)
+    tc.loadTests(CASES_PATHS, modules=run_tests, modules_required=run_tests)
     return tc.runTests()
 
 def dump_layer_debug(layer):
@@ -70,6 +70,8 @@ def main():
                        help='check that software layers do not change signatures (on by default)')
     group.add_argument('--without-software-layer-signature-check', action='store_false', dest='test_software_layer_signatures',
                        help='disable signature checking for software layers')
+    parser.add_argument('--run-tests', nargs='+', metavar='TEST', default=[],
+            help='Tests to run, as <module>[.<class>[.<name>]] (default: all)')
     parser.add_argument('-n', '--no-auto', help='Disable auto layer discovery',
             action='store_true')
     parser.add_argument('-d', '--debug', help='Enable debug output',
@@ -228,7 +230,7 @@ def main():
             layers_tested = layers_tested + 1
             continue
 
-        result = test_layer(td, layer, args.test_software_layer_signatures)
+        result = test_layer(td, layer, args.test_software_layer_signatures, args.run_tests)
         results[layer['name']] = result
         results_status[layer['name']] = 'PASS' if results[layer['name']].wasSuccessful() else 'FAIL'
         layers_tested = layers_tested + 1
-- 
2.54.0

[PATCH 2/3] yocto-check-layer: Skip initial signatures unless test_signatures runs

[Otavio Salvador]

Only test_signatures compares against the world signatures dumped before
the layer is added, so a run without it, such as "--run-tests bsp", spent
a world parse on a dump it never used. Dump them only when --run-tests
selects common tests, as it does by default.

Tested on meta-freescale with "--run-tests bsp": no initial dump, same
results.

Suggested-by: Luciano Dittgen <luciano.dittgen@ossystems.com.br>
AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
---
 scripts/yocto-check-layer | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/yocto-check-layer b/scripts/yocto-check-layer
index e747451366..2fcb90acdc 100755
--- a/scripts/yocto-check-layer
+++ b/scripts/yocto-check-layer
@@ -214,7 +214,9 @@ def main():
         logger.info('Getting initial signatures ...')
         td['builddir'] = builddir
         try:
-            td['sigs'], td['tunetasks'] = get_signatures(td['builddir'])
+            # Only test_signatures, a common test, compares against them
+            if not args.run_tests or 'common' in [t.split('.')[0] for t in args.run_tests]:
+                td['sigs'], td['tunetasks'] = get_signatures(td['builddir'])
         except RuntimeError as e:
             logger.info(str(e))
             results[layer['name']] = None
-- 
2.54.0

[PATCH 3/3] checklayer: Reuse signatures of an identical successful world dump

[Otavio Salvador]

test_signatures and test_world dump the same world signatures, with and
without -k, and so do test_machine_signatures and test_machine_world for
each machine; when the dump succeeds, the second only repeats a world
parse. Keep the signatures of each successful dump, keyed by the layers in
bblayers.conf, MACHINE and extra variables, and return them when the
same dump is asked for again. A failed dump is not kept, so the world
tests still run it and report the failure.

Tested on meta-freescale with two machines: 5 world dumps instead of 8,
same results; a recipe breaking world on one machine still fails
test_machine_world for that machine only.

Suggested-by: Luciano Dittgen <luciano.dittgen@ossystems.com.br>
AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
---
 scripts/lib/checklayer/__init__.py | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/checklayer/__init__.py b/scripts/lib/checklayer/__init__.py
index aa19a8e2ae..8838a7df48 100644
--- a/scripts/lib/checklayer/__init__.py
+++ b/scripts/lib/checklayer/__init__.py
@@ -287,9 +287,18 @@ def check_command(error_msg, cmd, cwd=None):
         raise RuntimeError(msg)
     return output
 
+# The world signatures of each dump that succeeded: the tests ask for the
+# same dump with and without -k, which only differ when it fails
+signatures_cache = {}
+
 def get_signatures(builddir, failsafe=False, machine=None, extravars=None):
     import re
 
+    with open(os.path.join(builddir, 'conf', 'bblayers.conf')) as f:
+        key = (f.read(), machine, extravars)
+    if key in signatures_cache:
+        return signatures_cache[key]
+
     # some recipes needs to be excluded like meta-world-pkgdata
     # because a layer can add recipes to a world build so signature
     # will be change
@@ -320,7 +329,7 @@ def get_signatures(builddir, failsafe=False, machine=None, extravars=None):
             # in a world build lack some dependencies. There is a
             # separate test_machine_world_build which exposes the
             # failure.
-            pass
+            key = None
         else:
             raise
 
@@ -362,6 +371,8 @@ def get_signatures(builddir, failsafe=False, machine=None, extravars=None):
     if not sigs:
         raise RuntimeError('Can\'t load signatures from %s' % sigs_file)
 
+    if key:
+        signatures_cache[key] = (sigs, tune2tasks)
     return (sigs, tune2tasks)
 
 def get_depgraph(targets=['world'], failsafe=False):
-- 
2.54.0

Re: [PATCH 1/3] yocto-check-layer: Add --run-tests to select the tests to run

[Paul Barker]

On Sat, 2026-10-17 at 19:56 -0300, Otavio Salvador wrote:
> The common and distro tests ignore --machines, so a BSP layer checked in
> several runs, one per group of machines, repeats them in each. Add
> --run-tests, taking <module>[.<class>[.<name>]] as oe-test does, so the
> common tests can run once and only the BSP ones in the others. A name
> matching no test is an error rather than a run that checked nothing.
> 
> Tested on meta-freescale with "--run-tests common distro" and
> "--run-tests bsp --machines imx8mm-ddr4-evk imx8mn-ddr4-evk".
> 
> Suggested-by: Luciano Dittgen <luciano.dittgen@ossystems.com.br>
> AI-Generated: Uses Claude Code (Claude Opus 5.5)
> Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>

Hi Otavio,

Why would yocto-check-layer be executed several times to repeatedly test
a BSP layer? Why not just run it once with all the MACHINEs to be tested
listed?

I ask because one of the things on my list to add is automatic discovery
of MACHINEs in a BSP layer so you don't need to specify them all on the
command line. That would by design test all the MACHINEs in one go.

If we allow running a subset of tests in yocto-check-layer, we need to
update the docs [1] and/or the YP compatible application form [2] to
explicitly say that *all* test cases must pass for a layer to be YP
compatible.

[1]: https://docs.yoctoproject.org/current/dev-manual/layers.html#yocto-check-layer-script
[2]: https://www.yoctoproject.org/compatible-registration/

Best regards,

-- 
Paul Barker

Re: [PATCH 3/3] checklayer: Reuse signatures of an identical successful world dump

[Paul Barker]

On Sat, 2026-10-17 at 19:56 -0300, Otavio Salvador wrote:
> test_signatures and test_world dump the same world signatures, with and
> without -k, and so do test_machine_signatures and test_machine_world for
> each machine; when the dump succeeds, the second only repeats a world
> parse. Keep the signatures of each successful dump, keyed by the layers in
> bblayers.conf, MACHINE and extra variables, and return them when the
> same dump is asked for again. A failed dump is not kept, so the world
> tests still run it and report the failure.
> 
> Tested on meta-freescale with two machines: 5 world dumps instead of 8,
> same results; a recipe breaking world on one machine still fails
> test_machine_world for that machine only.
> 
> Suggested-by: Luciano Dittgen <luciano.dittgen@ossystems.com.br>
> AI-Generated: Uses Claude Code (Claude Opus 5.5)
> Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
> ---
>  scripts/lib/checklayer/__init__.py | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/scripts/lib/checklayer/__init__.py b/scripts/lib/checklayer/__init__.py
> index aa19a8e2ae..8838a7df48 100644
> --- a/scripts/lib/checklayer/__init__.py
> +++ b/scripts/lib/checklayer/__init__.py
> @@ -287,9 +287,18 @@ def check_command(error_msg, cmd, cwd=None):
>          raise RuntimeError(msg)
>      return output
>  
> +# The world signatures of each dump that succeeded: the tests ask for the
> +# same dump with and without -k, which only differ when it fails
> +signatures_cache = {}
> +
>  def get_signatures(builddir, failsafe=False, machine=None, extravars=None):
>      import re
>  
> +    with open(os.path.join(builddir, 'conf', 'bblayers.conf')) as f:
> +        key = (f.read(), machine, extravars)
> +    if key in signatures_cache:
> +        return signatures_cache[key]
> +
>      # some recipes needs to be excluded like meta-world-pkgdata
>      # because a layer can add recipes to a world build so signature
>      # will be change
> @@ -320,7 +329,7 @@ def get_signatures(builddir, failsafe=False, machine=None, extravars=None):
>              # in a world build lack some dependencies. There is a
>              # separate test_machine_world_build which exposes the
>              # failure.
> -            pass
> +            key = None
>          else:
>              raise
>  
> @@ -362,6 +371,8 @@ def get_signatures(builddir, failsafe=False, machine=None, extravars=None):
>      if not sigs:
>          raise RuntimeError('Can\'t load signatures from %s' % sigs_file)
>  
> +    if key:
> +        signatures_cache[key] = (sigs, tune2tasks)
>      return (sigs, tune2tasks)
>  
>  def get_depgraph(targets=['world'], failsafe=False):

I can see this caching causing issues in the future if we change the
test cases but forget to update the way we determine `key` above.

I would prefer to modify the tests to call get_signatures() once,
possibly in a setUp() function for the CommonCheckLayer class, storing
the result so it can be used in multiple test cases. That may need a
change to get_signatures() so it always runs bitbake with `-k` and
returns the success/fail state to the caller. Then test_world() would
check the stored success/fail state and test_signatures() would ignore
it.

Best regards,

-- 
Paul Barker

Re: [PATCH 3/3] checklayer: Reuse signatures of an identical successful world dump

[Paul Barker]

On Mon, 2026-10-19 at 10:04 +0100, Paul Barker wrote:
> On Sat, 2026-10-17 at 19:56 -0300, Otavio Salvador wrote:
> > test_signatures and test_world dump the same world signatures, with and
> > without -k, and so do test_machine_signatures and test_machine_world for
> > each machine; when the dump succeeds, the second only repeats a world
> > parse. Keep the signatures of each successful dump, keyed by the layers in
> > bblayers.conf, MACHINE and extra variables, and return them when the
> > same dump is asked for again. A failed dump is not kept, so the world
> > tests still run it and report the failure.
> > 
> > Tested on meta-freescale with two machines: 5 world dumps instead of 8,
> > same results; a recipe breaking world on one machine still fails
> > test_machine_world for that machine only.
> > 
> > Suggested-by: Luciano Dittgen <luciano.dittgen@ossystems.com.br>
> > AI-Generated: Uses Claude Code (Claude Opus 5.5)
> > Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
> > ---
> >  scripts/lib/checklayer/__init__.py | 13 ++++++++++++-
> >  1 file changed, 12 insertions(+), 1 deletion(-)
> > 
> > diff --git a/scripts/lib/checklayer/__init__.py b/scripts/lib/checklayer/__init__.py
> > index aa19a8e2ae..8838a7df48 100644
> > --- a/scripts/lib/checklayer/__init__.py
> > +++ b/scripts/lib/checklayer/__init__.py
> > @@ -287,9 +287,18 @@ def check_command(error_msg, cmd, cwd=None):
> >          raise RuntimeError(msg)
> >      return output
> >  
> > +# The world signatures of each dump that succeeded: the tests ask for the
> > +# same dump with and without -k, which only differ when it fails
> > +signatures_cache = {}
> > +
> >  def get_signatures(builddir, failsafe=False, machine=None, extravars=None):
> >      import re
> >  
> > +    with open(os.path.join(builddir, 'conf', 'bblayers.conf')) as f:
> > +        key = (f.read(), machine, extravars)
> > +    if key in signatures_cache:
> > +        return signatures_cache[key]
> > +
> >      # some recipes needs to be excluded like meta-world-pkgdata
> >      # because a layer can add recipes to a world build so signature
> >      # will be change
> > @@ -320,7 +329,7 @@ def get_signatures(builddir, failsafe=False, machine=None, extravars=None):
> >              # in a world build lack some dependencies. There is a
> >              # separate test_machine_world_build which exposes the
> >              # failure.
> > -            pass
> > +            key = None
> >          else:
> >              raise
> >  
> > @@ -362,6 +371,8 @@ def get_signatures(builddir, failsafe=False, machine=None, extravars=None):
> >      if not sigs:
> >          raise RuntimeError('Can\'t load signatures from %s' % sigs_file)
> >  
> > +    if key:
> > +        signatures_cache[key] = (sigs, tune2tasks)
> >      return (sigs, tune2tasks)
> >  
> >  def get_depgraph(targets=['world'], failsafe=False):
> 
> I can see this caching causing issues in the future if we change the
> test cases but forget to update the way we determine `key` above.
> 
> I would prefer to modify the tests to call get_signatures() once,
> possibly in a setUp() function for the CommonCheckLayer class, storing
> the result so it can be used in multiple test cases. That may need a
> change to get_signatures() so it always runs bitbake with `-k` and
> returns the success/fail state to the caller. Then test_world() would
> check the stored success/fail state and test_signatures() would ignore
> it.

Sorry, I mean setUpClass(), not setUp().

Best regards,

-- 
Paul Barker

[PATCH v2 0/3] yocto-check-layer: Faster BSP checks over many machines

[Otavio Salvador]

Checking a BSP layer over many machines is dominated by world signature
dumps: two per machine, plus four for the common tests. In CI the
machines are usually split into several runs, which repeat the common
tests and their dumps in each.

This series lets such runs select their tests and dumps each
configuration once:

- --run-tests, as in oe-test, so the common tests run once and the BSP
  tests in each group of machines;
- no initial (pre-layer) dump when test_signatures is not selected;
- get_signatures() always runs with -k and returns the error with the
  signatures, so the common tests, and each machine of the BSP tests,
  dump once in setUpClass() and the world and signature tests share it.

A full run over N machines goes from 2N+4 dumps to N+3; a split run
does 3 dumps for the common tests and N for the BSP ones.

Changes in v2:
- Patch 3 replaces the signature cache in get_signatures() with one dump
  per configuration in setUpClass(), as suggested by Paul Barker. Failed
  dumps are no longer run a second time either.
- Patches 1 and 2 are unchanged.

Tested on meta-freescale against unpatched OE-core and against the
wrapper proposed for its CI, which does the same from outside
yocto-check-layer: with two and eight machines, a recipe breaking world
on one machine, and a recipe that does not parse. Every test gives the
same result in all of them, with the same machines reported as broken.
World dumps per run:

                                  unpatched  wrapper  v2
  full run, 2 machines                8          5      5
  BSP tests, 2 machines, 1 broken     -          3      2
  BSP tests, 8 machines, 1 broken     -          9      8
  parse error, 2 machines             7          7      5

  (-: unpatched yocto-check-layer cannot run the BSP tests alone)

The only visible change is in world failures: the dump now uses -k, so
it lists all the errors of the failing machine instead of the first one.

v1: https://lists.openembedded.org/g/openembedded-core/topic/121810615

Otavio Salvador (3):
  yocto-check-layer: Add --run-tests to select the tests to run
  yocto-check-layer: Skip initial signatures unless test_signatures runs
  checklayer: Share each world signature dump between the tests needing
    it

 scripts/lib/checklayer/__init__.py     | 28 +++++++++++++-------------
 scripts/lib/checklayer/cases/bsp.py    | 15 +++++++++-----
 scripts/lib/checklayer/cases/common.py | 20 +++++++++++-------
 scripts/yocto-check-layer              | 14 +++++++++----
 4 files changed, 47 insertions(+), 30 deletions(-)

-- 
2.54.0

[PATCH v2 1/3] yocto-check-layer: Add --run-tests to select the tests to run

[Otavio Salvador]

The common and distro tests ignore --machines, so a BSP layer checked in
several runs, one per group of machines, repeats them in each. Add
--run-tests, taking <module>[.<class>[.<name>]] as oe-test does, so the
common tests can run once and only the BSP ones in the others. A name
matching no test is an error rather than a run that checked nothing.

Tested on meta-freescale with "--run-tests common distro" and
"--run-tests bsp --machines imx8mm-ddr4-evk imx8mn-ddr4-evk".

Suggested-by: Luciano Dittgen <luciano.dittgen@ossystems.com.br>
AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
---
 scripts/yocto-check-layer | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/scripts/yocto-check-layer b/scripts/yocto-check-layer
index 8c8b4dbcad..e747451366 100755
--- a/scripts/yocto-check-layer
+++ b/scripts/yocto-check-layer
@@ -32,14 +32,14 @@ CASES_PATHS = [os.path.join(os.path.abspath(os.path.dirname(__file__)),
                 'lib', 'checklayer', 'cases')]
 logger = scriptutils.logger_create(PROGNAME, stream=sys.stdout)
 
-def test_layer(td, layer, test_software_layer_signatures):
+def test_layer(td, layer, test_software_layer_signatures, run_tests):
     from checklayer.context import CheckLayerTestContext
     logger.info("Starting to analyze: %s" % layer['name'])
     logger.info('Distro: %s' % td['bbvars']["DISTRO"])
     logger.info("----------------------------------------------------------------------")
 
     tc = CheckLayerTestContext(td=td, logger=logger, layer=layer, test_software_layer_signatures=test_software_layer_signatures)
-    tc.loadTests(CASES_PATHS)
+    tc.loadTests(CASES_PATHS, modules=run_tests, modules_required=run_tests)
     return tc.runTests()
 
 def dump_layer_debug(layer):
@@ -70,6 +70,8 @@ def main():
                        help='check that software layers do not change signatures (on by default)')
     group.add_argument('--without-software-layer-signature-check', action='store_false', dest='test_software_layer_signatures',
                        help='disable signature checking for software layers')
+    parser.add_argument('--run-tests', nargs='+', metavar='TEST', default=[],
+            help='Tests to run, as <module>[.<class>[.<name>]] (default: all)')
     parser.add_argument('-n', '--no-auto', help='Disable auto layer discovery',
             action='store_true')
     parser.add_argument('-d', '--debug', help='Enable debug output',
@@ -228,7 +230,7 @@ def main():
             layers_tested = layers_tested + 1
             continue
 
-        result = test_layer(td, layer, args.test_software_layer_signatures)
+        result = test_layer(td, layer, args.test_software_layer_signatures, args.run_tests)
         results[layer['name']] = result
         results_status[layer['name']] = 'PASS' if results[layer['name']].wasSuccessful() else 'FAIL'
         layers_tested = layers_tested + 1
-- 
2.54.0

[PATCH v2 2/3] yocto-check-layer: Skip initial signatures unless test_signatures runs

[Otavio Salvador]

Only test_signatures compares against the world signatures dumped before
the layer is added, so a run without it, such as "--run-tests bsp", spent
a world parse on a dump it never used. Dump them only when --run-tests
selects common tests, as it does by default.

Tested on meta-freescale with "--run-tests bsp": no initial dump, same
results.

Suggested-by: Luciano Dittgen <luciano.dittgen@ossystems.com.br>
AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
---
 scripts/yocto-check-layer | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/yocto-check-layer b/scripts/yocto-check-layer
index e747451366..2fcb90acdc 100755
--- a/scripts/yocto-check-layer
+++ b/scripts/yocto-check-layer
@@ -214,7 +214,9 @@ def main():
         logger.info('Getting initial signatures ...')
         td['builddir'] = builddir
         try:
-            td['sigs'], td['tunetasks'] = get_signatures(td['builddir'])
+            # Only test_signatures, a common test, compares against them
+            if not args.run_tests or 'common' in [t.split('.')[0] for t in args.run_tests]:
+                td['sigs'], td['tunetasks'] = get_signatures(td['builddir'])
         except RuntimeError as e:
             logger.info(str(e))
             results[layer['name']] = None
-- 
2.54.0

[PATCH v2 3/3] checklayer: Share each world signature dump between the tests needing it

[Otavio Salvador]

test_signatures and test_world dump the same world signatures, with and
without -k, and so do test_machine_signatures and test_machine_world for
each machine; when the dump succeeds, the second only repeats a world
parse. Make get_signatures() always run bitbake with -k and return the
error along with the signatures, and dump once in setUpClass(): for the
common tests, and per machine for the BSP ones. test_world and
test_machine_world check the stored error, the signature tests use the
stored signatures.

Tested on meta-freescale with two machines: 5 world dumps instead of 8,
same results; a recipe breaking world on one machine fails
test_machine_world for that machine only, now listing all its errors.

Suggested-by: Luciano Dittgen <luciano.dittgen@ossystems.com.br>
Suggested-by: Paul Barker <paul@pbarker.dev>
AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
---
Changes in v2:
- Replace the cache in get_signatures(), keyed on bblayers.conf, MACHINE
  and the extra variables, with one dump in setUpClass(): once for the
  common tests and once per machine for the BSP tests (Paul Barker).
- get_signatures() always runs with -k and returns the error, or None,
  with the signatures, instead of taking failsafe. A failed dump is no
  longer run a second time without -k.

 scripts/lib/checklayer/__init__.py     | 28 +++++++++++++-------------
 scripts/lib/checklayer/cases/bsp.py    | 15 +++++++++-----
 scripts/lib/checklayer/cases/common.py | 20 +++++++++++-------
 scripts/yocto-check-layer              |  4 +++-
 4 files changed, 40 insertions(+), 27 deletions(-)

diff --git a/scripts/lib/checklayer/__init__.py b/scripts/lib/checklayer/__init__.py
index aa19a8e2ae..9d79ae9cd0 100644
--- a/scripts/lib/checklayer/__init__.py
+++ b/scripts/lib/checklayer/__init__.py
@@ -287,7 +287,12 @@ def check_command(error_msg, cmd, cwd=None):
         raise RuntimeError(msg)
     return output
 
-def get_signatures(builddir, failsafe=False, machine=None, extravars=None):
+def get_signatures(builddir, machine=None, extravars=None):
+    '''
+    Returns the world signatures, the tasks of each tune and None, or the
+    error when some recipes failed, with the signatures of the others
+    (None when none were written).
+    '''
     import re
 
     # some recipes needs to be excluded like meta-world-pkgdata
@@ -304,25 +309,20 @@ def get_signatures(builddir, failsafe=False, machine=None, extravars=None):
         cmd += ' '
     if machine:
         cmd += 'MACHINE=%s ' % machine
-    cmd += 'bitbake '
-    if failsafe:
-        cmd += '-k '
-    cmd += '-S lockedsigs world'
+    cmd += 'bitbake -k -S lockedsigs world'
     sigs_file = os.path.join(builddir, 'locked-sigs.inc')
     if os.path.exists(sigs_file):
         os.unlink(sigs_file)
+    error = None
     try:
         check_command('Generating signatures failed. This might be due to some parse error and/or general layer incompatibilities.',
                       cmd, builddir)
     except RuntimeError as ex:
-        if failsafe and os.path.exists(sigs_file):
-            # Ignore the error here. Most likely some recipes active
-            # in a world build lack some dependencies. There is a
-            # separate test_machine_world_build which exposes the
-            # failure.
-            pass
-        else:
-            raise
+        # Most likely some recipes active in a world build lack some
+        # dependencies: the caller decides whether that is fatal.
+        error = str(ex)
+        if not os.path.exists(sigs_file):
+            return (None, None, error)
 
     sig_regex = re.compile(r"^(?P<task>[^:]*:[^:]*):(?P<hash>[^:]*) .$")
     assignment_regex = re.compile(r"^\s*(?P<var>\S+)\s*\+?=")
@@ -362,7 +362,7 @@ def get_signatures(builddir, failsafe=False, machine=None, extravars=None):
     if not sigs:
         raise RuntimeError('Can\'t load signatures from %s' % sigs_file)
 
-    return (sigs, tune2tasks)
+    return (sigs, tune2tasks, error)
 
 def get_depgraph(targets=['world'], failsafe=False):
     '''
diff --git a/scripts/lib/checklayer/cases/bsp.py b/scripts/lib/checklayer/cases/bsp.py
index b76163fb56..6bf04b5b00 100644
--- a/scripts/lib/checklayer/cases/bsp.py
+++ b/scripts/lib/checklayer/cases/bsp.py
@@ -14,6 +14,10 @@ class BSPCheckLayer(OECheckLayerTestCase):
         if self.tc.layer['type'] not in (LayerType.BSP, LayerType.CORE):
             raise unittest.SkipTest("BSPCheckLayer: Layer %s isn't BSP one." %\
                 self.tc.layer['name'])
+        # test_machine_world and test_machine_signatures share one world
+        # dump per machine
+        self.machine_sigs = {machine: get_signatures(self.td['builddir'], machine=machine)
+                             for machine in self.td['machines'] or []}
 
     def test_bsp_defines_machines(self):
         self.assertTrue(self.tc.layer['conf']['machines'], 
@@ -45,10 +49,9 @@ class BSPCheckLayer(OECheckLayerTestCase):
         msg = []
         for machine in self.td['machines']:
             # In contrast to test_machine_signatures() below, errors are fatal here.
-            try:
-                get_signatures(self.td['builddir'], failsafe=False, machine=machine)
-            except RuntimeError as ex:
-                msg.append(str(ex))
+            error = self.machine_sigs[machine][2]
+            if error:
+                msg.append(error)
         if msg:
             msg.insert(0, 'The following machines broke a world build:')
             self.fail('\n'.join(msg))
@@ -80,7 +83,9 @@ class BSPCheckLayer(OECheckLayerTestCase):
         tunes = {}
         # Preserve ordering of machines as chosen by the user.
         for machine in self.td['machines']:
-            curr_sigs, tune2tasks = get_signatures(self.td['builddir'], failsafe=True, machine=machine)
+            curr_sigs, tune2tasks, error = self.machine_sigs[machine]
+            if curr_sigs is None:
+                raise RuntimeError(error)
             # Invert the tune -> [tasks] mapping.
             tasks2tune = {}
             for tune, tasks in tune2tasks.items():
diff --git a/scripts/lib/checklayer/cases/common.py b/scripts/lib/checklayer/cases/common.py
index d20559e163..812bb6e637 100644
--- a/scripts/lib/checklayer/cases/common.py
+++ b/scripts/lib/checklayer/cases/common.py
@@ -11,6 +11,11 @@ from checklayer import get_signatures, LayerType, check_command, compare_signatu
 from checklayer.case import OECheckLayerTestCase
 
 class CommonCheckLayer(OECheckLayerTestCase):
+    @classmethod
+    def setUpClass(cls):
+        # test_world and test_signatures share one world dump
+        cls.world_sigs, _, cls.world_error = get_signatures(cls.td['builddir'])
+
     def test_readme(self):
         """
         Test that the layer contains a valid README file and that is meets
@@ -98,7 +103,8 @@ class CommonCheckLayer(OECheckLayerTestCase):
         are not actually buildable, so here we fail when "bitbake -S none world"
         fails.
         '''
-        get_signatures(self.td['builddir'], failsafe=False)
+        if self.world_error:
+            raise RuntimeError(self.world_error)
 
     def test_world_inherit_class(self):
         '''
@@ -106,10 +112,9 @@ class CommonCheckLayer(OECheckLayerTestCase):
         class, which can do additional per-recipe test cases.
         '''
         msg = []
-        try:
-            get_signatures(self.td['builddir'], failsafe=False, machine=None, extravars='BB_ENV_PASSTHROUGH_ADDITIONS="$BB_ENV_PASSTHROUGH_ADDITIONS INHERIT" INHERIT="yocto-check-layer"')
-        except RuntimeError as ex:
-            msg.append(str(ex))
+        _, _, error = get_signatures(self.td['builddir'], machine=None, extravars='BB_ENV_PASSTHROUGH_ADDITIONS="$BB_ENV_PASSTHROUGH_ADDITIONS INHERIT" INHERIT="yocto-check-layer"')
+        if error:
+            msg.append(error)
         if msg:
             msg.insert(0, 'Layer %s failed additional checks from yocto-check-layer.bbclass\nSee below log for specific recipe parsing errors:\n' % \
                 self.tc.layer['name'])
@@ -135,8 +140,9 @@ class CommonCheckLayer(OECheckLayerTestCase):
             raise unittest.SkipTest("Not testing for signature changes in a software layer %s." \
                      % self.tc.layer['name'])
 
-        curr_sigs, _ = get_signatures(self.td['builddir'], failsafe=True)
-        msg = compare_signatures(self.td['sigs'], curr_sigs)
+        if self.world_sigs is None:
+            raise RuntimeError(self.world_error)
+        msg = compare_signatures(self.td['sigs'], self.world_sigs)
         if msg is not None:
             self.fail('Adding layer %s changed signatures.\n%s' % (self.tc.layer['name'], msg))
 
diff --git a/scripts/yocto-check-layer b/scripts/yocto-check-layer
index 2fcb90acdc..ed8841d1f3 100755
--- a/scripts/yocto-check-layer
+++ b/scripts/yocto-check-layer
@@ -216,7 +216,9 @@ def main():
         try:
             # Only test_signatures, a common test, compares against them
             if not args.run_tests or 'common' in [t.split('.')[0] for t in args.run_tests]:
-                td['sigs'], td['tunetasks'] = get_signatures(td['builddir'])
+                td['sigs'], td['tunetasks'], error = get_signatures(td['builddir'])
+                if error:
+                    raise RuntimeError(error)
         except RuntimeError as e:
             logger.info(str(e))
             results[layer['name']] = None
-- 
2.54.0

Re: [OE-core] [PATCH v2 3/3] checklayer: Share each world signature dump between the tests needing it

[Otavio Salvador]

Hi Mathieu,

I saw this patch was reverted in master-next to test Ricardo's
"checklayer: run bitbake once per machine for the machine tests",
since the two conflict. Ricardo replied that his patch solves the same
issue and can be dropped in favour of this one:

https://patchwork.yoctoproject.org/comment/42922/

master-next still has the revert, with Ricardo's patch on top. Could
you drop both, so this patch goes back in with 1/3 and 2/3? It covers
the machine tests as his does, and the common tests too: test_world
and test_signatures share one dump.

If you'd rather keep Ricardo's version, I have a follow-up on top of
it that moves the common tests to his cache, and I can send that
instead.

Thanks,
Otavio

Re: [OE-core] [PATCH v2 3/3] checklayer: Share each world signature dump between the tests needing it

[Mathieu Dubois-Briand]

On Thu Oct 29, 2026 at 1:20 PM CET, Otavio Salvador wrote:
> Hi Mathieu,
>
> I saw this patch was reverted in master-next to test Ricardo's
> "checklayer: run bitbake once per machine for the machine tests",
> since the two conflict. Ricardo replied that his patch solves the same
> issue and can be dropped in favour of this one:
>
> https://patchwork.yoctoproject.org/comment/42922/
>
> master-next still has the revert, with Ricardo's patch on top. Could
> you drop both, so this patch goes back in with 1/3 and 2/3? It covers
> the machine tests as his does, and the common tests too: test_world
> and test_signatures share one dump.
>
> If you'd rather keep Ricardo's version, I have a follow-up on top of
> it that moves the common tests to his cache, and I can send that
> instead.
>
> Thanks,
> Otavio

Hi Otavio,

This was discussed during yesterday review meeting: we believe we should
proceed with your version of the changes, possibly with some
improvements. So yes, I dropped Ricardo patch.

Thanks,
Mathieu

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com