[PATCH v2 0/8] cooker/tinfoil: fix -b bbappend handling and add single-task prepared-task API

[PATCH v2 0/8] cooker/tinfoil: fix -b bbappend handling and add single-task prepared-task API

[AdrianF]

From: Adrian Freihofer <adrian.freihofer@siemens.com>

Hi Richard,

This v2 should answer your question:
  doesn't this only add limited collections data so whilst it fixes -b,
  it potentially corrupts the cache for the non -b cases?
  https://lists.openembedded.org/g/bitbake-devel/message/19957

The test case test_run_prepared_task_recipecache_self_heals should
prove that this works as expected.

The commits going into bitbake do not change in comparison to v1.
Only the test changed.

Regards,
Adrian

Adrian Freihofer (8):
  cooker: fix bitbake -b silently ignoring bbappends
  tests/cooker: add a shared bitbake-subprocess test base class
  tests/cooker: add a bitbake -b bbappend test
  command: fix setConfig coercing bool config values to truthy strings
  cooker: add a buildFile mode that runs a single task
  tinfoil: add a prepared task runner
  tests/cooker: add TinfoilTests for run_prepared_task
  parse/ast: skip empty BBPATH segments in include_all

 lib/bb/command.py      |   8 +-
 lib/bb/cooker.py       |  14 +-
 lib/bb/parse/ast.py    |   4 +
 lib/bb/tests/cooker.py | 342 ++++++++++++++++++++++++++++++++++++++++-
 lib/bb/tinfoil.py      |  23 ++-
 5 files changed, 383 insertions(+), 8 deletions(-)

-- 
2.55.0

[PATCH v2 1/8] cooker: fix bitbake -b silently ignoring bbappends

[AdrianF]

From: Adrian Freihofer <adrian.freihofer@siemens.com>

"bitbake -b <recipe.bb>" builds the recipe without applying any of its
.bbappend files.

buildFileInternal() resolves appends via
self.collections[mc].get_file_appends(fn), but self.collections[mc] is
only ever filled in by collect_bbfiles(), called from updateCache() -
a path -b deliberately skips. matchFiles(), the one -b-path function
that does call collect_bbfiles(), built a fresh CookerCollectFiles into
a throwaway local instead of self.collections[mc], so the append list
stayed empty (or, on a memory-resident server, stale from the last
full parse - e.g. missing a devtool/externalsrc workspace .bbappend
added since). Nothing warns that the built metadata differs from disk.

Make matchFiles() refresh self.collections[mc] itself so the later
append lookup for the same fn sees the same fresh collection.

AI-Generated: Uses GitHub Copilot

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 lib/bb/cooker.py | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 4b6ba3196..108551a60 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -1322,8 +1322,10 @@ You can also remove the BB_HASHSERVE_UPSTREAM setting, but this may result in si
         if bf.startswith("/") or bf.startswith("../"):
             bf = os.path.abspath(bf)
 
-        collections = {mc: CookerCollectFiles(self.bbfile_config_priorities, mc)}
-        filelist, masked, searchdirs = collections[mc].collect_bbfiles(self.databuilder.mcdata[mc], self.databuilder.mcdata[mc])
+        # The only place the "bitbake -b" path fills in the bbappends which
+        # buildFileInternal() then reads back from self.collections[mc].
+        self.collections[mc] = CookerCollectFiles(self.bbfile_config_priorities, mc)
+        filelist, masked, searchdirs = self.collections[mc].collect_bbfiles(self.databuilder.mcdata[mc], self.databuilder.mcdata[mc])
         try:
             os.stat(bf)
             bf = os.path.abspath(bf)
-- 
2.55.0

[PATCH v2 2/8] tests/cooker: add a shared bitbake-subprocess test base class

[AdrianF]

From: Adrian Freihofer <adrian.freihofer@siemens.com>

Tests that run bitbake or tinfoil in a subprocess against a temporary
build directory can leave behind a memory-resident bitbake server (and,
if BB_HASHSERVE=auto, a hashserv) rooted at TOPDIR. That server must
release the directory before the caller's TemporaryDirectory context
manager can safely remove it, or cleanup can race a still-running
server holding files open there.

Add _BitbakeSubprocessTestCase with _run_subprocess()/_shutdown() to
provide this consistently, so the tests added on top of it in the next
commits don't each have to hand-roll the subprocess/wait boilerplate -
and, more importantly, don't get a chance to forget the wait.

AI-Generated: Uses GitHub Copilot

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 lib/bb/tests/cooker.py | 52 +++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 51 insertions(+), 1 deletion(-)

diff --git a/lib/bb/tests/cooker.py b/lib/bb/tests/cooker.py
index 9e524ae34..c49375ed8 100644
--- a/lib/bb/tests/cooker.py
+++ b/lib/bb/tests/cooker.py
@@ -7,12 +7,62 @@
 #
 
 import unittest
+import contextlib
 import os
+import subprocess
+import sys
+import tempfile
+import time
 import bb, bb.cooker
 import re
 import logging
 
-# Cooker tests
+
+class _BitbakeSubprocessTestCase(unittest.TestCase):
+    """Common helpers for tests that run bitbake/tinfoil in a subprocess.
+
+    Shared because every such subprocess can start a memory-resident bitbake
+    server (and, if BB_HASHSERVE=auto, a hashserv) rooted at TOPDIR, and both
+    must release that directory before the caller's TemporaryDirectory can be
+    safely removed.
+    """
+
+    def _run_subprocess(self, cmd, env, cwd):
+        proc = subprocess.run(
+            cmd,
+            env=env,
+            stdout=subprocess.PIPE,
+            stderr=subprocess.STDOUT,
+            universal_newlines=True,
+            cwd=cwd,
+        )
+        if proc.returncode:
+            self.fail('%s failed: %s' % (cmd, proc.stdout))
+        return proc.stdout
+
+    def _shutdown(self, builddir):
+        """Wait for the bitbake server and hashserv to release builddir.
+
+        Must run before the caller's TemporaryDirectory is removed, so it
+        cannot be a tearDown().
+        """
+        deadline = time.monotonic() + 30
+        while time.monotonic() < deadline:
+            if not any(os.path.exists(os.path.join(builddir, p))
+                       for p in ('hashserve.sock', 'bitbake.lock')):
+                return
+            time.sleep(0.5)
+
+    @contextlib.contextmanager
+    def _build_dir(self, prefix='tinfoiltest'):
+        """TemporaryDirectory that also waits out _shutdown() before removal."""
+        with tempfile.TemporaryDirectory(prefix=prefix) as builddir:
+            try:
+                yield builddir
+            finally:
+                self._shutdown(builddir)
+
+
 class CookerTest(unittest.TestCase):
     def setUp(self):
         # At least one variable needs to be set
-- 
2.55.0

[PATCH v2 3/8] tests/cooker: add a bitbake -b bbappend test

[AdrianF]

From: Adrian Freihofer <adrian.freihofer@siemens.com>

Add BuildFileTest.test_buildfile_applies_bbappends() to cover the
buildfile ("bitbake -b") mode, which had no test coverage at all.

The test writes a recipe plus a matching .bbappend into a temporary
directory, points EXTRA_BBFILES at both, and runs a task that records
the value of a variable the bbappend overrides. It then asserts the
bbappend's value was the one in effect.

It uses the parse-tests BBPATH rather than runqueue-tests because its
bitbake.conf already globs *.bbappend and the recipe needs no task
scaffolding beyond its own.

Without the preceding fix the recorded value is the recipe's default,
i.e. -b built the recipe with the bbappend silently dropped.

Built on _BitbakeSubprocessTestCase so the "bitbake -b" subprocess's
server is properly waited on before the TemporaryDirectory cleanup
runs.

AI-Generated: Uses GitHub Copilot

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 lib/bb/tests/cooker.py | 46 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/lib/bb/tests/cooker.py b/lib/bb/tests/cooker.py
index c49375ed8..c32694cc5 100644
--- a/lib/bb/tests/cooker.py
+++ b/lib/bb/tests/cooker.py
@@ -119,3 +119,49 @@ class CookerTest(unittest.TestCase):
         expected = []
 
         self.assertEqual(log_handler.logdata, expected)
+
+
+class BuildFileTest(_BitbakeSubprocessTestCase):
+    """Tests for the buildfile ("bitbake -b") mode."""
+
+    # parse-tests BBPATH: minimal bitbake.conf whose BBFILES honours
+    # EXTRA_BBFILES and already includes *.bbappend
+    _parsetests = os.path.realpath(os.path.join(os.path.dirname(__file__), "parse-tests"))
+
+    recipe = """\
+MARKER ??= "no-bbappend"
+python do_marker() {
+    with open(d.expand("${TOPDIR}/marker.log"), "w") as f:
+        f.write(d.getVar("MARKER"))
+}
+addtask marker
+"""
+
+    bbappend = 'MARKER = "bbappend-applied"\n'
+
+    def test_buildfile_applies_bbappends(self):
+        """bitbake -b must build the recipe with its bbappends applied.
+
+        buildFileInternal() looks the appends up in self.collections[mc], which
+        on the -b path is only ever populated by matchFiles().
+        """
+        with tempfile.TemporaryDirectory(prefix="buildfilerecipes") as recipes, \
+             self._build_dir(prefix="buildfiletest") as builddir:
+            recipe = os.path.join(recipes, "appendtest.bb")
+            with open(recipe, "w") as f:
+                f.write(self.recipe)
+            with open(os.path.join(recipes, "appendtest.bbappend"), "w") as f:
+                f.write(self.bbappend)
+
+            env = os.environ.copy()
+            env["BBPATH"] = self._parsetests
+            env["BB_ENV_PASSTHROUGH_ADDITIONS"] = "TOPDIR EXTRA_BBFILES"
+            env["TOPDIR"] = builddir
+            env["EXTRA_BBFILES"] = "%s/*.bb %s/*.bbappend" % (recipes, recipes)
+
+            cmd = ["bitbake", "-b", recipe, "-c", "marker"]
+            self._run_subprocess(cmd, env, builddir)
+
+            with open(os.path.join(builddir, "marker.log")) as f:
+                self.assertEqual(f.read(), "bbappend-applied",
+                                 "bitbake -b did not apply the recipe's bbappend")
-- 
2.55.0

[PATCH v2 4/8] command: fix setConfig coercing bool config values to truthy strings

[AdrianF]

From: Adrian Freihofer <adrian.freihofer@siemens.com>

CommandsSync.setConfig() unconditionally stringified the value with
str(params[1]) before assigning it to the cooker configuration
attribute. This breaks boolean values, since str(True) and str(False)
are both non-empty and therefore both truthy.

That makes it impossible to turn a boolean option back off over the
command interface: setting 'force' to False leaves configuration.force
holding the truthy string "False", so it stays effectively enabled for
the rest of the bitbake server session and spuriously invalidates tasks
in later, unrelated builds sharing that session.

Preserve the caller's original type instead of coercing to str. The
only other caller (cookerdata.py) already passes a plain string, so
this does not change behavior for it.

AI-Generated: Uses GitHub Copilot

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 lib/bb/command.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/command.py b/lib/bb/command.py
index 59a979ee9..b57c5d4a3 100644
--- a/lib/bb/command.py
+++ b/lib/bb/command.py
@@ -228,7 +228,7 @@ class CommandsSync:
         Set the value of variable in configuration
         """
         varname = params[0]
-        value = str(params[1])
+        value = params[1]
         setattr(command.cooker.configuration, varname, value)
 
     def enableDataTracking(self, command, params):
-- 
2.55.0

[PATCH v2 5/8] cooker: add a buildFile mode that runs a single task

[AdrianF]

From: Adrian Freihofer <adrian.freihofer@siemens.com>

devtool ide-sdk lets the IDE build directly (e.g. via cmake or meson,
outside of bitbake). do_install still needs bitbake: it needs pseudo,
and it usually does more than the underlying build tool's own install
step (e.g. `cmake --build --target install`) would - packaging-related
fixups the recipe or its classes add on top. It must run as only that
one already-prepared task against the source the developer just edited
- not the whole recipe: predecessors, if they ran again, would
rebuild/overwrite the very output the caller is about to inspect or has
already staged.

Add a taskonly argument to buildFileInternal(), plumbed through the
buildFile command, that also clears the intra-recipe task parents,
leaving the requested task as the runqueue's only entry. Default
unchanged, so "bitbake -b" behaviour is unaffected.

Note that for example:
- buildFileInternal() cannot do this: it drops external dependencies but
  keeps intra-recipe task ordering ('addtask X after Y'), so requesting
  do_install on an unbuilt recipe still pulls in do_fetch, do_unpack,
  do_patch, do_prepare_recipe_sysroot, do_configure and do_compile.
  That's the right default for "bitbake -b", but not here.
- --runonly can't express this either: mark_active() ignores its depth
  argument and recurses over depends, so the whole chain stays active
  regardless.

AI-Generated: Uses GitHub Copilot

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 lib/bb/command.py | 6 +++++-
 lib/bb/cooker.py  | 8 +++++++-
 2 files changed, 12 insertions(+), 2 deletions(-)

diff --git a/lib/bb/command.py b/lib/bb/command.py
index b57c5d4a3..1b16884fa 100644
--- a/lib/bb/command.py
+++ b/lib/bb/command.py
@@ -622,9 +622,13 @@ class CommandsAsync:
             internal = params[2]
         else:
             internal = False
+        if len(params) > 3:
+            taskonly = params[3]
+        else:
+            taskonly = False
 
         if internal:
-            command.cooker.buildFileInternal(bfile, task, fireevents=False, quietlog=True)
+            command.cooker.buildFileInternal(bfile, task, fireevents=False, quietlog=True, taskonly=taskonly)
         else:
             command.cooker.buildFile(bfile, task)
     buildFile.needcache = False
diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 108551a60..0321478d3 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -1368,7 +1368,7 @@ You can also remove the BB_HASHSERVE_UPSTREAM setting, but this may result in si
 
         self.buildFileInternal(buildfile, task)
 
-    def buildFileInternal(self, buildfile, task, fireevents=True, quietlog=False):
+    def buildFileInternal(self, buildfile, task, fireevents=True, quietlog=False, taskonly=False):
         """
         Build the file matching regexp buildfile
         """
@@ -1418,6 +1418,12 @@ You can also remove the BB_HASHSERVE_UPSTREAM setting, but this may result in si
         self.recipecaches[mc].rundeps[fn] = defaultdict(list)
         self.recipecaches[mc].runrecs[fn] = defaultdict(list)
 
+        if taskonly:
+            # Drop the intra-recipe task ordering too ('addtask X after Y'), so
+            # that task is the only entry left in the runqueue.
+            task_deps = self.recipecaches[mc].task_deps[fn]
+            task_deps['parents'] = {t: [] for t in task_deps['tasks']}
+
         bb.parse.siggen.setup_datacache(self.recipecaches)
 
         # Invalidate task for target if force mode active
-- 
2.55.0

[PATCH v2 6/8] tinfoil: add a prepared task runner

[AdrianF]

From: Adrian Freihofer <adrian.freihofer@siemens.com>

Add run_prepared_task() to execute a single recipe task, and nothing
else. No dependencies are resolved and no other task of the recipe is
run, so everything the task consumes must already be in place. It uses
BitBake's normal worker path, retaining the standard dispatch for shell
and Python task bodies as well as fakeroot setup.

devtool ide-sdk needs to rerun an already prepared BitBake task after a
developer changes its source. A normal target build would create a
runqueue and resolve dependencies again, and even "bitbake -b" would
re-run the task's intra-recipe predecessors. That is wrong for this
workflow: the developer intentionally wants to rerun only the prepared
task against content the IDE has just produced. Expose the single-task
worker path so devtool can reuse it instead of maintaining a separate
task executor and pseudo-session setup.

AI-Generated: Uses GitHub Copilot

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 lib/bb/tinfoil.py | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/lib/bb/tinfoil.py b/lib/bb/tinfoil.py
index 634d7796f..42385c879 100644
--- a/lib/bb/tinfoil.py
+++ b/lib/bb/tinfoil.py
@@ -826,20 +826,39 @@ class Tinfoil:
             else:
                 return None
 
-    def build_file(self, buildfile, task, internal=True):
+    def build_file(self, buildfile, task, internal=True, taskonly=False):
         """
         Runs the specified task for just a single recipe (i.e. no dependencies).
         This is equivalent to bitbake -b, except with the default internal=True
         no warning about dependencies will be produced, normal info messages
         from the runqueue will be silenced and BuildInit, BuildStarted and
         BuildCompleted events will not be fired.
+        With taskonly=True the recipe's own task ordering is dropped as well, so
+        only the requested task runs.
         """
-        return self.run_command('buildFile', buildfile, task, internal)
+        return self.run_command('buildFile', buildfile, task, internal, taskonly)
 
     @wait_for
     def build_file_sync(self, *args):
         self.build_file(*args)
 
+    def run_prepared_task(self, recipe, task):
+        """Run *task* for one parsed recipe, and nothing else.
+
+        No dependencies are resolved and no other task of the recipe is run,
+        so everything the task consumes must already be in place. The task
+        runs through the normal BitBake worker path, including fakeroot setup
+        and dispatch of shell or Python task bodies.
+
+        The task is forced to execute rather than being skipped as up to date.
+        Returns False if the task failed.
+        """
+        self.run_command('setConfig', 'force', True)
+        try:
+            return self.build_file_sync(self.get_recipe_file(recipe), task, True, True)
+        finally:
+            self.run_command('setConfig', 'force', False)
+
     def build_targets(self, targets, task=None, handle_events=True, extra_events=None, event_callback=None):
         """
         Builds the specified targets. This is equivalent to a normal invocation
-- 
2.55.0

[PATCH v2 7/8] tests/cooker: add TinfoilTests for run_prepared_task

[AdrianF]

From: Adrian Freihofer <adrian.freihofer@siemens.com>

Add TinfoilTests, built on _BitbakeSubprocessTestCase, covering
'tinfoil: add a prepared task runner'. Each test spawns a subprocess to
isolate tinfoil's server lifecycle.

TestEquivHash is needed because the noop siggen's invalidate_task()
removes the base stamp path instead of the task-specific one, making
force=True a no-op otherwise.

test_run_prepared_task_recipecache_self_heals additionally proves that
run_prepared_task() (like "bitbake -b" before it) only wipes and
repopulates the recipe cache for the one recipe it touches as a
transient footprint: a subsequent normal, full parse restores the
complete recipe set again.

Lives in cooker.py rather than runqueue.py since it tests Tinfoil's
Python API, not CLI-level runqueue behaviour.

AI-Generated: Uses GitHub Copilot

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 lib/bb/tests/cooker.py | 244 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 244 insertions(+)

diff --git a/lib/bb/tests/cooker.py b/lib/bb/tests/cooker.py
index c32694cc5..db4104d40 100644
--- a/lib/bb/tests/cooker.py
+++ b/lib/bb/tests/cooker.py
@@ -63,6 +63,250 @@ class _BitbakeSubprocessTestCase(unittest.TestCase):
                 self._shutdown(builddir)
 
 
+class TinfoilTests(_BitbakeSubprocessTestCase):
+    """Tests for the Tinfoil API that require a running bitbake server."""
+
+    # Library directory containing bb.tinfoil
+    _bblib = os.path.realpath(os.path.join(os.path.dirname(__file__), '..', '..'))
+    # runqueue-tests BBPATH (provides the simple a1/b1/... test recipes)
+    _runqueuetests = os.path.realpath(os.path.join(os.path.dirname(__file__), 'runqueue-tests'))
+
+    failing_recipe = """\
+python do_install() {
+    bb.fatal("deliberate failure")
+}
+addtask install
+"""
+
+    # do_install genuinely depends on do_compile's output, unlike the
+    # a1 fixture recipe's dummy stamptask() tasks.
+    dependent_recipe = """\
+python do_compile() {
+    with open(d.expand("${TOPDIR}/compiled"), "w") as f:
+        f.write("compiled")
+}
+addtask compile
+
+python do_install() {
+    if not os.path.exists(d.expand("${TOPDIR}/compiled")):
+        bb.fatal("do_install ran without do_compile's output being present")
+}
+addtask install after do_compile
+"""
+
+    # Same contract as dependent_recipe, but with shell tasks, since a real
+    # recipe's do_install is usually shell (e.g. install/cp under pseudo)
+    # rather than a python function.
+    shell_dependent_recipe = """\
+do_compile() {
+    echo compiled > "${TOPDIR}/compiled"
+}
+addtask compile
+
+do_install() {
+    if [ ! -e "${TOPDIR}/compiled" ]; then
+        echo "do_install ran without do_compile's output being present" >&2
+        exit 1
+    fi
+}
+addtask install after do_compile
+"""
+
+    def _make_env(self, builddir, extra=None):
+        env = os.environ.copy()
+        env['PYTHONPATH'] = self._bblib + (':' + env['PYTHONPATH'] if 'PYTHONPATH' in env else '')
+        env['BBPATH'] = self._runqueuetests
+        env['BB_ENV_PASSTHROUGH_ADDITIONS'] = 'SSTATEVALID SLOWTASKS TOPDIR BB_HASHSERVE BB_SIGNATURE_HANDLER EXTRA_BBFILES'
+        env['SSTATEVALID'] = ''
+        env['SLOWTASKS'] = ''
+        env['TOPDIR'] = builddir
+        # TestEquivHash creates taint files so that force=True actually
+        # invalidates the task hash; the default noop siggen cannot do this.
+        env['BB_HASHSERVE'] = 'auto'
+        env['BB_SIGNATURE_HANDLER'] = 'TestEquivHash'
+        if extra:
+            env.update(extra)
+        return env
+
+    def _run_script(self, builddir, script, extra=None):
+        """Run script in a subprocess to isolate tinfoil's server lifecycle."""
+        return self._run_subprocess([sys.executable, '-c', script],
+                                    self._make_env(builddir, extra), builddir)
+
+    def _read_tasklog(self, builddir, cleanup=True):
+        tasklog = os.path.join(builddir, 'task.log')
+        tasks = []
+        if os.path.exists(tasklog):
+            with open(tasklog) as f:
+                tasks = [line.rstrip() for line in f]
+            if cleanup:
+                os.remove(tasklog)
+        return tasks
+
+    def test_run_prepared_task(self):
+        """tinfoil.run_prepared_task() reruns one task without resolving deps.
+
+        Uses do_install since that's the real devtool ide-sdk scenario: it
+        needs pseudo and so must run via bitbake, unlike the compile step which
+        the IDE invokes directly (e.g. via cmake/meson).
+
+        Builds a1 completely so all stamps/hashes are valid, then calls
+        run_prepared_task('a1', 'install') through the Python API and verifies
+        that only do_install re-runs while its intra-recipe predecessors
+        (fetch, unpack, patch, prepare_recipe_sysroot, configure, compile) are
+        skipped.
+        """
+        # The script runs inside a subprocess so that tinfoil's server
+        # lifecycle and environment modifications are isolated.
+        script = """
+import os, sys
+import bb.tinfoil
+
+builddir = os.environ['TOPDIR']
+tasklog  = os.path.join(builddir, 'task.log')
+
+with bb.tinfoil.Tinfoil() as tinfoil:
+    tinfoil.prepare(quiet=2)
+    # Full build so all stamps and hashes are valid.
+    tinfoil.build_targets(['a1'])
+    # Clear the log so only the run_prepared_task() entries are counted.
+    if os.path.exists(tasklog):
+        os.remove(tasklog)
+    # run_prepared_task() sets force=True (taint) and calls build_file_sync
+    # with the recipe file resolved via get_recipe_file(), bypassing the
+    # normal runqueue dependency resolver.
+    tinfoil.run_prepared_task('a1', 'install')
+"""
+        with self._build_dir() as builddir:
+            self._run_script(builddir, script)
+
+            tasks = self._read_tasklog(builddir)
+            self.assertEqual(tasks, ['a1:install'],
+                             'run_prepared_task should rerun only install, got: %s' % tasks)
+
+    def test_run_prepared_task_unbuilt(self):
+        """run_prepared_task() does not pull any dependency task into the runqueue.
+
+        buildFileInternal()'s taskonly=True clears task_deps['parents'] for
+        every task of the recipe, so do_install's intra-recipe predecessors
+        (fetch, unpack, patch, ...) are never added to the runqueue at all.
+        The recipe was never built here (no stamps exist for any of them), so
+        this is the case that would actually catch a regression: without
+        taskonly, those missing-stamp tasks would have to run to satisfy the
+        runqueue, and the tasklog assertion below would show more than just
+        'a1:install'.
+
+        Whether do_install itself then succeeds or fails is a separate matter
+        that does depend on the recipe: this fixture's do_install has no real
+        prerequisites, so it succeeds here; test_run_prepared_task_unbuilt_dependent_fails()
+        uses a recipe whose do_install does have one, and fails instead.
+        """
+        script = """
+import bb.tinfoil
+
+with bb.tinfoil.Tinfoil() as tinfoil:
+    tinfoil.prepare(quiet=2)
+    assert tinfoil.run_prepared_task('a1', 'install') is True
+"""
+        with self._build_dir() as builddir:
+            self._run_script(builddir, script)
+
+            tasks = self._read_tasklog(builddir)
+            self.assertEqual(tasks, ['a1:install'],
+                             'run_prepared_task should run no dependency task, got: %s' % tasks)
+
+    def test_run_prepared_task_unbuilt_dependent_fails(self):
+        """A task with a genuine dependency on a predecessor's output fails
+        when that predecessor never ran.
+
+        Unlike test_run_prepared_task_unbuilt()'s fixture recipe, whose tasks
+        are dummy stamptask() calls with no real prerequisites,
+        dependent_recipe's do_install actually needs do_compile's output.
+        run_prepared_task() only skips bitbake's own dependency resolution;
+        it does not make the prerequisites appear, exactly as documented:
+        "everything the task consumes must already be in place".
+        """
+        script = """
+import bb.tinfoil
+
+with bb.tinfoil.Tinfoil() as tinfoil:
+    tinfoil.prepare(quiet=2)
+    assert tinfoil.run_prepared_task('dependent', 'install') is False
+"""
+        with tempfile.TemporaryDirectory(prefix='tinfoilrecipes') as recipes, \
+             self._build_dir() as builddir:
+            with open(os.path.join(recipes, 'dependent.bb'), 'w') as f:
+                f.write(self.dependent_recipe)
+            self._run_script(builddir, script,
+                             {'EXTRA_BBFILES': '%s/*.bb' % recipes})
+
+    def test_run_prepared_task_unbuilt_dependent_fails_shell(self):
+        """Same as test_run_prepared_task_unbuilt_dependent_fails(), but with
+        shell do_compile/do_install tasks instead of python ones, matching how
+        a real recipe's do_install is usually written.
+        """
+        script = """
+import bb.tinfoil
+
+with bb.tinfoil.Tinfoil() as tinfoil:
+    tinfoil.prepare(quiet=2)
+    assert tinfoil.run_prepared_task('shelldependent', 'install') is False
+"""
+        with tempfile.TemporaryDirectory(prefix='tinfoilrecipes') as recipes, \
+             self._build_dir() as builddir:
+            with open(os.path.join(recipes, 'shelldependent.bb'), 'w') as f:
+                f.write(self.shell_dependent_recipe)
+            self._run_script(builddir, script,
+                             {'EXTRA_BBFILES': '%s/*.bb' % recipes})
+
+    def test_run_prepared_task_failure(self):
+        """A failing task makes run_prepared_task() return False, not raise."""
+        script = """
+import bb.tinfoil
+
+with bb.tinfoil.Tinfoil() as tinfoil:
+    tinfoil.prepare(quiet=2)
+    assert tinfoil.run_prepared_task('failer', 'install') is False
+"""
+        with tempfile.TemporaryDirectory(prefix='tinfoilrecipes') as recipes, \
+             self._build_dir() as builddir:
+            with open(os.path.join(recipes, 'failer.bb'), 'w') as f:
+                f.write(self.failing_recipe)
+            self._run_script(builddir, script,
+                             {'EXTRA_BBFILES': '%s/*.bb' % recipes})
+
+    def test_run_prepared_task_recipecache_self_heals(self):
+        """A normal full parse after run_prepared_task() sees every recipe.
+
+        run_prepared_task() -> buildFileInternal() -> parseConfiguration()
+        wipes and repopulates self.recipecaches[mc]/self.collections[mc]
+        for just the one recipe it builds - that has always been true of
+        "bitbake -b" too. Prove this is only a transient footprint: a
+        subsequent normal, full parse (as any non -b build would trigger)
+        must see the complete recipe set again, not just the one recipe
+        run_prepared_task() touched.
+        """
+        script = """
+import bb.tinfoil
+
+with bb.tinfoil.Tinfoil() as tinfoil:
+    tinfoil.prepare(quiet=2)
+    tinfoil.parse_recipes()
+    recipes_before = tinfoil.run_command('getRecipes')
+    assert len(recipes_before) > 1, 'test fixture should have more than one recipe'
+
+    tinfoil.run_prepared_task('a1', 'install')
+
+    tinfoil.parse_recipes()
+    recipes_after = tinfoil.run_command('getRecipes')
+    assert len(recipes_after) == len(recipes_before), (
+        'recipe cache not fully restored after run_prepared_task(): '
+        'before=%d after=%d' % (len(recipes_before), len(recipes_after)))
+"""
+        with self._build_dir() as builddir:
+            self._run_script(builddir, script)
+
+
 class CookerTest(unittest.TestCase):
     def setUp(self):
         # At least one variable needs to be set
-- 
2.55.0

[PATCH v2 8/8] parse/ast: skip empty BBPATH segments in include_all

[AdrianF]

From: Adrian Freihofer <adrian.freihofer@siemens.com>

BBPATH can end up with an empty ":"-split segment when different
layer.conf files mix the "${LAYERDIR}:" (prepend) and ":${LAYERDIR}"
(append) idioms, e.g. openembedded-core's own meta/conf/layer.conf uses
"BBPATH .= \":${LAYERDIR}\"" while every other layer.conf in a typical
poky setup uses "BBPATH =. \"${LAYERDIR}:\"". Combined, this produces a
literal "::" in the final value.

IncludeAllNode.eval() iterates every BBPATH segment and calls
os.path.join(path, s) to build the candidate file to include. For an
empty segment, os.path.join("", s) returns s unchanged, i.e. a
relative path instead of an absolute one. include_single_file() then
takes its relative-path branch, which does its own independent search
across the whole BBPATH and marks every path it tries (found or not)
as a dependency via mark_dependency(), as a side effect of resolving
that one (bogus) relative candidate.

If that side search happens to try the real target file before this
loop's own iteration for its actual BBPATH entry runs,
check_dependency() reports it as already seen and include_single_file()
logs a spurious "Duplicate inclusion" warning for it, even though the
file is only ever included once. This is how e.g. oe-core's
"include_all conf/distro/include/maintainers.inc" in defaultsetup.conf
ends up warning about itself on every parse.

Skip empty segments so an empty BBPATH entry cannot trigger this
false-positive dependency marking.

AI-Generated: Uses GitHub Copilot

Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
---
 lib/bb/parse/ast.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/bb/parse/ast.py b/lib/bb/parse/ast.py
index a372b3534..866ab8ed1 100644
--- a/lib/bb/parse/ast.py
+++ b/lib/bb/parse/ast.py
@@ -56,6 +56,10 @@ class IncludeAllNode(AstNode):
         logger.debug2("CONF %s:%s: including %s", self.filename, self.lineno, s)
 
         for path in data.getVar("BBPATH").split(":"):
+            # Skip empty segments (e.g. from a stray "::" if some layer.conf
+            # uses ".= \":${LAYERDIR}\"" instead of "=. \"${LAYERDIR}:\"").
+            if not path:
+                continue
             bb.parse.ConfHandler.include(self.filename, os.path.join(path, s), self.lineno, data, False)
 
 class ExportNode(AstNode):
-- 
2.55.0

Re: [bitbake-devel] [PATCH v2 1/8] cooker: fix bitbake -b silently ignoring bbappends

[Richard Purdie]

On Mon, 2026-08-17 at 00:14 +0200, Adrian Freihofer via lists.openembedded.org wrote:
> From: Adrian Freihofer <adrian.freihofer@siemens.com>
> 
> "bitbake -b <recipe.bb>" builds the recipe without applying any of its
> .bbappend files.
> 
> buildFileInternal() resolves appends via
> self.collections[mc].get_file_appends(fn), but self.collections[mc] is
> only ever filled in by collect_bbfiles(), called from updateCache() -
> a path -b deliberately skips. matchFiles(), the one -b-path function
> that does call collect_bbfiles(), built a fresh CookerCollectFiles into
> a throwaway local instead of self.collections[mc], so the append list
> stayed empty (or, on a memory-resident server, stale from the last
> full parse - e.g. missing a devtool/externalsrc workspace .bbappend
> added since). Nothing warns that the built metadata differs from disk.
> 
> Make matchFiles() refresh self.collections[mc] itself so the later
> append lookup for the same fn sees the same fresh collection.
> 
> AI-Generated: Uses GitHub Copilot
> 
> Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> ---
>  lib/bb/cooker.py | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
> index 4b6ba3196..108551a60 100644
> --- a/lib/bb/cooker.py
> +++ b/lib/bb/cooker.py
> @@ -1322,8 +1322,10 @@ You can also remove the BB_HASHSERVE_UPSTREAM setting, but this may result in si
>          if bf.startswith("/") or bf.startswith("../"):
>              bf = os.path.abspath(bf)
>  
> -        collections = {mc: CookerCollectFiles(self.bbfile_config_priorities, mc)}
> -        filelist, masked, searchdirs = collections[mc].collect_bbfiles(self.databuilder.mcdata[mc], self.databuilder.mcdata[mc])
> +        # The only place the "bitbake -b" path fills in the bbappends which
> +        # buildFileInternal() then reads back from self.collections[mc].
> +        self.collections[mc] = CookerCollectFiles(self.bbfile_config_priorities, mc)
> +        filelist, masked, searchdirs = self.collections[mc].collect_bbfiles(self.databuilder.mcdata[mc], self.databuilder.mcdata[mc])
>          try:
>              os.stat(bf)
>              bf = os.path.abspath(bf)

I've had some time to stare at the code and I'm not convinced this is
the right way to fix things.

self.collections is usually setup by parseConfiguration, which does so:

    def parseConfiguration(self):
        [...]
        self.handleCollections(self.data.getVar("BBFILE_COLLECTIONS"))
        self.collections = {}
        for mc in self.multiconfigs:
            self.collections[mc] = CookerCollectFiles(self.bbfile_config_priorities, mc)

The call site you're patching in matchFiles() is only called by
matchFile(). matchFile() can be called as a tinfoil command directly,
by buildFileInternal() or by showEnvironment(). The last two both call
parseConfiguration() before matchFile.

matchFile from command.py set "matchFile.needconfig = False" so that
does not need a configuration. I'm not seeing anything using the
matchFile api call so perhaps we could just change needconfig to True.

The bottom line is that tinfoil probably needs that
CookerCollectFiles() call but the rest don't, self.collections should
be ok for the others.

The collect_bbfiles() call is harder. That comes from updateCache()
when parsing finishes, so it needs a full recipe parse.

Looking at the code in collect_bbfiles(), I'm not convinced it does
need a full parse first, it can run without that. The code in that
function is horrible as it sets self.overlayed and self.bbappends but
also gives return values.

My feeling is that the function should either return values, or set
self.* things, but not both. We might be able to make collect_bbfiles
run earlier in parseConfiguration and then skip calling it at all in
matchFiles if it is already available?

Regardless, just making this unconditionally overwrite self.collections
is definitely not an improvement to the current mess, it will just make
it harder to disentangle things later...

Cheers,

Richard