Re: [bitbake-devel] [PATCH v2 1/8] cooker: fix bitbake -b silently ignoring bbappends

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Mon, 2026-08-17 at 00:14 +0200, Adrian Freihofer via lists.openembedded.org wrote:
> From: Adrian Freihofer <adrian.freihofer@siemens.com>
> 
> "bitbake -b <recipe.bb>" builds the recipe without applying any of its
> .bbappend files.
> 
> buildFileInternal() resolves appends via
> self.collections[mc].get_file_appends(fn), but self.collections[mc] is
> only ever filled in by collect_bbfiles(), called from updateCache() -
> a path -b deliberately skips. matchFiles(), the one -b-path function
> that does call collect_bbfiles(), built a fresh CookerCollectFiles into
> a throwaway local instead of self.collections[mc], so the append list
> stayed empty (or, on a memory-resident server, stale from the last
> full parse - e.g. missing a devtool/externalsrc workspace .bbappend
> added since). Nothing warns that the built metadata differs from disk.
> 
> Make matchFiles() refresh self.collections[mc] itself so the later
> append lookup for the same fn sees the same fresh collection.
> 
> AI-Generated: Uses GitHub Copilot
> 
> Signed-off-by: Adrian Freihofer <adrian.freihofer@siemens.com>
> ---
>  lib/bb/cooker.py | 6 ++++--
>  1 file changed, 4 insertions(+), 2 deletions(-)
> 
> diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
> index 4b6ba3196..108551a60 100644
> --- a/lib/bb/cooker.py
> +++ b/lib/bb/cooker.py
> @@ -1322,8 +1322,10 @@ You can also remove the BB_HASHSERVE_UPSTREAM setting, but this may result in si
>          if bf.startswith("/") or bf.startswith("../"):
>              bf = os.path.abspath(bf)
>  
> -        collections = {mc: CookerCollectFiles(self.bbfile_config_priorities, mc)}
> -        filelist, masked, searchdirs = collections[mc].collect_bbfiles(self.databuilder.mcdata[mc], self.databuilder.mcdata[mc])
> +        # The only place the "bitbake -b" path fills in the bbappends which
> +        # buildFileInternal() then reads back from self.collections[mc].
> +        self.collections[mc] = CookerCollectFiles(self.bbfile_config_priorities, mc)
> +        filelist, masked, searchdirs = self.collections[mc].collect_bbfiles(self.databuilder.mcdata[mc], self.databuilder.mcdata[mc])
>          try:
>              os.stat(bf)
>              bf = os.path.abspath(bf)

I've had some time to stare at the code and I'm not convinced this is
the right way to fix things.

self.collections is usually setup by parseConfiguration, which does so:

    def parseConfiguration(self):
        [...]
        self.handleCollections(self.data.getVar("BBFILE_COLLECTIONS"))
        self.collections = {}
        for mc in self.multiconfigs:
            self.collections[mc] = CookerCollectFiles(self.bbfile_config_priorities, mc)

The call site you're patching in matchFiles() is only called by
matchFile(). matchFile() can be called as a tinfoil command directly,
by buildFileInternal() or by showEnvironment(). The last two both call
parseConfiguration() before matchFile.

matchFile from command.py set "matchFile.needconfig = False" so that
does not need a configuration. I'm not seeing anything using the
matchFile api call so perhaps we could just change needconfig to True.

The bottom line is that tinfoil probably needs that
CookerCollectFiles() call but the rest don't, self.collections should
be ok for the others.

The collect_bbfiles() call is harder. That comes from updateCache()
when parsing finishes, so it needs a full recipe parse.

Looking at the code in collect_bbfiles(), I'm not convinced it does
need a full parse first, it can run without that. The code in that
function is horrible as it sets self.overlayed and self.bbappends but
also gives return values.

My feeling is that the function should either return values, or set
self.* things, but not both. We might be able to make collect_bbfiles
run earlier in parseConfiguration and then skip calling it at all in
matchFiles if it is already available?

Regardless, just making this unconditionally overwrite self.collections
is definitely not an improvement to the current mess, it will just make
it harder to disentangle things later...

Cheers,

Richard