[PATCH] parse: only search each BBPATH directory once in include_all

[PATCH] parse: only search each BBPATH directory once in include_all

[Tim Orling]

If a directory appears in BBPATH more than once, include_all includes
the file from it again, applying it twice and warning "Duplicate
inclusion". The layer index hits this because the core layer is in its
initial BBPATH and appended again by layer.conf. Skip directories
already searched, and add a test for it.

This fixes one source of the warnings in [YOCTO #16451]; the duplicate
inclusion of defaultsetup.conf reported there has a separate cause.

AI-Generated: Claude Opus 5.5
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 lib/bb/parse/ast.py   |  3 ++-
 lib/bb/tests/parse.py | 20 ++++++++++++++++++++
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/lib/bb/parse/ast.py b/lib/bb/parse/ast.py
index 5f2aaadfb..d5d53a7c7 100644
--- a/lib/bb/parse/ast.py
+++ b/lib/bb/parse/ast.py
@@ -55,7 +55,8 @@ class IncludeAllNode(AstNode):
         s = data.expand(self.what_file)
         logger.debug2("CONF %s:%s: including %s", self.filename, self.lineno, s)
 
-        for path in data.getVar("BBPATH").split(":"):
+        # A directory listed more than once in BBPATH is only searched once
+        for path in dict.fromkeys(data.getVar("BBPATH").split(":")):
             bb.parse.ConfHandler.include(self.filename, os.path.join(path, s), self.lineno, data, False)
 
 class ExportNode(AstNode):
diff --git a/lib/bb/tests/parse.py b/lib/bb/tests/parse.py
index 6ac2137e0..cd8f02388 100644
--- a/lib/bb/tests/parse.py
+++ b/lib/bb/tests/parse.py
@@ -511,6 +511,26 @@ EXTRA_OECONF:append = " foobar"
             test_helper("include_all some.conf", " bar foo")
             test_helper("include_all some3.conf", " foobar")
 
+    def test_include_all_duplicate_bbpath(self):
+        # A directory listed more than once in BBPATH is only searched once
+        with tempfile.TemporaryDirectory() as tempdir:
+            os.makedirs(tempdir + "/conf1")
+            os.makedirs(tempdir + "/conf2")
+            with open(tempdir + "/conf1/some.conf", "w") as f:
+                f.write(self.someconf1)
+            with open(tempdir + "/conf2/some.conf", "w") as f:
+                f.write(self.someconf2)
+
+            self.d.setVar("BBPATH", ":".join([tempdir + "/conf1", tempdir + "/conf2", tempdir + "/conf1"]))
+
+            with self.parsehelper("include_all some.conf") as f:
+                with self.assertLogs("BitBake.Parsing", level="WARNING") as logs:
+                    d = bb.parse.handle(f.name, bb.data.createCopy(self.d))['']
+                    # assertLogs fails if nothing is logged at all
+                    bb.parse.logger.warning("parse done")
+            self.assertEqual(d.getVar("EXTRA_OECONF"), " foo bar")
+            self.assertEqual(logs.output, ["WARNING:BitBake.Parsing:parse done"])
+
     def test_file_variables(self):
         # Tests the values of FILE and __BB_RECIPE_FILE in different
         # combinations of bbappends, includes, and inherits
-- 
2.56.0

Re: [bitbake-devel] [PATCH] parse: only search each BBPATH directory once in include_all

[Ross Burton]

On 19 Oct 2026, at 03:04, Tim Orling via lists.openembedded.org <ticotimo=gmail.com@lists.openembedded.org> wrote:
> 
> +        for path in dict.fromkeys(data.getVar("BBPATH").split(":")):

Feels like using a set() instead of dict.fromkeys() would be neater.

Ross

Re: [bitbake-devel] [PATCH] parse: only search each BBPATH directory once in include_all

[Richard Purdie]

On Mon, 2026-10-19 at 16:12 +0000, Ross Burton via
lists.openembedded.org wrote:
> On 19 Oct 2026, at 03:04, Tim Orling via lists.openembedded.org
> <ticotimo=gmail.com@lists.openembedded.org> wrote:
> > 
> > +        for path in
> > dict.fromkeys(data.getVar("BBPATH").split(":")):
> 
> Feels like using a set() instead of dict.fromkeys() would be neater.

Ordering matters here...

Cheers,

Richard

Re: [bitbake-devel] [PATCH] parse: only search each BBPATH directory once in include_all

[Richard Purdie]

On Sun, 2026-10-18 at 19:04 -0700, Tim Orling via lists.openembedded.org wrote:
> If a directory appears in BBPATH more than once, include_all includes
> the file from it again, applying it twice and warning "Duplicate
> inclusion". The layer index hits this because the core layer is in its
> initial BBPATH and appended again by layer.conf. Skip directories
> already searched, and add a test for it.
> 
> This fixes one source of the warnings in [YOCTO #16451]; the duplicate
> inclusion of defaultsetup.conf reported there has a separate cause.

Tim and I talked briefly on the review call.

The patch is probably ok however the real issue is the path duplication
in BBPATH and the "real" fix is probably to detect and error on that
happening in the first place.

Cheers,

Richard

Re: [bitbake-devel] [PATCH] parse: only search each BBPATH directory once in include_all

[Ross Burton]

On 20 Oct 2026, at 13:30, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> 
>> Feels like using a set() instead of dict.fromkeys() would be neater.
> 
> Ordering matters here…

Hm I thought sets have the same ordering behaviour as dictionary keys, but whilst they do in practise at the moment, this is not documented as behaviour.

Ross

Re: [bitbake-devel] [PATCH] parse: only search each BBPATH directory once in include_all

[Richard Purdie]

On Tue, 2026-10-20 at 12:33 +0000, Ross Burton wrote:
> On 20 Oct 2026, at 13:30, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > 
> > > Feels like using a set() instead of dict.fromkeys() would be
> > > neater.
> > 
> > Ordering matters here…
> 
> Hm I thought sets have the same ordering behaviour as dictionary
> keys, but whilst they do in practise at the moment, this is not
> documented as behaviour.

They don't even in practise, I've seen sets not remember order at
all...

Cheers,

Richard