[PATCH] parse: only search each BBPATH directory once in include_all

[PATCH] parse: only search each BBPATH directory once in include_all

[Tim Orling]

If a directory appears in BBPATH more than once, include_all includes
the file from it again, applying it twice and warning "Duplicate
inclusion". The layer index hits this because the core layer is in its
initial BBPATH and appended again by layer.conf. Skip directories
already searched, and add a test for it.

This fixes one source of the warnings in [YOCTO #16451]; the duplicate
inclusion of defaultsetup.conf reported there has a separate cause.

AI-Generated: Claude Opus 5.5
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 lib/bb/parse/ast.py   |  3 ++-
 lib/bb/tests/parse.py | 20 ++++++++++++++++++++
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/lib/bb/parse/ast.py b/lib/bb/parse/ast.py
index 5f2aaadfb..d5d53a7c7 100644
--- a/lib/bb/parse/ast.py
+++ b/lib/bb/parse/ast.py
@@ -55,7 +55,8 @@ class IncludeAllNode(AstNode):
         s = data.expand(self.what_file)
         logger.debug2("CONF %s:%s: including %s", self.filename, self.lineno, s)
 
-        for path in data.getVar("BBPATH").split(":"):
+        # A directory listed more than once in BBPATH is only searched once
+        for path in dict.fromkeys(data.getVar("BBPATH").split(":")):
             bb.parse.ConfHandler.include(self.filename, os.path.join(path, s), self.lineno, data, False)
 
 class ExportNode(AstNode):
diff --git a/lib/bb/tests/parse.py b/lib/bb/tests/parse.py
index 6ac2137e0..cd8f02388 100644
--- a/lib/bb/tests/parse.py
+++ b/lib/bb/tests/parse.py
@@ -511,6 +511,26 @@ EXTRA_OECONF:append = " foobar"
             test_helper("include_all some.conf", " bar foo")
             test_helper("include_all some3.conf", " foobar")
 
+    def test_include_all_duplicate_bbpath(self):
+        # A directory listed more than once in BBPATH is only searched once
+        with tempfile.TemporaryDirectory() as tempdir:
+            os.makedirs(tempdir + "/conf1")
+            os.makedirs(tempdir + "/conf2")
+            with open(tempdir + "/conf1/some.conf", "w") as f:
+                f.write(self.someconf1)
+            with open(tempdir + "/conf2/some.conf", "w") as f:
+                f.write(self.someconf2)
+
+            self.d.setVar("BBPATH", ":".join([tempdir + "/conf1", tempdir + "/conf2", tempdir + "/conf1"]))
+
+            with self.parsehelper("include_all some.conf") as f:
+                with self.assertLogs("BitBake.Parsing", level="WARNING") as logs:
+                    d = bb.parse.handle(f.name, bb.data.createCopy(self.d))['']
+                    # assertLogs fails if nothing is logged at all
+                    bb.parse.logger.warning("parse done")
+            self.assertEqual(d.getVar("EXTRA_OECONF"), " foo bar")
+            self.assertEqual(logs.output, ["WARNING:BitBake.Parsing:parse done"])
+
     def test_file_variables(self):
         # Tests the values of FILE and __BB_RECIPE_FILE in different
         # combinations of bbappends, includes, and inherits
-- 
2.56.0

Re: [bitbake-devel] [PATCH] parse: only search each BBPATH directory once in include_all

[Ross Burton]

On 19 Oct 2026, at 03:04, Tim Orling via lists.openembedded.org <ticotimo=gmail.com@lists.openembedded.org> wrote:
> 
> +        for path in dict.fromkeys(data.getVar("BBPATH").split(":")):

Feels like using a set() instead of dict.fromkeys() would be neater.

Ross

Re: [bitbake-devel] [PATCH] parse: only search each BBPATH directory once in include_all

[Richard Purdie]

On Mon, 2026-10-19 at 16:12 +0000, Ross Burton via
lists.openembedded.org wrote:
> On 19 Oct 2026, at 03:04, Tim Orling via lists.openembedded.org
> <ticotimo=gmail.com@lists.openembedded.org> wrote:
> > 
> > +        for path in
> > dict.fromkeys(data.getVar("BBPATH").split(":")):
> 
> Feels like using a set() instead of dict.fromkeys() would be neater.

Ordering matters here...

Cheers,

Richard

Re: [bitbake-devel] [PATCH] parse: only search each BBPATH directory once in include_all

[Ross Burton]

On 20 Oct 2026, at 13:30, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> 
>> Feels like using a set() instead of dict.fromkeys() would be neater.
> 
> Ordering matters here…

Hm I thought sets have the same ordering behaviour as dictionary keys, but whilst they do in practise at the moment, this is not documented as behaviour.

Ross

Re: [bitbake-devel] [PATCH] parse: only search each BBPATH directory once in include_all

[Richard Purdie]

On Tue, 2026-10-20 at 12:33 +0000, Ross Burton wrote:
> On 20 Oct 2026, at 13:30, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > 
> > > Feels like using a set() instead of dict.fromkeys() would be
> > > neater.
> > 
> > Ordering matters here…
> 
> Hm I thought sets have the same ordering behaviour as dictionary
> keys, but whilst they do in practise at the moment, this is not
> documented as behaviour.

They don't even in practise, I've seen sets not remember order at
all...

Cheers,

Richard

Re: [bitbake-devel] [PATCH] parse: only search each BBPATH directory once in include_all

[Richard Purdie]

On Sun, 2026-10-18 at 19:04 -0700, Tim Orling via lists.openembedded.org wrote:
> If a directory appears in BBPATH more than once, include_all includes
> the file from it again, applying it twice and warning "Duplicate
> inclusion". The layer index hits this because the core layer is in its
> initial BBPATH and appended again by layer.conf. Skip directories
> already searched, and add a test for it.
> 
> This fixes one source of the warnings in [YOCTO #16451]; the duplicate
> inclusion of defaultsetup.conf reported there has a separate cause.

Tim and I talked briefly on the review call.

The patch is probably ok however the real issue is the path duplication
in BBPATH and the "real" fix is probably to detect and error on that
happening in the first place.

Cheers,

Richard

[PATCH v2 0/2] Handle duplicate BBPATH entries

[Tim Orling]

Following Richard's review of v1, patch 2 makes duplicate BBPATH
entries a fatal error, so the problem is reported at its source.
Patch 1 is kept so include_all stays correct if BBPATH changes later.

Changes in v2:
- Patch 1: commit message trimmed, code unchanged.
- New patch 2.

Tim Orling (2):
  parse: only search each BBPATH directory once in include_all
  cookerdata: error on duplicate BBPATH entries

 lib/bb/cookerdata.py  | 12 +++++++++++
 lib/bb/parse/ast.py   |  3 ++-
 lib/bb/tests/parse.py | 47 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 61 insertions(+), 1 deletion(-)

-- 
2.56.0

[PATCH v2 1/2] parse: only search each BBPATH directory once in include_all

[Tim Orling]

A directory listed twice in BBPATH made include_all apply the file from
it twice and warn "Duplicate inclusion". The layer index hits this with
the core layer in both its initial BBPATH and layer.conf. Skip
directories already searched. Fixes one source of the warnings in
[YOCTO #16451].

AI-Generated: Claude Opus 5.5
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 lib/bb/parse/ast.py   |  3 ++-
 lib/bb/tests/parse.py | 20 ++++++++++++++++++++
 2 files changed, 22 insertions(+), 1 deletion(-)

diff --git a/lib/bb/parse/ast.py b/lib/bb/parse/ast.py
index 5f2aaadfb..d5d53a7c7 100644
--- a/lib/bb/parse/ast.py
+++ b/lib/bb/parse/ast.py
@@ -55,7 +55,8 @@ class IncludeAllNode(AstNode):
         s = data.expand(self.what_file)
         logger.debug2("CONF %s:%s: including %s", self.filename, self.lineno, s)
 
-        for path in data.getVar("BBPATH").split(":"):
+        # A directory listed more than once in BBPATH is only searched once
+        for path in dict.fromkeys(data.getVar("BBPATH").split(":")):
             bb.parse.ConfHandler.include(self.filename, os.path.join(path, s), self.lineno, data, False)
 
 class ExportNode(AstNode):
diff --git a/lib/bb/tests/parse.py b/lib/bb/tests/parse.py
index 6ac2137e0..cd8f02388 100644
--- a/lib/bb/tests/parse.py
+++ b/lib/bb/tests/parse.py
@@ -511,6 +511,26 @@ EXTRA_OECONF:append = " foobar"
             test_helper("include_all some.conf", " bar foo")
             test_helper("include_all some3.conf", " foobar")
 
+    def test_include_all_duplicate_bbpath(self):
+        # A directory listed more than once in BBPATH is only searched once
+        with tempfile.TemporaryDirectory() as tempdir:
+            os.makedirs(tempdir + "/conf1")
+            os.makedirs(tempdir + "/conf2")
+            with open(tempdir + "/conf1/some.conf", "w") as f:
+                f.write(self.someconf1)
+            with open(tempdir + "/conf2/some.conf", "w") as f:
+                f.write(self.someconf2)
+
+            self.d.setVar("BBPATH", ":".join([tempdir + "/conf1", tempdir + "/conf2", tempdir + "/conf1"]))
+
+            with self.parsehelper("include_all some.conf") as f:
+                with self.assertLogs("BitBake.Parsing", level="WARNING") as logs:
+                    d = bb.parse.handle(f.name, bb.data.createCopy(self.d))['']
+                    # assertLogs fails if nothing is logged at all
+                    bb.parse.logger.warning("parse done")
+            self.assertEqual(d.getVar("EXTRA_OECONF"), " foo bar")
+            self.assertEqual(logs.output, ["WARNING:BitBake.Parsing:parse done"])
+
     def test_file_variables(self):
         # Tests the values of FILE and __BB_RECIPE_FILE in different
         # combinations of bbappends, includes, and inherits
-- 
2.56.0

[PATCH v2 2/2] cookerdata: error on duplicate BBPATH entries

[Tim Orling]

A directory listed more than once in BBPATH makes include_all apply the
same file repeatedly. Fail once the layer configuration is parsed, so
the error points at the configuration rather than surfacing later as
duplicate inclusion warnings. Paths are compared after resolving them,
and empty entries are ignored.

Tested with bitbake-selftest and an oe-core tinfoil config parse.

Suggested-by: Richard Purdie <richard.purdie@linuxfoundation.org>
AI-Generated: Uses Claude Code (Claude Opus 5.5)
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 lib/bb/cookerdata.py  | 12 ++++++++++++
 lib/bb/tests/parse.py | 27 +++++++++++++++++++++++++++
 2 files changed, 39 insertions(+)

diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
index d65e3c974..87310af7c 100644
--- a/lib/bb/cookerdata.py
+++ b/lib/bb/cookerdata.py
@@ -482,6 +482,18 @@ class CookerDataBuilder(object):
                         " invoked bitbake from the wrong directory?")
             bb.fatal(msg)
 
+        # A directory listed more than once in BBPATH is a configuration
+        # error, e.g. a layer in both the BBPATH env var and BBLAYERS. Compare
+        # resolved paths so "dir" and "dir/" or a symlink to it also match, and
+        # ignore the empty entries left by stray colons
+        entries = {}
+        for p in data.getVar("BBPATH").split(":"):
+            if p:
+                entries.setdefault(os.path.realpath(p), []).append(p)
+        duplicates = [" ".join(paths) for paths in entries.values() if len(paths) > 1]
+        if duplicates:
+            bb.fatal("BBPATH contains duplicate entries:\n    %s\nCheck the BBPATH environment variable, bblayers.conf and the layer.conf files." % "\n    ".join(duplicates))
+
         if not data.getVar("TOPDIR"):
             data.setVar("TOPDIR", os.path.abspath(os.getcwd()))
         if not data.getVar("BB_CACHEDIR"):
diff --git a/lib/bb/tests/parse.py b/lib/bb/tests/parse.py
index cd8f02388..90a63e7dc 100644
--- a/lib/bb/tests/parse.py
+++ b/lib/bb/tests/parse.py
@@ -531,6 +531,33 @@ EXTRA_OECONF:append = " foobar"
             self.assertEqual(d.getVar("EXTRA_OECONF"), " foo bar")
             self.assertEqual(logs.output, ["WARNING:BitBake.Parsing:parse done"])
 
+    def test_duplicate_bbpath(self):
+        # A directory listed more than once in BBPATH is fatal
+        bbpath = os.path.realpath(os.path.join(os.path.dirname(__file__), "parse-tests"))
+        with tempfile.TemporaryDirectory(prefix="parsetest") as builddir:
+            def run_bitbake(entries):
+                env = os.environ.copy()
+                env["BBPATH"] = ":".join(entries)
+                env["BB_ENV_PASSTHROUGH_ADDITIONS"] = "TOPDIR"
+                env["TOPDIR"] = builddir
+                return subprocess.run(["bitbake", "-e"], env=env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, universal_newlines=True, cwd=builddir)
+
+            link = os.path.join(builddir, "link")
+            os.symlink(bbpath, link)
+
+            # The same directory spelled the same, with a trailing slash or
+            # through a symlink is a duplicate
+            for dup in (bbpath, bbpath + "/", link):
+                with self.subTest(dup=dup):
+                    result = run_bitbake([bbpath, dup])
+                    self.assertNotEqual(result.returncode, 0)
+                    self.assertIn("BBPATH contains duplicate entries:\n    %s %s\n" % (bbpath, dup), result.stdout)
+
+            # Empty entries from stray colons are not duplicates
+            result = run_bitbake(["", bbpath, "", ""])
+            self.assertEqual(result.returncode, 0, result.stdout)
+            self.assertNotIn("BBPATH contains duplicate entries", result.stdout)
+
     def test_file_variables(self):
         # Tests the values of FILE and __BB_RECIPE_FILE in different
         # combinations of bbappends, includes, and inherits
-- 
2.56.0

Re: [bitbake-devel] [PATCH v2 2/2] cookerdata: error on duplicate BBPATH entries

[Mathieu Dubois-Briand]

On Mon Oct 26, 2026 at 1:20 AM CET, Tim Orling via lists.openembedded.org wrote:
> A directory listed more than once in BBPATH makes include_all apply the
> same file repeatedly. Fail once the layer configuration is parsed, so
> the error points at the configuration rather than surfacing later as
> duplicate inclusion warnings. Paths are compared after resolving them,
> and empty entries are ignored.
>
> Tested with bitbake-selftest and an oe-core tinfoil config parse.
>
> Suggested-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> AI-Generated: Uses Claude Code (Claude Opus 5.5)
> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> ---

Hi Tim,

I just note this patch is sometimes finding duplicate entries on the
autobuilder: I opened a bug for this.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=16481

Thanks,
Mathieu

-- 
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com