Openembedded Core Discussions
 help / color / mirror / Atom feed
* [PATCH v2 0/2] package: replace copydebugsources shell pipelines
@ 2026-06-18  7:19 Anders Heimer
  2026-06-18  7:19 ` [PATCH v2 1/2] " Anders Heimer
  2026-06-18  7:19 ` [PATCH v2 2/2] oeqa/selftest: add copydebugsources tests Anders Heimer
  0 siblings, 2 replies; 4+ messages in thread
From: Anders Heimer @ 2026-06-18  7:19 UTC (permalink / raw)
  To: openembedded-core; +Cc: Anders Heimer

Replace the copydebugsources() sort/grep/sed shell pipeline with Python
filtering over the NUL-separated source list while keeping cpio for the
copy pass. Use an explicit prefix + "/" match before stripping the mapped
debug source prefix, replace the symlink fixup pipeline with os.walk()
plus cpio, use an argv-list mv for externalsrc relocation, and pass the
empty-directory find command as an argv list.

The externalsrc relocation keeps mv, but now with an argv list and
glob.glob(glob.escape(...)). This preserves the old shell "*" and mv
overwrite semantics.

The first cpio copy pass keeps the previous failure-tolerant behavior,
while the symlink fixup copy still reports cpio failures.

Benchmarks did not indicate regression.

Changes in v2:
  - Replace sort/grep/sed filtering with Python filtering while keeping cpio.
  - Preserve externalsrc mv behavior using argv-list mv.
  - Add test coverage for filtering, copy failures, symlinks, and relocation.

Anders Heimer (2):
  package: replace copydebugsources shell pipelines
  oeqa/selftest: add copydebugsources tests

 meta/lib/oe/package.py                        |  70 +++--
 meta/lib/oeqa/selftest/cases/oelib/package.py | 274 ++++++++++++++++++
 2 files changed, 322 insertions(+), 22 deletions(-)
 create mode 100644 meta/lib/oeqa/selftest/cases/oelib/package.py



^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH v2 1/2] package: replace copydebugsources shell pipelines
  2026-06-18  7:19 [PATCH v2 0/2] package: replace copydebugsources shell pipelines Anders Heimer
@ 2026-06-18  7:19 ` Anders Heimer
  2026-06-18 14:02   ` [OE-core] " Paul Barker
  2026-06-18  7:19 ` [PATCH v2 2/2] oeqa/selftest: add copydebugsources tests Anders Heimer
  1 sibling, 1 reply; 4+ messages in thread
From: Anders Heimer @ 2026-06-18  7:19 UTC (permalink / raw)
  To: openembedded-core; +Cc: Anders Heimer

Replace the sort/grep/sed parts of copydebugsources with Python
filtering on the NUL-separated source list. Keep cpio since it is
faster than doing it in python.

Use an explicit prefix + "/" match before stripping the prefix so source
selection is limited to files in the mapped debug source directory.

Replace the find/sed symlink fixup pipeline with os.walk() plus cpio,
use an argv-list mv for the externalsrc relocation, and pass the
empty-directory find command as an argv list.

The first cpio copy pass continues to ignore failures as before since
some inputs are expected to fail. The symlink fixup copy still reports
cpio failures.

Signed-off-by: Anders Heimer <anders.heimer@est.tech>
---
 meta/lib/oe/package.py | 70 +++++++++++++++++++++++++++++-------------
 1 file changed, 48 insertions(+), 22 deletions(-)

diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index c375acc124..c4ad364b64 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -1017,26 +1017,48 @@ def copydebugsources(debugsrcdir, sources, d):
         bb.utils.mkdirhier(basepath)
         cpath.updatecache(basepath)
 
-        for pmap in prefixmap:
-            # Ignore files from the recipe sysroots (target and native)
-            cmd =  "LC_ALL=C ; sort -z -u '%s' | egrep -v -z '((<internal>|<built-in>)$|/.*recipe-sysroot.*/)' | " % sourcefile
-            # We need to ignore files that are not actually ours
-            # we do this by only paying attention to items from this package
-            cmd += "fgrep -zw '%s' | " % prefixmap[pmap]
-            # Remove prefix in the source paths
-            cmd += "sed 's#%s/##g' | " % (prefixmap[pmap])
-            cmd += "(cd '%s' ; cpio -pd0mlLu --no-preserve-owner '%s%s' 2>/dev/null)" % (pmap, dvar, prefixmap[pmap])
+        # Ignore files from the recipe sysroots (target and native), and
+        # compiler internal entries.
+        with open(sourcefile, "rb") as f:
+            sourcepaths = sorted({path for path in f.read().split(b"\0")
+                                  if path
+                                  and not path.endswith((b"<internal>", b"<built-in>"))
+                                  and b"recipe-sysroot" not in os.path.dirname(path)})
+
+        for pmap, prefix in prefixmap.items():
+            dstroot = dvar + prefix
+            prefix_slash = os.fsencode(prefix) + b"/"
+            relpaths = [path[len(prefix_slash):]
+                        for path in sourcepaths
+                        if path.startswith(prefix_slash)]
+
+            if relpaths:
+                subprocess.run(["cpio", "-pd0mlLu", "--no-preserve-owner", dstroot],
+                               input=b"\0".join(relpaths) + b"\0",
+                               cwd=pmap, stdout=subprocess.DEVNULL,
+                               stderr=subprocess.DEVNULL, check=False)
 
-            try:
-                subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
-            except subprocess.CalledProcessError:
-                # Can "fail" if internal headers/transient sources are attempted
-                pass
             # cpio seems to have a bug with -lL together and symbolic links are just copied, not dereferenced.
             # Work around this by manually finding and copying any symbolic links that made it through.
-            cmd = "find %s%s -type l -print0 -delete | sed s#%s%s/##g | (cd '%s' ; cpio -pd0mL --no-preserve-owner '%s%s')" % \
-                    (dvar, prefixmap[pmap], dvar, prefixmap[pmap], pmap, dvar, prefixmap[pmap])
-            subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
+            symlinks = []
+            for root, dirs, files in os.walk(dstroot, topdown=True, followlinks=False):
+                for name in dirs[:]:
+                    path = os.path.join(root, name)
+                    if os.path.islink(path):
+                        symlinks.append(os.fsencode(os.path.relpath(path, dstroot)))
+                        os.unlink(path)
+                        dirs.remove(name)
+
+                for name in files:
+                    path = os.path.join(root, name)
+                    if os.path.islink(path):
+                        symlinks.append(os.fsencode(os.path.relpath(path, dstroot)))
+                        os.unlink(path)
+
+            if symlinks:
+                subprocess.check_output(["cpio", "-pd0mL", "--no-preserve-owner", dstroot],
+                                        input=b"\0".join(sorted(symlinks)) + b"\0",
+                                        cwd=pmap, stderr=subprocess.STDOUT)
 
         # debugsources.list may be polluted from the host if we used externalsrc,
         # cpio uses copy-pass and may have just created a directory structure
@@ -1046,13 +1068,17 @@ def copydebugsources(debugsrcdir, sources, d):
 
         # Same check as above for externalsrc
         if workdir not in sdir:
-            if os.path.exists(dvar + debugsrcdir + sdir):
-                cmd = "mv %s%s%s/* %s%s" % (dvar, debugsrcdir, sdir, dvar,debugsrcdir)
-                subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
+            srcdir = dvar + debugsrcdir + sdir
+            dstdir = dvar + debugsrcdir
+            if os.path.exists(srcdir):
+                entries = sorted(glob.glob(os.path.join(glob.escape(srcdir), "*")))
+                if entries:
+                    subprocess.check_output(["mv", "--"] + entries + [dstdir],
+                                            stderr=subprocess.STDOUT)
 
         # The copy by cpio may have resulted in some empty directories!  Remove these
-        cmd = "find %s%s -empty -type d -delete" % (dvar, debugsrcdir)
-        subprocess.check_output(cmd, shell=True, stderr=subprocess.STDOUT)
+        cmd = ["find", dvar + debugsrcdir, "-empty", "-type", "d", "-delete"]
+        subprocess.check_output(cmd, stderr=subprocess.STDOUT)
 
         # Also remove debugsrcdir if its empty
         for p in nosuchdir[::-1]:


^ permalink raw reply related	[flat|nested] 4+ messages in thread

* [PATCH v2 2/2] oeqa/selftest: add copydebugsources tests
  2026-06-18  7:19 [PATCH v2 0/2] package: replace copydebugsources shell pipelines Anders Heimer
  2026-06-18  7:19 ` [PATCH v2 1/2] " Anders Heimer
@ 2026-06-18  7:19 ` Anders Heimer
  1 sibling, 0 replies; 4+ messages in thread
From: Anders Heimer @ 2026-06-18  7:19 UTC (permalink / raw)
  To: openembedded-core; +Cc: Anders Heimer

Cover multiple source-prefix mappings, recipe-sysroot and compiler
entry filtering, filenames requiring safe argument handling, symlink
dereferencing, tolerated cpio copy failures, empty-directory cleanup,
and externalsrc relocation.

Exercise paths containing spaces and glob metacharacters, leading-dash
filenames, mv overwrite behavior, an empty relocation directory, and a
source copy failure which leaves the prefix-specific destination
uncreated.

AI-Generated: Claude Opus 4.6

Signed-off-by: Anders Heimer <anders.heimer@est.tech>
---
 meta/lib/oeqa/selftest/cases/oelib/package.py | 274 ++++++++++++++++++
 1 file changed, 274 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/oelib/package.py

diff --git a/meta/lib/oeqa/selftest/cases/oelib/package.py b/meta/lib/oeqa/selftest/cases/oelib/package.py
new file mode 100644
index 0000000000..b9f9324a10
--- /dev/null
+++ b/meta/lib/oeqa/selftest/cases/oelib/package.py
@@ -0,0 +1,274 @@
+#
+# Copyright OpenEmbedded Contributors
+#
+# SPDX-License-Identifier: MIT
+#
+
+import os
+import shutil
+import tempfile
+from unittest.case import TestCase
+
+from oe.package import copydebugsources
+
+
+class FakeDataStore:
+    def __init__(self, values):
+        self.values = values
+
+    def getVar(self, name):
+        return self.values.get(name)
+
+    def expand(self, value):
+        for name, replacement in self.values.items():
+            value = value.replace("${%s}" % name, replacement)
+        return value
+
+
+class TestCopyDebugSources(TestCase):
+    def setUp(self):
+        for tool in ("cpio", "find", "mv"):
+            if shutil.which(tool) is None:
+                self.skipTest("Required tool %s not found" % tool)
+
+    def test_copydebugsources_copies_files_and_dereferences_links(self):
+        with tempfile.TemporaryDirectory(prefix="oe-test-package-") as tmpdir:
+            source_root = os.path.join(tmpdir, "source")
+            second_source_root = os.path.join(tmpdir, "second-source")
+            workdir = os.path.join(tmpdir, "work")
+            pkgd = os.path.join(tmpdir, "pkgd")
+            debugsrcdir = "/usr/src/debug/testpkg/1.0"
+            second_debugsrcdir = "/usr/src/debug/secondpkg/1.0"
+
+            os.makedirs(os.path.join(source_root, "nested"))
+            os.makedirs(second_source_root)
+            os.makedirs(workdir)
+            os.makedirs(pkgd)
+
+            normal = os.path.join(source_root, "nested", "normal.c")
+            special = os.path.join(source_root, "nested", "name with ; spaces.c")
+            leading_dash = os.path.join(source_root, "nested", "-leading-dash.c")
+            recipe_sysroot_basename = os.path.join(source_root, "nested",
+                                                   "recipe-sysroot-file.c")
+            internal = os.path.join(source_root, "nested", "compiler<internal>")
+            built_in = os.path.join(source_root, "nested", "compiler<built-in>")
+            target = os.path.join(source_root, "nested", "target.c")
+            link = os.path.join(source_root, "nested", "link.c")
+            ignored_src = os.path.join(source_root, "recipe-sysroot", "ignored.c")
+            ignored_native_src = os.path.join(source_root,
+                                             "foo-recipe-sysroot-native",
+                                             "ignored.c")
+            second = os.path.join(second_source_root, "second.c")
+
+            with open(normal, "w") as f:
+                f.write("normal\n")
+            with open(special, "w") as f:
+                f.write("special\n")
+            with open(leading_dash, "w") as f:
+                f.write("leading dash\n")
+            with open(recipe_sysroot_basename, "w") as f:
+                f.write("recipe sysroot basename\n")
+            with open(internal, "w") as f:
+                f.write("internal\n")
+            with open(built_in, "w") as f:
+                f.write("built in\n")
+            with open(target, "w") as f:
+                f.write("target\n")
+            os.symlink("target.c", link)
+            os.makedirs(os.path.dirname(ignored_src))
+            with open(ignored_src, "w") as f:
+                f.write("ignored\n")
+            os.makedirs(os.path.dirname(ignored_native_src))
+            with open(ignored_native_src, "w") as f:
+                f.write("ignored native\n")
+            with open(second, "w") as f:
+                f.write("second\n")
+
+            empty_dir = os.path.join(pkgd, debugsrcdir.lstrip("/"),
+                                     "empty", "dir")
+            os.makedirs(empty_dir)
+
+            sources = [
+                os.path.join(debugsrcdir, "nested", "normal.c"),
+                os.path.join(debugsrcdir, "nested", "name with ; spaces.c"),
+                os.path.join(debugsrcdir, "nested", "-leading-dash.c"),
+                os.path.join(debugsrcdir, "nested", "recipe-sysroot-file.c"),
+                os.path.join(debugsrcdir, "nested", "compiler<internal>"),
+                os.path.join(debugsrcdir, "nested", "compiler<built-in>"),
+                os.path.join(debugsrcdir, "nested", "link.c"),
+                os.path.join(debugsrcdir, "recipe-sysroot", "ignored.c"),
+                os.path.join(debugsrcdir, "foo-recipe-sysroot-native",
+                             "ignored.c"),
+                os.path.join(second_debugsrcdir, "second.c"),
+            ]
+            d = FakeDataStore({
+                "WORKDIR": workdir,
+                "PKGD": pkgd,
+                "STRIP": "strip",
+                "OBJCOPY": "objcopy",
+                "S": os.path.join(workdir, "source"),
+                "CFLAGS": (
+                    "-ffile-prefix-map=%s=%s "
+                    "-ffile-prefix-map=%s=%s"
+                ) % (
+                    source_root,
+                    debugsrcdir,
+                    second_source_root,
+                    second_debugsrcdir,
+                ),
+            })
+
+            copydebugsources(debugsrcdir, sources, d)
+
+            copied_normal = os.path.join(pkgd, debugsrcdir.lstrip("/"),
+                                         "nested", "normal.c")
+            copied_special = os.path.join(pkgd, debugsrcdir.lstrip("/"),
+                                          "nested", "name with ; spaces.c")
+            copied_leading_dash = os.path.join(pkgd, debugsrcdir.lstrip("/"),
+                                               "nested", "-leading-dash.c")
+            copied_recipe_sysroot_basename = os.path.join(
+                pkgd, debugsrcdir.lstrip("/"), "nested", "recipe-sysroot-file.c")
+            copied_internal = os.path.join(pkgd, debugsrcdir.lstrip("/"),
+                                           "nested", "compiler<internal>")
+            copied_built_in = os.path.join(pkgd, debugsrcdir.lstrip("/"),
+                                           "nested", "compiler<built-in>")
+            copied_link = os.path.join(pkgd, debugsrcdir.lstrip("/"),
+                                       "nested", "link.c")
+            copied_second = os.path.join(pkgd, second_debugsrcdir.lstrip("/"),
+                                         "second.c")
+            ignored = os.path.join(pkgd, debugsrcdir.lstrip("/"),
+                                   "recipe-sysroot", "ignored.c")
+            ignored_native = os.path.join(pkgd, debugsrcdir.lstrip("/"),
+                                          "foo-recipe-sysroot-native",
+                                          "ignored.c")
+
+            with open(copied_normal) as f:
+                self.assertEqual(f.read(), "normal\n")
+            with open(copied_special) as f:
+                self.assertEqual(f.read(), "special\n")
+            with open(copied_leading_dash) as f:
+                self.assertEqual(f.read(), "leading dash\n")
+            with open(copied_recipe_sysroot_basename) as f:
+                self.assertEqual(f.read(), "recipe sysroot basename\n")
+            with open(copied_link) as f:
+                self.assertEqual(f.read(), "target\n")
+            with open(copied_second) as f:
+                self.assertEqual(f.read(), "second\n")
+            self.assertFalse(os.path.islink(copied_link))
+            self.assertFalse(os.path.exists(copied_internal))
+            self.assertFalse(os.path.exists(copied_built_in))
+            self.assertFalse(os.path.exists(ignored))
+            self.assertFalse(os.path.exists(ignored_native))
+            self.assertFalse(os.path.exists(empty_dir))
+
+    def test_copydebugsources_ignores_copy_failure(self):
+        with tempfile.TemporaryDirectory(prefix="oe-test-package-") as tmpdir:
+            source_root = os.path.join(tmpdir, "source")
+            workdir = os.path.join(tmpdir, "work")
+            pkgd = os.path.join(tmpdir, "pkgd")
+            debugsrcdir = "/usr/src/debug/testpkg/1.0"
+            mapped_debugsrcdir = os.path.join(debugsrcdir, "mapped")
+
+            os.makedirs(source_root)
+            os.makedirs(workdir)
+            os.makedirs(pkgd)
+
+            sources = [
+                os.path.join(mapped_debugsrcdir, "missing.c"),
+            ]
+            d = FakeDataStore({
+                "WORKDIR": workdir,
+                "PKGD": pkgd,
+                "STRIP": "strip",
+                "OBJCOPY": "objcopy",
+                "S": os.path.join(workdir, "source"),
+                "CFLAGS": "-ffile-prefix-map=%s=%s" % (
+                    source_root,
+                    mapped_debugsrcdir,
+                ),
+            })
+
+            copydebugsources(debugsrcdir, sources, d)
+
+            self.assertFalse(os.path.exists(pkgd + mapped_debugsrcdir))
+            self.assertFalse(os.path.exists(pkgd + debugsrcdir))
+
+    def test_copydebugsources_moves_externalsrc_relocation(self):
+        with tempfile.TemporaryDirectory(prefix="oe-test-package-") as tmpdir:
+            source_root = os.path.join(tmpdir, "external-[source]")
+            workdir = os.path.join(tmpdir, "work")
+            pkgd = os.path.join(tmpdir, "pkgd")
+            debugsrcdir = "/usr/src/debug/testpkg/1.0"
+
+            os.makedirs(source_root)
+            os.makedirs(workdir)
+            os.makedirs(pkgd)
+
+            source = os.path.join(source_root, "real.c")
+            with open(source, "w") as f:
+                f.write("real\n")
+
+            relocation = pkgd + debugsrcdir + source_root
+            relocated_name = "-relocated with ; spaces.c"
+            os.makedirs(relocation)
+            with open(os.path.join(relocation, relocated_name), "w") as f:
+                f.write("relocated\n")
+            with open(os.path.join(pkgd + debugsrcdir, relocated_name), "w") as f:
+                f.write("old\n")
+
+            sources = [os.path.join(debugsrcdir, "real.c")]
+            d = FakeDataStore({
+                "WORKDIR": workdir,
+                "PKGD": pkgd,
+                "STRIP": "strip",
+                "OBJCOPY": "objcopy",
+                "S": source_root,
+                "CFLAGS": "-ffile-prefix-map=%s=%s" % (source_root, debugsrcdir),
+            })
+
+            copydebugsources(debugsrcdir, sources, d)
+
+            copied_source = os.path.join(pkgd, debugsrcdir.lstrip("/"), "real.c")
+            moved_source = os.path.join(pkgd, debugsrcdir.lstrip("/"), relocated_name)
+
+            with open(copied_source) as f:
+                self.assertEqual(f.read(), "real\n")
+            with open(moved_source) as f:
+                self.assertEqual(f.read(), "relocated\n")
+            self.assertFalse(os.path.exists(relocation))
+
+    def test_copydebugsources_ignores_empty_externalsrc_relocation(self):
+        with tempfile.TemporaryDirectory(prefix="oe-test-package-") as tmpdir:
+            source_root = os.path.join(tmpdir, "external-source")
+            workdir = os.path.join(tmpdir, "work")
+            pkgd = os.path.join(tmpdir, "pkgd")
+            debugsrcdir = "/usr/src/debug/testpkg/1.0"
+
+            os.makedirs(source_root)
+            os.makedirs(workdir)
+            os.makedirs(pkgd)
+
+            source = os.path.join(source_root, "real.c")
+            with open(source, "w") as f:
+                f.write("real\n")
+
+            relocation = pkgd + debugsrcdir + source_root
+            os.makedirs(relocation)
+
+            sources = [os.path.join(debugsrcdir, "real.c")]
+            d = FakeDataStore({
+                "WORKDIR": workdir,
+                "PKGD": pkgd,
+                "STRIP": "strip",
+                "OBJCOPY": "objcopy",
+                "S": source_root,
+                "CFLAGS": "-ffile-prefix-map=%s=%s" % (source_root, debugsrcdir),
+            })
+
+            copydebugsources(debugsrcdir, sources, d)
+
+            copied_source = os.path.join(pkgd, debugsrcdir.lstrip("/"), "real.c")
+
+            with open(copied_source) as f:
+                self.assertEqual(f.read(), "real\n")
+            self.assertFalse(os.path.exists(relocation))


^ permalink raw reply related	[flat|nested] 4+ messages in thread

* Re: [OE-core] [PATCH v2 1/2] package: replace copydebugsources shell pipelines
  2026-06-18  7:19 ` [PATCH v2 1/2] " Anders Heimer
@ 2026-06-18 14:02   ` Paul Barker
  0 siblings, 0 replies; 4+ messages in thread
From: Paul Barker @ 2026-06-18 14:02 UTC (permalink / raw)
  To: Anders Heimer, openembedded-core

[-- Attachment #1: Type: text/plain, Size: 4828 bytes --]

On Thu, 2026-06-18 at 09:19 +0200, Anders Heimer wrote:
> Replace the sort/grep/sed parts of copydebugsources with Python
> filtering on the NUL-separated source list. Keep cpio since it is
> faster than doing it in python.
> 
> Use an explicit prefix + "/" match before stripping the prefix so source
> selection is limited to files in the mapped debug source directory.
> 
> Replace the find/sed symlink fixup pipeline with os.walk() plus cpio,
> use an argv-list mv for the externalsrc relocation, and pass the
> empty-directory find command as an argv list.
> 
> The first cpio copy pass continues to ignore failures as before since
> some inputs are expected to fail. The symlink fixup copy still reports
> cpio failures.
> 
> Signed-off-by: Anders Heimer <anders.heimer@est.tech>
> ---
>  meta/lib/oe/package.py | 70 +++++++++++++++++++++++++++++-------------
>  1 file changed, 48 insertions(+), 22 deletions(-)
> 
> diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
> index c375acc124..c4ad364b64 100644
> --- a/meta/lib/oe/package.py
> +++ b/meta/lib/oe/package.py
> @@ -1017,26 +1017,48 @@ def copydebugsources(debugsrcdir, sources, d):
>          bb.utils.mkdirhier(basepath)
>          cpath.updatecache(basepath)
>  
> -        for pmap in prefixmap:
> -            # Ignore files from the recipe sysroots (target and native)
> -            cmd =  "LC_ALL=C ; sort -z -u '%s' | egrep -v -z '((<internal>|<built-in>)$|/.*recipe-sysroot.*/)' | " % sourcefile
> -            # We need to ignore files that are not actually ours
> -            # we do this by only paying attention to items from this package
> -            cmd += "fgrep -zw '%s' | " % prefixmap[pmap]
> -            # Remove prefix in the source paths
> -            cmd += "sed 's#%s/##g' | " % (prefixmap[pmap])
> -            cmd += "(cd '%s' ; cpio -pd0mlLu --no-preserve-owner '%s%s' 2>/dev/null)" % (pmap, dvar, prefixmap[pmap])
> +        # Ignore files from the recipe sysroots (target and native), and
> +        # compiler internal entries.
> +        with open(sourcefile, "rb") as f:
> +            sourcepaths = sorted({path for path in f.read().split(b"\0")
> +                                  if path
> +                                  and not path.endswith((b"<internal>", b"<built-in>"))
> +                                  and b"recipe-sysroot" not in os.path.dirname(path)})
> +
> +        for pmap, prefix in prefixmap.items():
> +            dstroot = dvar + prefix
> +            prefix_slash = os.fsencode(prefix) + b"/"
> +            relpaths = [path[len(prefix_slash):]
> +                        for path in sourcepaths
> +                        if path.startswith(prefix_slash)]
> +
> +            if relpaths:
> +                subprocess.run(["cpio", "-pd0mlLu", "--no-preserve-owner", dstroot],
> +                               input=b"\0".join(relpaths) + b"\0",
> +                               cwd=pmap, stdout=subprocess.DEVNULL,
> +                               stderr=subprocess.DEVNULL, check=False)

I feel like there are a couple of cases of doing a bit too much at once
here. It took me a few looks at it to realise that a set comprehension
is used to populate sourcepaths, so it's removing duplicate entries.

I would find the following more readable:

    with open(sourcefile, "rb") as f:
        rawpaths = f.read().split(b"\0")

    # Ignore files from the recipe sysroots (target and native), and
    # compiler internal entries. Use a set comprehension to prevent
    # duplicate entries.
    sourcepaths = {path for path in rawpaths
                   if path
                   and not path.endswith((b"<internal>", b"<built-in>"))
                   and b"recipe-sysroot" not in os.path.dirname(path)}

    for pmap, prefix in prefixmap.items():
        dstroot = dvar + prefix
        prefix_slash = os.fsencode(prefix) + b"/"
        relpaths = [path.removeprefix(prefix_slash) for path in sourcepaths
                    if path.startswith(prefix_slash)]

        if relpaths:
            subprocess.run(["cpio", "-pd0mlLu", "--no-preserve-owner", dstroot],
                           input=b"\0".join(sorted(relpaths)) + b"\0",
                           cwd=pmap, stdout=subprocess.DEVNULL,
                           stderr=subprocess.DEVNULL, check=False)

Our minimum Python version is 3.9 so we can use removeprefix(). And
placing the sorted() call in the cpio invocation matches what we do
again later in the function.

I assume that path.removeprefix(prefix_slash) can never result in a
zero-length string because we'll never see an entry in sourcefile
exactly matching the prefix with the trailing '/'.

The rest of the patch LGTM!

Best regards,

-- 
Paul Barker


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-06-18 14:03 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-06-18  7:19 [PATCH v2 0/2] package: replace copydebugsources shell pipelines Anders Heimer
2026-06-18  7:19 ` [PATCH v2 1/2] " Anders Heimer
2026-06-18 14:02   ` [OE-core] " Paul Barker
2026-06-18  7:19 ` [PATCH v2 2/2] oeqa/selftest: add copydebugsources tests Anders Heimer

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox