Re: [OE-core] [PATCH 1/2] package: replace copydebugsources shell pipelines with Popen

[Paul Barker <paul@pbarker.dev>]

[-- Attachment #1: Type: text/plain, Size: 3082 bytes --]

On Tue, 2026-06-16 at 15:35 +0200, Anders Heimer via
lists.openembedded.org wrote:
> Hi Paul,
> 
> On 6/16/26 14:12, Paul Barker wrote:
> > On Tue, 2026-06-16 at 10:25 +0200, Anders Heimer wrote:
> > > -        for pmap in prefixmap:
> > > +        env = os.environ.copy()
> > > +        env["LC_ALL"] = "C"
> > > +
> > > +        for pmap, prefix in prefixmap.items():
> > > +            dstroot = dvar + prefix
> > >               # Ignore files from the recipe sysroots (target and native)
> > > -            cmd =  "LC_ALL=C ; sort -z -u '%s' | egrep -v -z '((<internal>|<built-in>)$|/.*recipe-sysroot.*/)' | " % sourcefile
> > > +            sort_p = subprocess.Popen(["sort", "-z", "-u", "--", sourcefile], stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, env=env)
> > > +            egrep_p = subprocess.Popen(["egrep", "-v", "-z", "-e", r"((<internal>|<built-in>)$|/.*recipe-sysroot.*/)"], stdin=sort_p.stdout, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, env=env)
> > > +            sort_p.stdout.close()
> > > +
> > >               # We need to ignore files that are not actually ours
> > >               # we do this by only paying attention to items from this package
> > > -            cmd += "fgrep -zw '%s' | " % prefixmap[pmap]
> > > +            fgrep_p = subprocess.Popen(["fgrep", "-zw", "-e", prefix], stdin=egrep_p.stdout, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, env=env)
> > > +            egrep_p.stdout.close()
> > > +
> > >               # Remove prefix in the source paths
> > > -            cmd += "sed 's#%s/##g' | " % (prefixmap[pmap])
> > > -            cmd += "(cd '%s' ; cpio -pd0mlLu --no-preserve-owner '%s%s' 2>/dev/null)" % (pmap, dvar, prefixmap[pmap])
> > > +            sed_p = subprocess.Popen(["sed", "s#%s/##g" % prefix], stdin=fgrep_p.stdout, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, env=env)
> > > +            fgrep_p.stdout.close()
> > > +
> > > +            cpio_p = subprocess.Popen(["cpio", "-pd0mlLu", "--no-preserve-owner", dstroot], stdin=sed_p.stdout, cwd=pmap, stderr=subprocess.DEVNULL, env=env)
> > > +            sed_p.stdout.close()
> > > +
> > > +            for proc in (cpio_p, sed_p, fgrep_p, egrep_p, sort_p):
> > > +                proc.wait()
> > Hi Anders, thanks for the patches!
> > 
> > If we're reworking this code, I think we should replace the complex
> > sed/grep/sort pipeline with Python code. We can read into a Python list
> > and sort/filter using the Python standard library, then pass the results
> > to cpio.
> 
> Thank you,  I am very happy to implement this approach instead. I 
> strongly agree with all your comments.

Hi Anders,

I was discussing this with Richard just now, I may be a bit over eager!

This code is performance sensitive as there's sometimes a lot of debug
sources to copy. We've seen previously that shutil.copy() is much slower
than shelling out to `mv`, so we probably want to keep that one as a
subprocess. The rest of the cleanup I suggested is worth doing.

Thanks,

-- 
Paul Barker


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]