[PATCH 0/3] Split split_and_strip_files()

[PATCH 0/3] Split split_and_strip_files()

[Peter Kjellerstedt]

This is my third attempt to make it possible for me to hook into the
split_and_strip_files() function. We need this to be able to rename
the debug files for some packages to avoid conflicts when installing
them.

In this version, the first patch splits out the part of
split_and_strip_files() that is responsible for setting up the
variables used to implement the various debug splitting methods
specified by PACKAGE_DEBUG_SPLIT_STYLE into a new function,
package_debug_vars(). This I believe to be a good thing in itself as
that function is way too long anyway.

The second and third patch are not needed to allow me to hook into the
split_and_strip_files() function, but they simplify the APIs used for
passing the variables returned by package_debug_vars() around.

With these patches in place, I can now wrap the package_debug_vars()
function with the following local wrapper package.bbclass:

require ${COREBASE}/meta/classes/package.bbclass

PACKAGE_DEBUG_FILE_SUFFIX ??= ""

# Provide the original package_debug_vars() as org_package_debug_vars().
org_package_debug_vars := "${@d.getVar('package_debug_vars', False).replace('package_debug_vars', 'org_package_debug_vars')}"
python () {
    bb.methodpool.insert_method("", d.getVar("org_package_debug_vars", False), "<code>", 0)
}

# This version of package_debug_vars() wraps the original package_debug_vars()
# by calling org_package_debug_vars() and then appends PACKAGE_DEBUG_FILE_SUFFIX
# to dv["append"] and dv["staticappend"].
def package_debug_vars(d):
    dv = org_package_debug_vars(d)

    dv["append"] += d.getVar('PACKAGE_DEBUG_FILE_SUFFIX')
    dv["staticappend"] += d.getVar('PACKAGE_DEBUG_FILE_SUFFIX')

    return dv

The definition of the org_package_debug_vars() function is not very
pretty, but it seems to work, and this solution means a heck of a lot
less maintenance than to try to maintain a backported version of the
over 250 lines long split_and_strip_files() function (as it was before
my changes).

//Peter

The following changes since commit 74a8a74a553a33dc5f41939f8070d75e6d57d3da:

  busybox: Add shell arithmetic to work with poky-tiny (2022-02-01 07:32:08 +0000)

are available in the Git repository at:

  git://git.yoctoproject.org/poky-contrib pkj/package_debug_vars
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=pkj/package_debug_vars

Peter Kjellerstedt (3):
  package.bbclass: Split out package_debug_vars from
    split_and_strip_files
  package.bbclass: Make package_debug_vars() return a dict
  package.bbclass: Pass dv (debug_vars) around instead of individual
    vars

 meta/classes/package.bbclass | 127 +++++++++++++++++++----------------
 1 file changed, 70 insertions(+), 57 deletions(-)

[PATCH 1/3] package.bbclass: Split out package_debug_vars from split_and_strip_files

[Peter Kjellerstedt]

The split_and_strip_files() function is big and hard to follow. This
takes a small step to improve that by splitting out the part that sets
up the variables used to implement the various debug splitting methods
specified by PACKAGE_DEBUG_SPLIT_STYLE.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/package.bbclass | 29 ++++++++++++++++++-----------
 1 file changed, 18 insertions(+), 11 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 4927fb99ff..545471468c 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1065,17 +1065,7 @@ python fixup_perms () {
                     fix_perms(each_file, fs_perms_table[dir].fmode, fs_perms_table[dir].fuid, fs_perms_table[dir].fgid, dir)
 }
 
-python split_and_strip_files () {
-    import stat, errno
-    import subprocess
-
-    dvar = d.getVar('PKGD')
-    pn = d.getVar('PN')
-    hostos = d.getVar('HOST_OS')
-
-    oldcwd = os.getcwd()
-    os.chdir(dvar)
-
+def package_debug_vars(d):
     # We default to '.debug' style
     if d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-file-directory':
         # Single debug-file-directory style debug info
@@ -1113,6 +1103,23 @@ python split_and_strip_files () {
         debugstaticlibdir = ""
         debugsrcdir = "/usr/src/debug"
 
+    return (debugappend, debugstaticappend, debugdir, debugstaticdir,
+            debuglibdir, debugstaticlibdir, debugsrcdir)
+
+python split_and_strip_files () {
+    import stat, errno
+    import subprocess
+
+    dvar = d.getVar('PKGD')
+    pn = d.getVar('PN')
+    hostos = d.getVar('HOST_OS')
+
+    oldcwd = os.getcwd()
+    os.chdir(dvar)
+
+    debugappend, debugstaticappend, debugdir, debugstaticdir, \
+    debuglibdir, debugstaticlibdir, debugsrcdir = package_debug_vars(d)
+
     #
     # First lets figure out all of the files we may have to process ... do this only once!
     #

[PATCH 2/3] package.bbclass: Make package_debug_vars() return a dict

[Peter Kjellerstedt]

It simplifies the API to return one dict instead of seven strings.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/package.bbclass | 96 +++++++++++++++++++-----------------
 1 file changed, 51 insertions(+), 45 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 545471468c..9e895b4ecc 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1069,42 +1069,49 @@ def package_debug_vars(d):
     # We default to '.debug' style
     if d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-file-directory':
         # Single debug-file-directory style debug info
-        debugappend = ".debug"
-        debugstaticappend = ""
-        debugdir = ""
-        debugstaticdir = ""
-        debuglibdir = "/usr/lib/debug"
-        debugstaticlibdir = "/usr/lib/debug-static"
-        debugsrcdir = "/usr/src/debug"
+        debug_vars = {
+            "append": ".debug",
+            "staticappend": "",
+            "dir": "",
+            "staticdir": "",
+            "libdir": "/usr/lib/debug",
+            "staticlibdir": "/usr/lib/debug-static",
+            "srcdir": "/usr/src/debug",
+        }
     elif d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-without-src':
         # Original OE-core, a.k.a. ".debug", style debug info, but without sources in /usr/src/debug
-        debugappend = ""
-        debugstaticappend = ""
-        debugdir = "/.debug"
-        debugstaticdir = "/.debug-static"
-        debuglibdir = ""
-        debugstaticlibdir = ""
-        debugsrcdir = ""
+        debug_vars = {
+            "append": "",
+            "staticappend": "",
+            "dir": "/.debug",
+            "staticdir": "/.debug-static",
+            "libdir": "",
+            "staticlibdir": "",
+            "srcdir": "",
+        }
     elif d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-with-srcpkg':
-        debugappend = ""
-        debugstaticappend = ""
-        debugdir = "/.debug"
-        debugstaticdir = "/.debug-static"
-        debuglibdir = ""
-        debugstaticlibdir = ""
-        debugsrcdir = "/usr/src/debug"
+        debug_vars = {
+            "append": "",
+            "staticappend": "",
+            "dir": "/.debug",
+            "staticdir": "/.debug-static",
+            "libdir": "",
+            "staticlibdir": "",
+            "srcdir": "/usr/src/debug",
+        }
     else:
         # Original OE-core, a.k.a. ".debug", style debug info
-        debugappend = ""
-        debugstaticappend = ""
-        debugdir = "/.debug"
-        debugstaticdir = "/.debug-static"
-        debuglibdir = ""
-        debugstaticlibdir = ""
-        debugsrcdir = "/usr/src/debug"
+        debug_vars = {
+            "append": "",
+            "staticappend": "",
+            "dir": "/.debug",
+            "staticdir": "/.debug-static",
+            "libdir": "",
+            "staticlibdir": "",
+            "srcdir": "/usr/src/debug",
+        }
 
-    return (debugappend, debugstaticappend, debugdir, debugstaticdir,
-            debuglibdir, debugstaticlibdir, debugsrcdir)
+    return debug_vars
 
 python split_and_strip_files () {
     import stat, errno
@@ -1117,8 +1124,7 @@ python split_and_strip_files () {
     oldcwd = os.getcwd()
     os.chdir(dvar)
 
-    debugappend, debugstaticappend, debugdir, debugstaticdir, \
-    debuglibdir, debugstaticlibdir, debugsrcdir = package_debug_vars(d)
+    dv = package_debug_vars(d)
 
     #
     # First lets figure out all of the files we may have to process ... do this only once!
@@ -1139,9 +1145,9 @@ python split_and_strip_files () {
                 file = os.path.join(root, f)
 
                 # Skip debug files
-                if debugappend and file.endswith(debugappend):
+                if dv["append"] and file.endswith(dv["append"]):
                     continue
-                if debugdir and debugdir in os.path.dirname(file[len(dvar):]):
+                if dv["dir"] and dv["dir"] in os.path.dirname(file[len(dvar):]):
                     continue
 
                 if file in skipfiles:
@@ -1238,11 +1244,11 @@ python split_and_strip_files () {
     # First lets process debug splitting
     #
     if (d.getVar('INHIBIT_PACKAGE_DEBUG_SPLIT') != '1'):
-        results = oe.utils.multiprocess_launch(splitdebuginfo, list(elffiles), d, extraargs=(dvar, debugdir, debuglibdir, debugappend, debugsrcdir, d))
+        results = oe.utils.multiprocess_launch(splitdebuginfo, list(elffiles), d, extraargs=(dvar, dv["dir"], dv["libdir"], dv["append"], dv["srcdir"], d))
 
-        if debugsrcdir and not hostos.startswith("mingw"):
+        if dv["srcdir"] and not hostos.startswith("mingw"):
             if (d.getVar('PACKAGE_DEBUG_STATIC_SPLIT') == '1'):
-                results = oe.utils.multiprocess_launch(splitstaticdebuginfo, staticlibs, d, extraargs=(dvar, debugstaticdir, debugstaticlibdir, debugstaticappend, debugsrcdir, d))
+                results = oe.utils.multiprocess_launch(splitstaticdebuginfo, staticlibs, d, extraargs=(dvar, dv["staticdir"], dv["staticlibdir"], dv["staticappend"], dv["srcdir"], d))
             else:
                 for file in staticlibs:
                     results.append( (file,source_info(file, d)) )
@@ -1261,9 +1267,9 @@ python split_and_strip_files () {
             target = inodes[ref][0][len(dvar):]
             for file in inodes[ref][1:]:
                 src = file[len(dvar):]
-                dest = debuglibdir + os.path.dirname(src) + debugdir + "/" + os.path.basename(target) + debugappend
+                dest = dv["libdir"] + os.path.dirname(src) + dv["dir"] + "/" + os.path.basename(target) + dv["append"]
                 fpath = dvar + dest
-                ftarget = dvar + debuglibdir + os.path.dirname(target) + debugdir + "/" + os.path.basename(target) + debugappend
+                ftarget = dvar + dv["libdir"] + os.path.dirname(target) + dv["dir"] + "/" + os.path.basename(target) + dv["append"]
                 bb.utils.mkdirhier(os.path.dirname(fpath))
                 # Only one hardlink of separated debug info file in each directory
                 if not os.access(fpath, os.R_OK):
@@ -1273,7 +1279,7 @@ python split_and_strip_files () {
         # Create symlinks for all cases we were able to split symbols
         for file in symlinks:
             src = file[len(dvar):]
-            dest = debuglibdir + os.path.dirname(src) + debugdir + "/" + os.path.basename(src) + debugappend
+            dest = dv["libdir"] + os.path.dirname(src) + dv["dir"] + "/" + os.path.basename(src) + dv["append"]
             fpath = dvar + dest
             # Skip it if the target doesn't exist
             try:
@@ -1289,17 +1295,17 @@ python split_and_strip_files () {
             lbase = os.path.basename(ltarget)
             ftarget = ""
             if lpath and lpath != ".":
-                ftarget += lpath + debugdir + "/"
-            ftarget += lbase + debugappend
+                ftarget += lpath + dv["dir"] + "/"
+            ftarget += lbase + dv["append"]
             if lpath.startswith(".."):
                 ftarget = os.path.join("..", ftarget)
             bb.utils.mkdirhier(os.path.dirname(fpath))
             #bb.note("Symlink %s -> %s" % (fpath, ftarget))
             os.symlink(ftarget, fpath)
 
-        # Process the debugsrcdir if requested...
+        # Process the dv["srcdir"] if requested...
         # This copies and places the referenced sources for later debugging...
-        copydebugsources(debugsrcdir, sources, d)
+        copydebugsources(dv["srcdir"], sources, d)
     #
     # End of debug splitting
     #
@@ -1323,7 +1329,7 @@ python split_and_strip_files () {
     # Build "minidebuginfo" and reinject it back into the stripped binaries
     if d.getVar('PACKAGE_MINIDEBUGINFO') == '1':
         oe.utils.multiprocess_launch(inject_minidebuginfo, list(elffiles), d,
-                                     extraargs=(dvar, debugdir, debuglibdir, debugappend, debugsrcdir, d))
+                                     extraargs=(dvar, dv["dir"], dv["libdir"], dv["append"], dv["srcdir"], d))
 
     #
     # End of strip

[PATCH 3/3] package.bbclass: Pass dv (debug_vars) around instead of individual vars

[Peter Kjellerstedt]

This simplifies the APIs for splitdebuginfo(), splitstaticdebuginfo()
and inject_minidebuginfo()

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/classes/package.bbclass | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 9e895b4ecc..f4a661ba25 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -367,7 +367,7 @@ def source_info(file, d, fatal=True):
 
     return list(debugsources)
 
-def splitdebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir, d):
+def splitdebuginfo(file, dvar, dv, d):
     # Function to split a single file into two components, one is the stripped
     # target system binary, the other contains any debugging information. The
     # two files are linked to reference each other.
@@ -378,7 +378,7 @@ def splitdebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir,
     import subprocess
 
     src = file[len(dvar):]
-    dest = debuglibdir + os.path.dirname(src) + debugdir + "/" + os.path.basename(src) + debugappend
+    dest = dv["libdir"] + os.path.dirname(src) + dv["dir"] + "/" + os.path.basename(src) + dv["append"]
     debugfile = dvar + dest
     sources = []
 
@@ -397,7 +397,7 @@ def splitdebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir,
         os.chmod(file, newmode)
 
     # We need to extract the debug src information here...
-    if debugsrcdir:
+    if dv["srcdir"]:
         sources = source_info(file, d)
 
     bb.utils.mkdirhier(os.path.dirname(debugfile))
@@ -412,7 +412,7 @@ def splitdebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir,
 
     return (file, sources)
 
-def splitstaticdebuginfo(file, dvar, debugstaticdir, debugstaticlibdir, debugstaticappend, debugsrcdir, d):
+def splitstaticdebuginfo(file, dvar, dv, d):
     # Unlike the function above, there is no way to split a static library
     # two components.  So to get similar results we will copy the unmodified
     # static library (containing the debug symbols) into a new directory.
@@ -425,7 +425,7 @@ def splitstaticdebuginfo(file, dvar, debugstaticdir, debugstaticlibdir, debugsta
     import shutil
 
     src = file[len(dvar):]
-    dest = debugstaticlibdir + os.path.dirname(src) + debugstaticdir + "/" + os.path.basename(src) + debugstaticappend
+    dest = dv["staticlibdir"] + os.path.dirname(src) + dv["staticdir"] + "/" + os.path.basename(src) + dv["staticappend"]
     debugfile = dvar + dest
     sources = []
 
@@ -442,7 +442,7 @@ def splitstaticdebuginfo(file, dvar, debugstaticdir, debugstaticlibdir, debugsta
         os.chmod(file, newmode)
 
     # We need to extract the debug src information here...
-    if debugsrcdir:
+    if dv["srcdir"]:
         sources = source_info(file, d)
 
     bb.utils.mkdirhier(os.path.dirname(debugfile))
@@ -455,7 +455,7 @@ def splitstaticdebuginfo(file, dvar, debugstaticdir, debugstaticlibdir, debugsta
 
     return (file, sources)
 
-def inject_minidebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsrcdir, d):
+def inject_minidebuginfo(file, dvar, dv, d):
     # Extract just the symbols from debuginfo into minidebuginfo,
     # compress it with xz and inject it back into the binary in a .gnu_debugdata section.
     # https://sourceware.org/gdb/onlinedocs/gdb/MiniDebugInfo.html
@@ -469,7 +469,7 @@ def inject_minidebuginfo(file, dvar, debugdir, debuglibdir, debugappend, debugsr
     minidebuginfodir = d.expand('${WORKDIR}/minidebuginfo')
 
     src = file[len(dvar):]
-    dest = debuglibdir + os.path.dirname(src) + debugdir + "/" + os.path.basename(src) + debugappend
+    dest = dv["libdir"] + os.path.dirname(src) + dv["dir"] + "/" + os.path.basename(src) + dv["append"]
     debugfile = dvar + dest
     minidebugfile = minidebuginfodir + src + '.minidebug'
     bb.utils.mkdirhier(os.path.dirname(minidebugfile))
@@ -1244,11 +1244,11 @@ python split_and_strip_files () {
     # First lets process debug splitting
     #
     if (d.getVar('INHIBIT_PACKAGE_DEBUG_SPLIT') != '1'):
-        results = oe.utils.multiprocess_launch(splitdebuginfo, list(elffiles), d, extraargs=(dvar, dv["dir"], dv["libdir"], dv["append"], dv["srcdir"], d))
+        results = oe.utils.multiprocess_launch(splitdebuginfo, list(elffiles), d, extraargs=(dvar, dv, d))
 
         if dv["srcdir"] and not hostos.startswith("mingw"):
             if (d.getVar('PACKAGE_DEBUG_STATIC_SPLIT') == '1'):
-                results = oe.utils.multiprocess_launch(splitstaticdebuginfo, staticlibs, d, extraargs=(dvar, dv["staticdir"], dv["staticlibdir"], dv["staticappend"], dv["srcdir"], d))
+                results = oe.utils.multiprocess_launch(splitstaticdebuginfo, staticlibs, d, extraargs=(dvar, dv, d))
             else:
                 for file in staticlibs:
                     results.append( (file,source_info(file, d)) )
@@ -1329,7 +1329,7 @@ python split_and_strip_files () {
     # Build "minidebuginfo" and reinject it back into the stripped binaries
     if d.getVar('PACKAGE_MINIDEBUGINFO') == '1':
         oe.utils.multiprocess_launch(inject_minidebuginfo, list(elffiles), d,
-                                     extraargs=(dvar, dv["dir"], dv["libdir"], dv["append"], dv["srcdir"], d))
+                                     extraargs=(dvar, dv, d))
 
     #
     # End of strip

Re: [OE-core] [PATCH 0/3] Split split_and_strip_files()

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 3546 bytes --]

Sorry Peter, but this still strikes me as rather heavy handed for what is a
niche use case. Is it possible to simply do what needs to be done from a
ROOTFS_POSTPROCESS hook?

Alex

On Wed, 2 Feb 2022 at 03:00, Peter Kjellerstedt <peter.kjellerstedt@axis.com>
wrote:

> This is my third attempt to make it possible for me to hook into the
> split_and_strip_files() function. We need this to be able to rename
> the debug files for some packages to avoid conflicts when installing
> them.
>
> In this version, the first patch splits out the part of
> split_and_strip_files() that is responsible for setting up the
> variables used to implement the various debug splitting methods
> specified by PACKAGE_DEBUG_SPLIT_STYLE into a new function,
> package_debug_vars(). This I believe to be a good thing in itself as
> that function is way too long anyway.
>
> The second and third patch are not needed to allow me to hook into the
> split_and_strip_files() function, but they simplify the APIs used for
> passing the variables returned by package_debug_vars() around.
>
> With these patches in place, I can now wrap the package_debug_vars()
> function with the following local wrapper package.bbclass:
>
> require ${COREBASE}/meta/classes/package.bbclass
>
> PACKAGE_DEBUG_FILE_SUFFIX ??= ""
>
> # Provide the original package_debug_vars() as org_package_debug_vars().
> org_package_debug_vars := "${@d.getVar('package_debug_vars',
> False).replace('package_debug_vars', 'org_package_debug_vars')}"
> python () {
>     bb.methodpool.insert_method("", d.getVar("org_package_debug_vars",
> False), "<code>", 0)
> }
>
> # This version of package_debug_vars() wraps the original
> package_debug_vars()
> # by calling org_package_debug_vars() and then appends
> PACKAGE_DEBUG_FILE_SUFFIX
> # to dv["append"] and dv["staticappend"].
> def package_debug_vars(d):
>     dv = org_package_debug_vars(d)
>
>     dv["append"] += d.getVar('PACKAGE_DEBUG_FILE_SUFFIX')
>     dv["staticappend"] += d.getVar('PACKAGE_DEBUG_FILE_SUFFIX')
>
>     return dv
>
> The definition of the org_package_debug_vars() function is not very
> pretty, but it seems to work, and this solution means a heck of a lot
> less maintenance than to try to maintain a backported version of the
> over 250 lines long split_and_strip_files() function (as it was before
> my changes).
>
> //Peter
>
> The following changes since commit
> 74a8a74a553a33dc5f41939f8070d75e6d57d3da:
>
>   busybox: Add shell arithmetic to work with poky-tiny (2022-02-01
> 07:32:08 +0000)
>
> are available in the Git repository at:
>
>   git://git.yoctoproject.org/poky-contrib pkj/package_debug_vars
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=pkj/package_debug_vars
>
> Peter Kjellerstedt (3):
>   package.bbclass: Split out package_debug_vars from
>     split_and_strip_files
>   package.bbclass: Make package_debug_vars() return a dict
>   package.bbclass: Pass dv (debug_vars) around instead of individual
>     vars
>
>  meta/classes/package.bbclass | 127 +++++++++++++++++++----------------
>  1 file changed, 70 insertions(+), 57 deletions(-)
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#161174):
> https://lists.openembedded.org/g/openembedded-core/message/161174
> Mute This Topic: https://lists.openembedded.org/mt/88849297/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 4935 bytes --]

RE: [OE-core] [PATCH 0/3] Split split_and_strip_files()

[Peter Kjellerstedt]

[-- Attachment #1: Type: text/plain, Size: 4157 bytes --]

Why do you consider splitting a huge function heavy? Honestly, I would like to split it even further to make it more manageable.

//Peter

From: Alexander Kanavin <alex.kanavin@gmail.com>
Sent: den 2 februari 2022 09:54
To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Cc: OE-core <openembedded-core@lists.openembedded.org>
Subject: Re: [OE-core] [PATCH 0/3] Split split_and_strip_files()

Sorry Peter, but this still strikes me as rather heavy handed for what is a niche use case. Is it possible to simply do what needs to be done from a ROOTFS_POSTPROCESS hook?

Alex

On Wed, 2 Feb 2022 at 03:00, Peter Kjellerstedt <peter.kjellerstedt@axis.com<mailto:peter.kjellerstedt@axis.com>> wrote:
This is my third attempt to make it possible for me to hook into the
split_and_strip_files() function. We need this to be able to rename
the debug files for some packages to avoid conflicts when installing
them.

In this version, the first patch splits out the part of
split_and_strip_files() that is responsible for setting up the
variables used to implement the various debug splitting methods
specified by PACKAGE_DEBUG_SPLIT_STYLE into a new function,
package_debug_vars(). This I believe to be a good thing in itself as
that function is way too long anyway.

The second and third patch are not needed to allow me to hook into the
split_and_strip_files() function, but they simplify the APIs used for
passing the variables returned by package_debug_vars() around.

With these patches in place, I can now wrap the package_debug_vars()
function with the following local wrapper package.bbclass:

require ${COREBASE}/meta/classes/package.bbclass

PACKAGE_DEBUG_FILE_SUFFIX ??= ""

# Provide the original package_debug_vars() as org_package_debug_vars().
org_package_debug_vars := "${@d.getVar('package_debug_vars', False).replace('package_debug_vars', 'org_package_debug_vars')}<mailto:$%7b@d.getVar('package_debug_vars',%20False).replace('package_debug_vars',%20'org_package_debug_vars')%7d>"
python () {
    bb.methodpool.insert_method("", d.getVar("org_package_debug_vars", False), "<code>", 0)
}

# This version of package_debug_vars() wraps the original package_debug_vars()
# by calling org_package_debug_vars() and then appends PACKAGE_DEBUG_FILE_SUFFIX
# to dv["append"] and dv["staticappend"].
def package_debug_vars(d):
    dv = org_package_debug_vars(d)

    dv["append"] += d.getVar('PACKAGE_DEBUG_FILE_SUFFIX')
    dv["staticappend"] += d.getVar('PACKAGE_DEBUG_FILE_SUFFIX')

    return dv

The definition of the org_package_debug_vars() function is not very
pretty, but it seems to work, and this solution means a heck of a lot
less maintenance than to try to maintain a backported version of the
over 250 lines long split_and_strip_files() function (as it was before
my changes).

//Peter

The following changes since commit 74a8a74a553a33dc5f41939f8070d75e6d57d3da:

  busybox: Add shell arithmetic to work with poky-tiny (2022-02-01 07:32:08 +0000)

are available in the Git repository at:

  git://git.yoctoproject.org/poky-contrib<http://git.yoctoproject.org/poky-contrib> pkj/package_debug_vars
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=pkj/package_debug_vars

Peter Kjellerstedt (3):
  package.bbclass: Split out package_debug_vars from
    split_and_strip_files
  package.bbclass: Make package_debug_vars() return a dict
  package.bbclass: Pass dv (debug_vars) around instead of individual
    vars

 meta/classes/package.bbclass | 127 +++++++++++++++++++----------------
 1 file changed, 70 insertions(+), 57 deletions(-)


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161174): https://lists.openembedded.org/g/openembedded-core/message/161174
Mute This Topic: https://lists.openembedded.org/mt/88849297/1686489
Group Owner: openembedded-core+owner@lists.openembedded.org<mailto:openembedded-core%2Bowner@lists.openembedded.org>
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com<mailto:alex.kanavin@gmail.com>]
-=-=-=-=-=-=-=-=-=-=-=-

[-- Attachment #2: Type: text/html, Size: 8174 bytes --]

Re: [OE-core] [PATCH 0/3] Split split_and_strip_files()

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 4303 bytes --]

Right, the patchset is good for maintainability, my comment was more about
whether it's easier to just package files in a way that doesn't clash in
do_rootfs, then rearrange them in rootfs postprocessing.

Alex

On Wed, 2 Feb 2022 at 12:20, Peter Kjellerstedt <peter.kjellerstedt@axis.com>
wrote:

> Why do you consider splitting a huge function heavy? Honestly, I would
> like to split it even further to make it more manageable.
>
>
>
> //Peter
>
>
>
> *From:* Alexander Kanavin <alex.kanavin@gmail.com>
> *Sent:* den 2 februari 2022 09:54
> *To:* Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> *Cc:* OE-core <openembedded-core@lists.openembedded.org>
> *Subject:* Re: [OE-core] [PATCH 0/3] Split split_and_strip_files()
>
>
>
> Sorry Peter, but this still strikes me as rather heavy handed for what is
> a niche use case. Is it possible to simply do what needs to be done from a
> ROOTFS_POSTPROCESS hook?
>
>
>
> Alex
>
>
>
> On Wed, 2 Feb 2022 at 03:00, Peter Kjellerstedt <
> peter.kjellerstedt@axis.com> wrote:
>
> This is my third attempt to make it possible for me to hook into the
> split_and_strip_files() function. We need this to be able to rename
> the debug files for some packages to avoid conflicts when installing
> them.
>
> In this version, the first patch splits out the part of
> split_and_strip_files() that is responsible for setting up the
> variables used to implement the various debug splitting methods
> specified by PACKAGE_DEBUG_SPLIT_STYLE into a new function,
> package_debug_vars(). This I believe to be a good thing in itself as
> that function is way too long anyway.
>
> The second and third patch are not needed to allow me to hook into the
> split_and_strip_files() function, but they simplify the APIs used for
> passing the variables returned by package_debug_vars() around.
>
> With these patches in place, I can now wrap the package_debug_vars()
> function with the following local wrapper package.bbclass:
>
> require ${COREBASE}/meta/classes/package.bbclass
>
> PACKAGE_DEBUG_FILE_SUFFIX ??= ""
>
> # Provide the original package_debug_vars() as org_package_debug_vars().
> org_package_debug_vars := "${@d.getVar('package_debug_vars',
> False).replace('package_debug_vars', 'org_package_debug_vars')}"
> python () {
>     bb.methodpool.insert_method("", d.getVar("org_package_debug_vars",
> False), "<code>", 0)
> }
>
> # This version of package_debug_vars() wraps the original
> package_debug_vars()
> # by calling org_package_debug_vars() and then appends
> PACKAGE_DEBUG_FILE_SUFFIX
> # to dv["append"] and dv["staticappend"].
> def package_debug_vars(d):
>     dv = org_package_debug_vars(d)
>
>     dv["append"] += d.getVar('PACKAGE_DEBUG_FILE_SUFFIX')
>     dv["staticappend"] += d.getVar('PACKAGE_DEBUG_FILE_SUFFIX')
>
>     return dv
>
> The definition of the org_package_debug_vars() function is not very
> pretty, but it seems to work, and this solution means a heck of a lot
> less maintenance than to try to maintain a backported version of the
> over 250 lines long split_and_strip_files() function (as it was before
> my changes).
>
> //Peter
>
> The following changes since commit
> 74a8a74a553a33dc5f41939f8070d75e6d57d3da:
>
>   busybox: Add shell arithmetic to work with poky-tiny (2022-02-01
> 07:32:08 +0000)
>
> are available in the Git repository at:
>
>   git://git.yoctoproject.org/poky-contrib pkj/package_debug_vars
>
> http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=pkj/package_debug_vars
>
> Peter Kjellerstedt (3):
>   package.bbclass: Split out package_debug_vars from
>     split_and_strip_files
>   package.bbclass: Make package_debug_vars() return a dict
>   package.bbclass: Pass dv (debug_vars) around instead of individual
>     vars
>
>  meta/classes/package.bbclass | 127 +++++++++++++++++++----------------
>  1 file changed, 70 insertions(+), 57 deletions(-)
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#161174):
> https://lists.openembedded.org/g/openembedded-core/message/161174
> Mute This Topic: https://lists.openembedded.org/mt/88849297/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 7669 bytes --]

RE: [OE-core] [PATCH 0/3] Split split_and_strip_files()

[Peter Kjellerstedt]

[-- Attachment #1: Type: text/plain, Size: 5718 bytes --]

Well, I believe it is better to package the files correctly from the start rather than having to redo the packaging later. The first variant I suggested was two added lines, compared to the ~30 lines we had in a recipe that did it after the fact, which somewhat indicates what it means to do it as part of the packaging process, compared to what it takes to do it later.

I still believe it would be better if the -dbg packages are always created in a way that two -dbg packages cannot clash with each other, since we do not split them the same way we do for target packages. That way everyone would benefit from it and it should lead to less problems. However, with the proposed changes I can solve my immediate problem, and we can continue the discussion if we can improve how -dbg packages are created for everyone separately.

//Peter

From: Alexander Kanavin <alex.kanavin@gmail.com>
Sent: den 2 februari 2022 12:33
To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Cc: OE-core <openembedded-core@lists.openembedded.org>
Subject: Re: [OE-core] [PATCH 0/3] Split split_and_strip_files()

Right, the patchset is good for maintainability, my comment was more about whether it's easier to just package files in a way that doesn't clash in do_rootfs, then rearrange them in rootfs postprocessing.

Alex

On Wed, 2 Feb 2022 at 12:20, Peter Kjellerstedt <peter.kjellerstedt@axis.com<mailto:peter.kjellerstedt@axis.com>> wrote:
Why do you consider splitting a huge function heavy? Honestly, I would like to split it even further to make it more manageable.

//Peter

From: Alexander Kanavin <alex.kanavin@gmail.com<mailto:alex.kanavin@gmail.com>>
Sent: den 2 februari 2022 09:54
To: Peter Kjellerstedt <peter.kjellerstedt@axis.com<mailto:peter.kjellerstedt@axis.com>>
Cc: OE-core <openembedded-core@lists.openembedded.org<mailto:openembedded-core@lists.openembedded.org>>
Subject: Re: [OE-core] [PATCH 0/3] Split split_and_strip_files()

Sorry Peter, but this still strikes me as rather heavy handed for what is a niche use case. Is it possible to simply do what needs to be done from a ROOTFS_POSTPROCESS hook?

Alex

On Wed, 2 Feb 2022 at 03:00, Peter Kjellerstedt <peter.kjellerstedt@axis.com<mailto:peter.kjellerstedt@axis.com>> wrote:
This is my third attempt to make it possible for me to hook into the
split_and_strip_files() function. We need this to be able to rename
the debug files for some packages to avoid conflicts when installing
them.

In this version, the first patch splits out the part of
split_and_strip_files() that is responsible for setting up the
variables used to implement the various debug splitting methods
specified by PACKAGE_DEBUG_SPLIT_STYLE into a new function,
package_debug_vars(). This I believe to be a good thing in itself as
that function is way too long anyway.

The second and third patch are not needed to allow me to hook into the
split_and_strip_files() function, but they simplify the APIs used for
passing the variables returned by package_debug_vars() around.

With these patches in place, I can now wrap the package_debug_vars()
function with the following local wrapper package.bbclass:

require ${COREBASE}/meta/classes/package.bbclass

PACKAGE_DEBUG_FILE_SUFFIX ??= ""

# Provide the original package_debug_vars() as org_package_debug_vars().
org_package_debug_vars := "${@d.getVar('package_debug_vars', False).replace('package_debug_vars', 'org_package_debug_vars')}<mailto:$%7b@d.getVar('package_debug_vars',%20False).replace('package_debug_vars',%20'org_package_debug_vars')%7d>"
python () {
    bb.methodpool.insert_method("", d.getVar("org_package_debug_vars", False), "<code>", 0)
}

# This version of package_debug_vars() wraps the original package_debug_vars()
# by calling org_package_debug_vars() and then appends PACKAGE_DEBUG_FILE_SUFFIX
# to dv["append"] and dv["staticappend"].
def package_debug_vars(d):
    dv = org_package_debug_vars(d)

    dv["append"] += d.getVar('PACKAGE_DEBUG_FILE_SUFFIX')
    dv["staticappend"] += d.getVar('PACKAGE_DEBUG_FILE_SUFFIX')

    return dv

The definition of the org_package_debug_vars() function is not very
pretty, but it seems to work, and this solution means a heck of a lot
less maintenance than to try to maintain a backported version of the
over 250 lines long split_and_strip_files() function (as it was before
my changes).

//Peter

The following changes since commit 74a8a74a553a33dc5f41939f8070d75e6d57d3da:

  busybox: Add shell arithmetic to work with poky-tiny (2022-02-01 07:32:08 +0000)

are available in the Git repository at:

  git://git.yoctoproject.org/poky-contrib<http://git.yoctoproject.org/poky-contrib> pkj/package_debug_vars
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=pkj/package_debug_vars

Peter Kjellerstedt (3):
  package.bbclass: Split out package_debug_vars from
    split_and_strip_files
  package.bbclass: Make package_debug_vars() return a dict
  package.bbclass: Pass dv (debug_vars) around instead of individual
    vars

 meta/classes/package.bbclass | 127 +++++++++++++++++++----------------
 1 file changed, 70 insertions(+), 57 deletions(-)


-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#161174): https://lists.openembedded.org/g/openembedded-core/message/161174
Mute This Topic: https://lists.openembedded.org/mt/88849297/1686489
Group Owner: openembedded-core+owner@lists.openembedded.org<mailto:openembedded-core%2Bowner@lists.openembedded.org>
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com<mailto:alex.kanavin@gmail.com>]
-=-=-=-=-=-=-=-=-=-=-=-

[-- Attachment #2: Type: text/html, Size: 12194 bytes --]

Re: [OE-core] [PATCH 0/3] Split split_and_strip_files()

[Richard Purdie]

On Wed, 2022-02-02 at 02:59 +0100, Peter Kjellerstedt wrote:
> This is my third attempt to make it possible for me to hook into the
> split_and_strip_files() function. We need this to be able to rename
> the debug files for some packages to avoid conflicts when installing
> them.
> 
> In this version, the first patch splits out the part of
> split_and_strip_files() that is responsible for setting up the
> variables used to implement the various debug splitting methods
> specified by PACKAGE_DEBUG_SPLIT_STYLE into a new function,
> package_debug_vars(). This I believe to be a good thing in itself as
> that function is way too long anyway.
> 
> The second and third patch are not needed to allow me to hook into the
> split_and_strip_files() function, but they simplify the APIs used for
> passing the variables returned by package_debug_vars() around.
> 
> With these patches in place, I can now wrap the package_debug_vars()
> function with the following local wrapper package.bbclass:
> 
> require ${COREBASE}/meta/classes/package.bbclass
> 
> PACKAGE_DEBUG_FILE_SUFFIX ??= ""
> 
> # Provide the original package_debug_vars() as org_package_debug_vars().
> org_package_debug_vars := "${@d.getVar('package_debug_vars', False).replace('package_debug_vars', 'org_package_debug_vars')}"
> python () {
>     bb.methodpool.insert_method("", d.getVar("org_package_debug_vars", False), "<code>", 0)
> }
> 
> # This version of package_debug_vars() wraps the original package_debug_vars()
> # by calling org_package_debug_vars() and then appends PACKAGE_DEBUG_FILE_SUFFIX
> # to dv["append"] and dv["staticappend"].
> def package_debug_vars(d):
>     dv = org_package_debug_vars(d)
> 
>     dv["append"] += d.getVar('PACKAGE_DEBUG_FILE_SUFFIX')
>     dv["staticappend"] += d.getVar('PACKAGE_DEBUG_FILE_SUFFIX')
> 
>     return dv
> 
> The definition of the org_package_debug_vars() function is not very
> pretty, but it seems to work, and this solution means a heck of a lot
> less maintenance than to try to maintain a backported version of the
> over 250 lines long split_and_strip_files() function (as it was before
> my changes).
> 
> //Peter
> 
> The following changes since commit 74a8a74a553a33dc5f41939f8070d75e6d57d3da:
> 
>   busybox: Add shell arithmetic to work with poky-tiny (2022-02-01 07:32:08 +0000)
> 
> are available in the Git repository at:
> 
>   git://git.yoctoproject.org/poky-contrib pkj/package_debug_vars
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=pkj/package_debug_vars
> 
> Peter Kjellerstedt (3):
>   package.bbclass: Split out package_debug_vars from
>     split_and_strip_files
>   package.bbclass: Make package_debug_vars() return a dict
>   package.bbclass: Pass dv (debug_vars) around instead of individual
>     vars
> 
>  meta/classes/package.bbclass | 127 +++++++++++++++++++----------------
>  1 file changed, 70 insertions(+), 57 deletions(-)

I know I'm often negative about patches so I just wanted to mention I like this
series for a change! :)

Definitely better than the other patch with all those variables.

Cheers,

Richard