[PATCH] package: fix src packaging path for nativesdk

[PATCH] package: fix src packaging path for nativesdk

[Maxime Roussin-Bélanger]

From: Maxime Roussin-Belanger <maxime.roussinbelanger@gmail.com>

Directory tree doesn't contain the correct structure when packaging
for source files for the sdk. Since the structure is incorrect,
they won't be part of the installed sdk.

Having source files in nativesdk is necessary when using it as a
development sysroot for debugging. Otherwise, when debugging in an
emulation mode it's impossible to step in 3rd party library code.

Signed-off-by: Maxime Roussin-Belanger <maxime.roussinbelanger@gmail.com>
---
 meta/conf/bitbake.conf |  8 ++++----
 meta/lib/oe/package.py | 25 +++++++++++++++++++------
 2 files changed, 23 insertions(+), 10 deletions(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 702881144e..eb438a1499 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -649,10 +649,10 @@ EXTRA_OEMAKE:prepend:task-install = "${PARALLEL_MAKEINST} "
 ##################################################################
 # Beware: applied last to first
 DEBUG_PREFIX_MAP ?= "-fcanon-prefix-map \
- -fmacro-prefix-map=${S}=/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR} \
- -fdebug-prefix-map=${S}=/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR} \
- -fmacro-prefix-map=${B}=/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR} \
- -fdebug-prefix-map=${B}=/usr/src/debug/${PN}/${EXTENDPE}${PV}-${PR} \
+ -fmacro-prefix-map=${S}=${prefix}/src/debug/${PN}/${EXTENDPE}${PV}-${PR} \
+ -fdebug-prefix-map=${S}=${prefix}/src/debug/${PN}/${EXTENDPE}${PV}-${PR} \
+ -fmacro-prefix-map=${B}=${prefix}/src/debug/${PN}/${EXTENDPE}${PV}-${PR} \
+ -fdebug-prefix-map=${B}=${prefix}/src/debug/${PN}/${EXTENDPE}${PV}-${PR} \
  -fdebug-prefix-map=${STAGING_DIR_HOST}= \
  -fmacro-prefix-map=${STAGING_DIR_HOST}= \
  -fdebug-prefix-map=${STAGING_DIR_NATIVE}= \
diff --git a/meta/lib/oe/package.py b/meta/lib/oe/package.py
index 70040f09e7..b2799aec3e 100644
--- a/meta/lib/oe/package.py
+++ b/meta/lib/oe/package.py
@@ -687,6 +687,10 @@ def split_locales(d):
     #d.setVar('RDEPENDS:%s' % pn, ' '.join(rdep))
 
 def package_debug_vars(d):
+    prefix = d.getVar('prefix')
+    srcdir = ("%s/src/debug" % prefix)
+    libdir = ("%s/lib/debug" % prefix)
+    staticlibdir = ("%s/lib/debug-static" % prefix)
     # We default to '.debug' style
     if d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-file-directory':
         # Single debug-file-directory style debug info
@@ -695,9 +699,9 @@ def package_debug_vars(d):
             "staticappend": "",
             "dir": "",
             "staticdir": "",
-            "libdir": "/usr/lib/debug",
-            "staticlibdir": "/usr/lib/debug-static",
-            "srcdir": "/usr/src/debug",
+            "libdir": libdir,
+            "staticlibdir": staticlibdir,
+            "srcdir": srcdir,
         }
     elif d.getVar('PACKAGE_DEBUG_SPLIT_STYLE') == 'debug-without-src':
         # Original OE-core, a.k.a. ".debug", style debug info, but without sources in /usr/src/debug
@@ -718,7 +722,7 @@ def package_debug_vars(d):
             "staticdir": "/.debug-static",
             "libdir": "",
             "staticlibdir": "",
-            "srcdir": "/usr/src/debug",
+            "srcdir": srcdir,
         }
     else:
         # Original OE-core, a.k.a. ".debug", style debug info
@@ -729,7 +733,7 @@ def package_debug_vars(d):
             "staticdir": "/.debug-static",
             "libdir": "",
             "staticlibdir": "",
-            "srcdir": "/usr/src/debug",
+            "srcdir": srcdir,
         }
 
     return debug_vars
@@ -1257,6 +1261,7 @@ def populate_packages(d):
     dvar = d.getVar('PKGD')
     packages = d.getVar('PACKAGES').split()
     pn = d.getVar('PN')
+    prefix = d.getVar('prefix')
 
     bb.utils.mkdirhier(outdir)
     os.chdir(dvar)
@@ -1271,7 +1276,7 @@ def populate_packages(d):
         src_package_name = ('%s-src' % d.getVar('PN'))
         if not src_package_name in packages:
             packages.append(src_package_name)
-        d.setVar('FILES:%s' % src_package_name, '/usr/src/debug')
+        d.setVar('FILES:%s' % src_package_name, ('%s/src/debug' % prefix))
 
     # Sanity check PACKAGES for duplicates
     # Sanity should be moved to sanity.bbclass once we have the infrastructure
@@ -1301,6 +1306,7 @@ def populate_packages(d):
     oldumask = os.umask(0)
 
     debug = []
+    source = []
     for root, dirs, files in cpath.walk(dvar):
         dir = root[len(dvar):]
         if not dir:
@@ -1309,6 +1315,11 @@ def populate_packages(d):
             path = "." + os.path.join(dir, f)
             if "/.debug/" in path or "/.debug-static/" in path or path.endswith("/.debug"):
                 debug.append(path)
+            elif ("%s/src/debug" % prefix) in path:
+                if split_source_package:
+                    source.append(path)
+                else:
+                    debug.append(path)
 
     for pkg in packages:
         root = os.path.join(pkgdest, pkg)
@@ -1325,6 +1336,8 @@ def populate_packages(d):
 
         if autodebug and pkg.endswith("-dbg"):
             files.extend(debug)
+        if split_source_package and pkg.endswith("-src"):
+            files.extend(source)
 
         for file in files:
             if (not cpath.islink(file)) and (not cpath.exists(file)):
-- 
2.36.0

Re: [OE-core] [PATCH] package: fix src packaging path for nativesdk

[Luca Ceresoli]

Hello Maxime,

On Tue, 20 Jun 2023 17:36:40 -0400
Maxime Roussin-Bélanger <maxime.roussinbelanger@gmail.com> wrote:

> From: Maxime Roussin-Belanger <maxime.roussinbelanger@gmail.com>
> 
> Directory tree doesn't contain the correct structure when packaging
> for source files for the sdk. Since the structure is incorrect,
> they won't be part of the installed sdk.
> 
> Having source files in nativesdk is necessary when using it as a
> development sysroot for debugging. Otherwise, when debugging in an
> emulation mode it's impossible to step in 3rd party library code.
> 
> Signed-off-by: Maxime Roussin-Belanger <maxime.roussinbelanger@gmail.com>

These errors appeared when testing on the autobuilders with your patch
applied. I'm not totally sure this patch is the cause for all of them,
but it looks like the most likely cause.

Some errors appear as:

  ERROR: nativesdk-libgcc-13.1.1-r0 do_package: Error executing a
  ...
  python function in exec_func_python() autogenerated: Exception: Exception: KeyError: 'getpwuid(): uid not found: 6000'
  Path
  ./package/usr/local/oe-sdk-hardcoded-buildpath/sysroots/x86_64-pokysdk-linux/usr/src/debug/nativesdk-libgcc/13.1.1-r0/gcc/gcov-io.cc is owned by uid 6000, gid 6000, which doesn't match any user/group on target. This may be due to host contamination.
  NOTE: recipe nativesdk-libgcc-13.1.1-r0: task do_package: Failed

Here are some logs:

https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/7737/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/7737/steps/15/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/7356/steps/12/logs/stdio

Others come in this form:

  ERROR: buildtools-tarball-1.0-r0 do_testsdk: The toolchain /home/pokybuild/yocto-worker/buildtools/build/build/tmp/deploy/sdk/x86_64-buildtools-nativesdk-standalone-4.2.sh is not built. Build it before running the tests: 'bitbake <image> -c populate_sdk' .

Some logs for this are:

https://autobuilder.yoctoproject.org/typhoon/#/builders/20/builds/7737/steps/13/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/7356/steps/13/logs/stdio

Luca

-- 
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com