[PATCH 0/2] package.bbclass: improve auto debug package splitting order

[PATCH 0/2] package.bbclass: improve auto debug package splitting order

[Hongxu Jia]

Hi RP,

Is it necessary to split multiple sub dbg packages or only
one dbg is allowed? If the former, this fix makes sense.

I know in oe-core, almost one dbg package per recipe, but
in oe, 21 recipes have extra sub dbg package and 18 of them
did not set NOAUTOPACKAGEDEBUG = "1".

So rather than modify them one by one tediously, how about to
improve auto debug package splitting order? If ${PN}-dbg exists,
split the unsplit debug file to ${PN}-dbg, if ${PN}-dbg does not 
exist, still split them to first dbg package.

//Hongxu

The following changes since commit d180dc758710c7259d45eeb9304e7284a8fd8825:

  oeqa/selftest/recipetool: fix non-determinism in cmake test (2018-09-13 16:28:26 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib hongxu/dbg-split
  http://cgit.openembedded.org/openembedded-core-contrib/log/?h=hongxu/dbg-split

Hongxu Jia (2):
  package.bbclass: improve auto debug package splitting order
  selftest/package: improve case to test debug package splitting

 .../selftest-hardlink/selftest-hardlink.bb         |  4 +++-
 meta/classes/package.bbclass                       | 10 +++++++-
 meta/lib/oeqa/selftest/cases/package.py            | 27 ++++++++++++++++------
 3 files changed, 32 insertions(+), 9 deletions(-)

-- 
2.7.4

[PATCH 1/2] package.bbclass: improve auto debug package splitting order

[Hongxu Jia]

Since `da5ec06 package: Add auto package splitting
of .debug files' applied in oe-core, the first dbg
package will have all debug files automatically.

If there are several sub dbg packages, except the
first one, the others are empty.

Even though var-NOAUTOPACKAGEDEBUG is provided for manual
control, but many recipes (such as in oe) requires to
pick up a few debug files into sub dbg packages, and then
split the rest to ${PN}-dbg. If set NOAUTOPACKAGEDEBUG = "1",
they have to do the extra splitting manually.

So improve splitting order, if ${PN}-dbg exists, split
the unsplit debug file to ${PN}-dbg, if ${PN}-dbg does not
exist, still split them to first dbg package.

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/package.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index d1e9138..cada46b 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1175,6 +1175,14 @@ python populate_packages () {
             if "/.debug/" in path or path.endswith("/.debug"):
                 debug.append(path)
 
+    pn_dbg = pn + "-dbg"
+    # If ${PN}-dbg does not exist, use first *-dbg to replace
+    if pn_dbg not in package_list:
+        for pkg in package_list:
+            if pkg.endswith("-dbg"):
+                pn_dbg = pkg
+                break
+
     for pkg in package_list:
         root = os.path.join(pkgdest, pkg)
         bb.utils.mkdirhier(root)
@@ -1188,7 +1196,7 @@ python populate_packages () {
         origfiles = filesvar.split()
         files, symlink_paths = files_from_filevars(origfiles)
 
-        if autodebug and pkg.endswith("-dbg"):
+        if autodebug and pn_dbg == pkg:
             files.extend(debug)
 
         for file in files:
-- 
2.7.4

[PATCH 2/2] selftest/package: improve case to test debug package splitting

[Hongxu Jia]

Tweak recipe selftest-hardlink
- Add ${PN}-bin, ${PN}-bin-dbg

Tweak case test_gdb_hardlink_debug
- Test gdb debug package separately, verify the debug
  package splitting works correctly.

[Test without commit `package.bbclass: improve auto debug package splitting order']
2018-09-14 15:58:06,233 - oe-selftest - INFO - ======================================================================
2018-09-14 15:58:06,233 - oe-selftest - INFO - FAIL: test_gdb_hardlink_debug (package.PackageTests)
2018-09-14 15:58:06,233 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-09-14 15:58:06,233 - oe-selftest - INFO - Traceback (most recent call last):
  File "oe-core/meta/lib/oeqa/selftest/cases/package.py", line 123, in test_gdb_hardlink_debug
    self._test_gdb_debug(packages, binaries)
  File "oe-core/meta/lib/oeqa/selftest/cases/package.py", line 162, in _test_gdb_debug
    self.fail('GDB %s failed' % binary)
AssertionError: GDB /usr/libexec/hello3 failed
[Test without commit `package.bbclass: improve auto debug package splitting order']

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 .../selftest-hardlink/selftest-hardlink.bb         |  4 +++-
 meta/lib/oeqa/selftest/cases/package.py            | 27 ++++++++++++++++------
 2 files changed, 23 insertions(+), 8 deletions(-)

diff --git a/meta-selftest/recipes-test/selftest-hardlink/selftest-hardlink.bb b/meta-selftest/recipes-test/selftest-hardlink/selftest-hardlink.bb
index 842a977..7d544b4 100644
--- a/meta-selftest/recipes-test/selftest-hardlink/selftest-hardlink.bb
+++ b/meta-selftest/recipes-test/selftest-hardlink/selftest-hardlink.bb
@@ -26,5 +26,7 @@ do_install () {
 }
 
 RDEPENDS_${PN}-gdb += "gdb"
-PACKAGES =+ "${PN}-gdb"
+PACKAGES =+ "${PN}-gdb ${PN}-bin ${PN}-bin-dbg"
 FILES_${PN}-gdb = "${bindir}/gdb.sh"
+FILES_${PN}-bin-dbg = "${bindir}/.debug/*"
+
diff --git a/meta/lib/oeqa/selftest/cases/package.py b/meta/lib/oeqa/selftest/cases/package.py
index 0a88dc2..d00cd49 100644
--- a/meta/lib/oeqa/selftest/cases/package.py
+++ b/meta/lib/oeqa/selftest/cases/package.py
@@ -115,9 +115,25 @@ class PackageTests(OESelftestTestCase):
 
     # Verify gdb to read symbols from separated debug hardlink file correctly
     def test_gdb_hardlink_debug(self):
-        features = 'IMAGE_INSTALL_append = " selftest-hardlink"\n'
-        features += 'IMAGE_INSTALL_append = " selftest-hardlink-dbg"\n'
-        features += 'IMAGE_INSTALL_append = " selftest-hardlink-gdb"\n'
+        packages = ['selftest-hardlink',
+                    'selftest-hardlink-dbg',
+                    'selftest-hardlink-gdb']
+        binaries = ['/usr/libexec/hello3',
+                    '/usr/libexec/hello4']
+        self._test_gdb_debug(packages, binaries)
+
+        packages = ['selftest-hardlink-bin',
+                    'selftest-hardlink-bin-dbg',
+                    'selftest-hardlink-gdb']
+        binaries = ['/usr/bin/hello1',
+                    '/usr/bin/hello2']
+        self._test_gdb_debug(packages, binaries)
+
+    def _test_gdb_debug(self, packages, binaries):
+        features = ""
+        for pkg in packages:
+            features += 'IMAGE_INSTALL_append = " %s"\n' % pkg
+
         self.write_config(features)
         bitbake("core-image-minimal")
 
@@ -141,9 +157,6 @@ class PackageTests(OESelftestTestCase):
             return False
 
         with runqemu('core-image-minimal') as qemu:
-            for binary in ['/usr/bin/hello1',
-                           '/usr/bin/hello2',
-                           '/usr/libexec/hello3',
-                           '/usr/libexec/hello4']:
+            for binary in binaries:
                 if not gdbtest(qemu, binary):
                     self.fail('GDB %s failed' % binary)
-- 
2.7.4

[PATCH V2 2/2] selftest/package: improve case to test debug package splitting

[Hongxu Jia]

Tweak recipe selftest-hardlink
- Add ${PN}-bin, ${PN}-bin-dbg

Tweak test_preserve_sparse_hardlinks
- s/selftest-hardlink/selftest-hardlink-bin/

Tweak case test_gdb_hardlink_debug
- Test gdb debug package separately, verify the debug
  package splitting works correctly.

[Test without commit `package.bbclass: improve auto debug package splitting order']
2018-09-14 15:58:06,233 - oe-selftest - INFO - ======================================================================
2018-09-14 15:58:06,233 - oe-selftest - INFO - FAIL: test_gdb_hardlink_debug (package.PackageTests)
2018-09-14 15:58:06,233 - oe-selftest - INFO - ----------------------------------------------------------------------
2018-09-14 15:58:06,233 - oe-selftest - INFO - Traceback (most recent call last):
  File "oe-core/meta/lib/oeqa/selftest/cases/package.py", line 123, in test_gdb_hardlink_debug
    self._test_gdb_debug(packages, binaries)
  File "oe-core/meta/lib/oeqa/selftest/cases/package.py", line 162, in _test_gdb_debug
    self.fail('GDB %s failed' % binary)
AssertionError: GDB /usr/libexec/hello3 failed
[Test without commit `package.bbclass: improve auto debug package splitting order']

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 .../selftest-hardlink/selftest-hardlink.bb         |  4 ++-
 meta/lib/oeqa/selftest/cases/package.py            | 31 +++++++++++++++-------
 2 files changed, 25 insertions(+), 10 deletions(-)

diff --git a/meta-selftest/recipes-test/selftest-hardlink/selftest-hardlink.bb b/meta-selftest/recipes-test/selftest-hardlink/selftest-hardlink.bb
index 842a977..7d544b4 100644
--- a/meta-selftest/recipes-test/selftest-hardlink/selftest-hardlink.bb
+++ b/meta-selftest/recipes-test/selftest-hardlink/selftest-hardlink.bb
@@ -26,5 +26,7 @@ do_install () {
 }
 
 RDEPENDS_${PN}-gdb += "gdb"
-PACKAGES =+ "${PN}-gdb"
+PACKAGES =+ "${PN}-gdb ${PN}-bin ${PN}-bin-dbg"
 FILES_${PN}-gdb = "${bindir}/gdb.sh"
+FILES_${PN}-bin-dbg = "${bindir}/.debug/*"
+
diff --git a/meta/lib/oeqa/selftest/cases/package.py b/meta/lib/oeqa/selftest/cases/package.py
index 0a88dc2..d90297a 100644
--- a/meta/lib/oeqa/selftest/cases/package.py
+++ b/meta/lib/oeqa/selftest/cases/package.py
@@ -98,10 +98,10 @@ class PackageTests(OESelftestTestCase):
         def checkfiles():
             # Recipe creates 4 hardlinked files, there is a copy in package/ and a copy in packages-split/
             # so expect 8 in total.
-            self.assertEqual(os.stat(dest + "/selftest-hardlink" + bindir + "/hello1").st_nlink, 8)
+            self.assertEqual(os.stat(dest + "/selftest-hardlink-bin" + bindir + "/hello1").st_nlink, 8)
 
             # Test a sparse file remains sparse
-            sparsestat = os.stat(dest + "/selftest-hardlink" + bindir + "/sparsetest")
+            sparsestat = os.stat(dest + "/selftest-hardlink-bin" + bindir + "/sparsetest")
             self.assertEqual(sparsestat.st_blocks, 0)
             self.assertEqual(sparsestat.st_size, 1048576)
 
@@ -115,9 +115,25 @@ class PackageTests(OESelftestTestCase):
 
     # Verify gdb to read symbols from separated debug hardlink file correctly
     def test_gdb_hardlink_debug(self):
-        features = 'IMAGE_INSTALL_append = " selftest-hardlink"\n'
-        features += 'IMAGE_INSTALL_append = " selftest-hardlink-dbg"\n'
-        features += 'IMAGE_INSTALL_append = " selftest-hardlink-gdb"\n'
+        packages = ['selftest-hardlink',
+                    'selftest-hardlink-dbg',
+                    'selftest-hardlink-gdb']
+        binaries = ['/usr/libexec/hello3',
+                    '/usr/libexec/hello4']
+        self._test_gdb_debug(packages, binaries)
+
+        packages = ['selftest-hardlink-bin',
+                    'selftest-hardlink-bin-dbg',
+                    'selftest-hardlink-gdb']
+        binaries = ['/usr/bin/hello1',
+                    '/usr/bin/hello2']
+        self._test_gdb_debug(packages, binaries)
+
+    def _test_gdb_debug(self, packages, binaries):
+        features = ""
+        for pkg in packages:
+            features += 'IMAGE_INSTALL_append = " %s"\n' % pkg
+
         self.write_config(features)
         bitbake("core-image-minimal")
 
@@ -141,9 +157,6 @@ class PackageTests(OESelftestTestCase):
             return False
 
         with runqemu('core-image-minimal') as qemu:
-            for binary in ['/usr/bin/hello1',
-                           '/usr/bin/hello2',
-                           '/usr/libexec/hello3',
-                           '/usr/libexec/hello4']:
+            for binary in binaries:
                 if not gdbtest(qemu, binary):
                     self.fail('GDB %s failed' % binary)
-- 
2.7.4

Re: [PATCH 0/2] package.bbclass: improve auto debug package splitting order

[Hongxu Jia]

On 2018年09月14日 16:49, Hongxu Jia wrote:
> Hi RP,
Ping RP,

//Hongxu
> Is it necessary to split multiple sub dbg packages or only
> one dbg is allowed? If the former, this fix makes sense.
>
> I know in oe-core, almost one dbg package per recipe, but
> in oe, 21 recipes have extra sub dbg package and 18 of them
> did not set NOAUTOPACKAGEDEBUG = "1".
>
> So rather than modify them one by one tediously, how about to
> improve auto debug package splitting order? If ${PN}-dbg exists,
> split the unsplit debug file to ${PN}-dbg, if ${PN}-dbg does not
> exist, still split them to first dbg package.
>
> //Hongxu
>
> The following changes since commit d180dc758710c7259d45eeb9304e7284a8fd8825:
>
>    oeqa/selftest/recipetool: fix non-determinism in cmake test (2018-09-13 16:28:26 +0100)
>
> are available in the git repository at:
>
>    git://git.openembedded.org/openembedded-core-contrib hongxu/dbg-split
>    http://cgit.openembedded.org/openembedded-core-contrib/log/?h=hongxu/dbg-split
>
> Hongxu Jia (2):
>    package.bbclass: improve auto debug package splitting order
>    selftest/package: improve case to test debug package splitting
>
>   .../selftest-hardlink/selftest-hardlink.bb         |  4 +++-
>   meta/classes/package.bbclass                       | 10 +++++++-
>   meta/lib/oeqa/selftest/cases/package.py            | 27 ++++++++++++++++------
>   3 files changed, 32 insertions(+), 9 deletions(-)
>

Re: [PATCH 0/2] package.bbclass: improve auto debug package splitting order

[richard.purdie]

On Fri, 2018-09-14 at 16:49 +0800, Hongxu Jia wrote:
> Hi RP,
> 
> Is it necessary to split multiple sub dbg packages or only
> one dbg is allowed? If the former, this fix makes sense.

Only one is recommended and the code only works properly in some cases
for one. I'd like to be in a position where we could have that as the
only thing we supported but its simply not practical in some cases.

The debug package dependency handling probably needs a rethink but that
would be 2.7 development cycle, its too late for 2.6 now.

> I know in oe-core, almost one dbg package per recipe, but
> in oe, 21 recipes have extra sub dbg package and 18 of them
> did not set NOAUTOPACKAGEDEBUG = "1".
> 
> So rather than modify them one by one tediously, how about to
> improve auto debug package splitting order? If ${PN}-dbg exists,
> split the unsplit debug file to ${PN}-dbg, if ${PN}-dbg does not 
> exist, still split them to first dbg package.

I don't really want to encourage multiple -dbg packages per recipe and
I am a little concerned about making changes to the core packaging code
with the release coming up. Fixing the recipes, although tedious may be
safer/better rather than encouraging people to do things which aren't
well supported.

Cheers,

Richard

Re: [PATCH 0/2] package.bbclass: improve auto debug package splitting order

[Hongxu Jia]

On 2018年09月17日 20:06, richard.purdie@linuxfoundation.org wrote:
> On Fri, 2018-09-14 at 16:49 +0800, Hongxu Jia wrote:
>> Hi RP,
>>
>> Is it necessary to split multiple sub dbg packages or only
>> one dbg is allowed? If the former, this fix makes sense.
> Only one is recommended and the code only works properly in some cases
> for one. I'd like to be in a position where we could have that as the
> only thing we supported but its simply not practical in some cases.
>
> The debug package dependency handling probably needs a rethink but that
> would be 2.7 development cycle, its too late for 2.6 now.
>
>> I know in oe-core, almost one dbg package per recipe, but
>> in oe, 21 recipes have extra sub dbg package and 18 of them
>> did not set NOAUTOPACKAGEDEBUG = "1".
>>
>> So rather than modify them one by one tediously, how about to
>> improve auto debug package splitting order? If ${PN}-dbg exists,
>> split the unsplit debug file to ${PN}-dbg, if ${PN}-dbg does not
>> exist, still split them to first dbg package.
> I don't really want to encourage multiple -dbg packages per recipe and
> I am a little concerned about making changes to the core packaging code
> with the release coming up. Fixing the recipes, although tedious may be
> safer/better rather than encouraging people to do things which aren't
> well supported.

Got it, I will fix affected recipes

Thanks for replying

//Hongxu

> Cheers,
>
> Richard