[PATCH 1/7] Revert "populate_sdk_ext: whitelist do_package tasks"

[PATCH 1/7] Revert "populate_sdk_ext: whitelist do_package tasks"

[Richard Purdie]

Since Paul reverted the sstate.bbclass change which was checking the sstate
mirror test results, this change should also not be needed anymore.

This reverts commit e30f5002c4f216757ace27ad8d06164716ca46b5.
---
 meta/classes/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 9b711d1..b383bed 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -306,7 +306,7 @@ python copy_buildsystem () {
             f.write('SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"\n\n')
 
             # Set up whitelist for run on install
-            f.write('BB_SETSCENE_ENFORCE_WHITELIST = "%:* *:do_shared_workdir *:do_rm_work *:do_package"\n\n')
+            f.write('BB_SETSCENE_ENFORCE_WHITELIST = "%:* *:do_shared_workdir *:do_rm_work"\n\n')
 
             # Hide the config information from bitbake output (since it's fixed within the SDK)
             f.write('BUILDCFG_HEADER = ""\n\n')
-- 
2.7.4

[PATCH 2/7] populate_sdk_ext: Add wic-tools to BB_SETSCENE_ENFORCE_WHITELIST

[Richard Purdie]

wic-tools has tasks which would always rerun and not come from sstate
to ensure we have a correctly populated sysroot. This is low overhead
and can be ignored from an eSDK perspective.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/populate_sdk_ext.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index b383bed..39e0c83 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -306,7 +306,7 @@ python copy_buildsystem () {
             f.write('SIGGEN_LOCKEDSIGS_TASKSIG_CHECK = "warn"\n\n')
 
             # Set up whitelist for run on install
-            f.write('BB_SETSCENE_ENFORCE_WHITELIST = "%:* *:do_shared_workdir *:do_rm_work"\n\n')
+            f.write('BB_SETSCENE_ENFORCE_WHITELIST = "%:* *:do_shared_workdir *:do_rm_work wic-tools:*"\n\n')
 
             # Hide the config information from bitbake output (since it's fixed within the SDK)
             f.write('BUILDCFG_HEADER = ""\n\n')
-- 
2.7.4

[PATCH 3/7] package_deb: Improve failure debug output

[Richard Purdie]

Currently if the dpkg-deb command fails you see an error message like this:

ERROR: nss-3.27.1-r0 do_package_write_deb: dpkg-deb execution failed
ERROR: nss-3.27.1-r0 do_package_write_deb: Function failed: do_package_deb

which is pretty much useless. If we use subprocess.check_output, we see a
traceback and then:

Exception: subprocess.CalledProcessError: Command '<cmd>' returned non-zero exit status 1

Subprocess output:
<output>

which is much easier to debug from.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/package_deb.bbclass | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index 6ce008f..2a70b50c 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -308,10 +308,7 @@ python do_package_deb () {
             conffiles.close()
 
         os.chdir(basedir)
-        ret = subprocess.call("PATH=\"%s\" dpkg-deb -b %s %s" % (localdata.getVar("PATH"), root, pkgoutdir), shell=True)
-        if ret != 0:
-            bb.utils.unlockfile(lf)
-            bb.fatal("dpkg-deb execution failed")
+        subprocess.check_output("PATH=\"%s\" dpkg-deb -b %s %s" % (localdata.getVar("PATH"), root, pkgoutdir), shell=True)
 
         cleanupcontrol(root)
         bb.utils.unlockfile(lf)
-- 
2.7.4

[PATCH 4/7] package_deb: Clean up pointless exception handling

[Richard Purdie]

The exception handling in this function seemed mildly crazy. Python will
given perfectly good or in several cases better information if we let its
standard traceback/exception handling happen. Remove the pointless code
along with the duplicated key checking which was broken in the inner loop
by usage of the wrong variable.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/package_deb.bbclass | 73 ++++++++++++++--------------------------
 1 file changed, 26 insertions(+), 47 deletions(-)

diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index 2a70b50c..47fcd6b 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -53,6 +53,7 @@ python do_package_deb () {
     import textwrap
     import subprocess
     import collections
+    import codecs
 
     oldcwd = os.getcwd()
 
@@ -121,12 +122,8 @@ python do_package_deb () {
         controldir = os.path.join(root, 'DEBIAN')
         bb.utils.mkdirhier(controldir)
         os.chmod(controldir, 0o755)
-        try:
-            import codecs
-            ctrlfile = codecs.open(os.path.join(controldir, 'control'), 'w', 'utf-8')
-        except OSError:
-            bb.utils.unlockfile(lf)
-            bb.fatal("unable to open control file for writing")
+
+        ctrlfile = codecs.open(os.path.join(controldir, 'control'), 'w', 'utf-8')
 
         fields = []
         pe = d.getVar('PKGE')
@@ -153,7 +150,7 @@ python do_package_deb () {
             for i in l:
                 data = d.getVar(i)
                 if data is None:
-                    raise KeyError(f)
+                    raise KeyError(i)
                 if i == 'DPKG_ARCH' and d.getVar('PACKAGE_ARCH') == 'all':
                     data = 'all'
                 elif i == 'PACKAGE_ARCH' or i == 'DPKG_ARCH':
@@ -168,36 +165,26 @@ python do_package_deb () {
         if d.getVar('PACKAGE_ARCH') == "all":
             ctrlfile.write("Multi-Arch: foreign\n")
         # check for required fields
-        try:
-            for (c, fs) in fields:
-                for f in fs:
-                     if localdata.getVar(f, False) is None:
-                         raise KeyError(f)
-                # Special behavior for description...
-                if 'DESCRIPTION' in fs:
-                     summary = localdata.getVar('SUMMARY') or localdata.getVar('DESCRIPTION') or "."
-                     ctrlfile.write('Description: %s\n' % summary)
-                     description = localdata.getVar('DESCRIPTION') or "."
-                     description = textwrap.dedent(description).strip()
-                     if '\\n' in description:
-                         # Manually indent
-                         for t in description.split('\\n'):
-                             # We don't limit the width when manually indent, but we do
-                             # need the textwrap.fill() to set the initial_indent and
-                             # subsequent_indent, so set a large width
-                             ctrlfile.write('%s\n' % textwrap.fill(t, width=100000, initial_indent=' ', subsequent_indent=' '))
-                     else:
-                         # Auto indent
-                         ctrlfile.write('%s\n' % textwrap.fill(description.strip(), width=74, initial_indent=' ', subsequent_indent=' '))
-
-                else:
-                     ctrlfile.write(c % tuple(pullData(fs, localdata)))
-        except KeyError:
-            import sys
-            (type, value, traceback) = sys.exc_info()
-            bb.utils.unlockfile(lf)
-            ctrlfile.close()
-            bb.fatal("Missing field for deb generation: %s" % value)
+        for (c, fs) in fields:
+            # Special behavior for description...
+            if 'DESCRIPTION' in fs:
+                 summary = localdata.getVar('SUMMARY') or localdata.getVar('DESCRIPTION') or "."
+                 ctrlfile.write('Description: %s\n' % summary)
+                 description = localdata.getVar('DESCRIPTION') or "."
+                 description = textwrap.dedent(description).strip()
+                 if '\\n' in description:
+                     # Manually indent
+                     for t in description.split('\\n'):
+                         # We don't limit the width when manually indent, but we do
+                         # need the textwrap.fill() to set the initial_indent and
+                         # subsequent_indent, so set a large width
+                         ctrlfile.write('%s\n' % textwrap.fill(t, width=100000, initial_indent=' ', subsequent_indent=' '))
+                 else:
+                     # Auto indent
+                     ctrlfile.write('%s\n' % textwrap.fill(description.strip(), width=74, initial_indent=' ', subsequent_indent=' '))
+
+            else:
+                 ctrlfile.write(c % tuple(pullData(fs, localdata)))
 
         # more fields
 
@@ -273,11 +260,7 @@ python do_package_deb () {
             if not scriptvar:
                 continue
             scriptvar = scriptvar.strip()
-            try:
-                scriptfile = open(os.path.join(controldir, script), 'w')
-            except OSError:
-                bb.utils.unlockfile(lf)
-                bb.fatal("unable to open %s script file for writing" % script)
+            scriptfile = open(os.path.join(controldir, script), 'w')
 
             if scriptvar.startswith("#!"):
                 pos = scriptvar.find("\n") + 1
@@ -297,11 +280,7 @@ python do_package_deb () {
 
         conffiles_str = ' '.join(get_conffiles(pkg, d))
         if conffiles_str:
-            try:
-                conffiles = open(os.path.join(controldir, 'conffiles'), 'w')
-            except OSError:
-                bb.utils.unlockfile(lf)
-                bb.fatal("unable to open conffiles for writing")
+            conffiles = open(os.path.join(controldir, 'conffiles'), 'w')
             for f in conffiles_str.split():
                 if os.path.exists(oe.path.join(root, f)):
                     conffiles.write('%s\n' % f)
-- 
2.7.4

[PATCH 5/7] package_ipk: Improve failure debug output

[Richard Purdie]

Currently if the dpkg-deb command fails you see an error message like this:

ERROR: nss-3.27.1-r0 do_package_write_ipk: opkg-build execution failed
ERROR: nss-3.27.1-r0 do_package_write_ipk: Function failed: do_package_ipk

which is pretty much useless. If we use subprocess.check_output, we see a
traceback and then:

Exception: subprocess.CalledProcessError: Command '<cmd>' returned non-zero exit status 1

Subprocess output:
<output>

which is much easier to debug from.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/package_ipk.bbclass | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index 039b6ab..9fb128b 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -244,11 +244,8 @@ python do_package_ipk () {
             conffiles.close()
 
         os.chdir(basedir)
-        ret = subprocess.call("PATH=\"%s\" %s %s %s" % (localdata.getVar("PATH"),
+        subprocess.check_output("PATH=\"%s\" %s %s %s" % (localdata.getVar("PATH"),
                                                           d.getVar("OPKGBUILDCMD"), pkg, pkgoutdir), shell=True)
-        if ret != 0:
-            bb.utils.unlockfile(lf)
-            bb.fatal("opkg-build execution failed")
 
         if d.getVar('IPK_SIGN_PACKAGES') == '1':
             ipkver = "%s-%s" % (d.getVar('PKGV'), d.getVar('PKGR'))
-- 
2.7.4

[PATCH 6/7] package_ipk: Clean up pointless exception handling

[Richard Purdie]

The exception handling in this function seemed mildly crazy. Python will
given perfectly good or in several cases better information if we let its
standard traceback/exception handling happen. Remove the pointless code.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/package_ipk.bbclass | 67 ++++++++++++++--------------------------
 1 file changed, 24 insertions(+), 43 deletions(-)

diff --git a/meta/classes/package_ipk.bbclass b/meta/classes/package_ipk.bbclass
index 9fb128b..a76b235 100644
--- a/meta/classes/package_ipk.bbclass
+++ b/meta/classes/package_ipk.bbclass
@@ -104,11 +104,7 @@ python do_package_ipk () {
 
         controldir = os.path.join(root, 'CONTROL')
         bb.utils.mkdirhier(controldir)
-        try:
-            ctrlfile = open(os.path.join(controldir, 'control'), 'w')
-        except OSError:
-            bb.utils.unlockfile(lf)
-            bb.fatal("unable to open control file for writing")
+        ctrlfile = open(os.path.join(controldir, 'control'), 'w')
 
         fields = []
         pe = d.getVar('PKGE')
@@ -134,35 +130,28 @@ python do_package_ipk () {
 
         ctrlfile.write("Package: %s\n" % pkgname)
         # check for required fields
-        try:
-            for (c, fs) in fields:
-                for f in fs:
-                    if localdata.getVar(f, False) is None:
-                        raise KeyError(f)
-                # Special behavior for description...
-                if 'DESCRIPTION' in fs:
-                    summary = localdata.getVar('SUMMARY') or localdata.getVar('DESCRIPTION') or "."
-                    ctrlfile.write('Description: %s\n' % summary)
-                    description = localdata.getVar('DESCRIPTION') or "."
-                    description = textwrap.dedent(description).strip()
-                    if '\\n' in description:
-                        # Manually indent
-                        for t in description.split('\\n'):
-                            # We don't limit the width when manually indent, but we do
-                            # need the textwrap.fill() to set the initial_indent and
-                            # subsequent_indent, so set a large width
-                            ctrlfile.write('%s\n' % textwrap.fill(t.strip(), width=100000, initial_indent=' ', subsequent_indent=' '))
-                    else:
-                        # Auto indent
-                        ctrlfile.write('%s\n' % textwrap.fill(description, width=74, initial_indent=' ', subsequent_indent=' '))
+        for (c, fs) in fields:
+            for f in fs:
+                if localdata.getVar(f, False) is None:
+                    raise KeyError(f)
+            # Special behavior for description...
+            if 'DESCRIPTION' in fs:
+                summary = localdata.getVar('SUMMARY') or localdata.getVar('DESCRIPTION') or "."
+                ctrlfile.write('Description: %s\n' % summary)
+                description = localdata.getVar('DESCRIPTION') or "."
+                description = textwrap.dedent(description).strip()
+                if '\\n' in description:
+                    # Manually indent
+                    for t in description.split('\\n'):
+                        # We don't limit the width when manually indent, but we do
+                        # need the textwrap.fill() to set the initial_indent and
+                        # subsequent_indent, so set a large width
+                        ctrlfile.write('%s\n' % textwrap.fill(t.strip(), width=100000, initial_indent=' ', subsequent_indent=' '))
                 else:
-                    ctrlfile.write(c % tuple(pullData(fs, localdata)))
-        except KeyError:
-            import sys
-            (type, value, traceback) = sys.exc_info()
-            ctrlfile.close()
-            bb.utils.unlockfile(lf)
-            bb.fatal("Missing field for ipk generation: %s" % value)
+                    # Auto indent
+                    ctrlfile.write('%s\n' % textwrap.fill(description, width=74, initial_indent=' ', subsequent_indent=' '))
+            else:
+                ctrlfile.write(c % tuple(pullData(fs, localdata)))
         # more fields
 
         custom_fields_chunk = get_package_additional_metadata("ipk", localdata)
@@ -222,22 +211,14 @@ python do_package_ipk () {
             scriptvar = localdata.getVar('pkg_%s' % script)
             if not scriptvar:
                 continue
-            try:
-                scriptfile = open(os.path.join(controldir, script), 'w')
-            except OSError:
-                bb.utils.unlockfile(lf)
-                bb.fatal("unable to open %s script file for writing" % script)
+            scriptfile = open(os.path.join(controldir, script), 'w')
             scriptfile.write(scriptvar)
             scriptfile.close()
             os.chmod(os.path.join(controldir, script), 0o755)
 
         conffiles_str = ' '.join(get_conffiles(pkg, d))
         if conffiles_str:
-            try:
-                conffiles = open(os.path.join(controldir, 'conffiles'), 'w')
-            except OSError:
-                bb.utils.unlockfile(lf)
-                bb.fatal("unable to open conffiles for writing")
+            conffiles = open(os.path.join(controldir, 'conffiles'), 'w')
             for f in conffiles_str.split():
                 if os.path.exists(oe.path.join(root, f)):
                     conffiles.write('%s\n' % f)
-- 
2.7.4

[PATCH 7/7] package_rpm: Clean up pointless exception handling

[Richard Purdie]

The exception handling in this function seemed mildly crazy. Python will
given perfectly good or in several cases better information if we let its
standard traceback/exception handling happen. Remove the pointless code.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes/package_rpm.bbclass | 15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)

diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index c978ec5..bde0453 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -55,10 +55,7 @@ def write_rpm_perfiledata(srcname, d):
     # OE-core dependencies a.k.a. RPM requires
     outdepends = workdir + "/" + srcname + ".requires"
 
-    try:
-        dependsfile = open(outdepends, 'w')
-    except OSError:
-        bb.fatal("unable to open spec file for writing")
+    dependsfile = open(outdepends, 'w')
 
     dump_filerdeps('RDEPENDS', dependsfile, d)
 
@@ -68,10 +65,7 @@ def write_rpm_perfiledata(srcname, d):
     # OE-core / RPM Provides
     outprovides = workdir + "/" + srcname + ".provides"
 
-    try:
-        providesfile = open(outprovides, 'w')
-    except OSError:
-        bb.fatal("unable to open spec file for writing")
+    providesfile = open(outprovides, 'w')
 
     dump_filerdeps('RPROVIDES', providesfile, d)
 
@@ -620,10 +614,7 @@ python write_specfile () {
         spec_scriptlets_top.append('')
 
     # Write the SPEC file
-    try:
-        specfile = open(outspecfile, 'w')
-    except OSError:
-        bb.fatal("unable to open spec file for writing")
+    specfile = open(outspecfile, 'w')
 
     # RPMSPEC_PREAMBLE is a way to add arbitrary text to the top
     # of the generated spec file
-- 
2.7.4

✗ patchtest: failure for "Revert "populate_sdk_ext: whit..." and 6 more

[Patchwork]

== Series Details ==

Series: "Revert "populate_sdk_ext: whit..." and 6 more
Revision: 1
URL   : https://patchwork.openembedded.org/series/4888/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Patch            [1/7] Revert "populate_sdk_ext: whitelist do_package tasks"
 Issue             Patch is missing Signed-off-by [test_signed_off_by_presence] 
  Suggested fix    Sign off the patch (either manually or with "git commit --amend -s")



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: ✗ patchtest: failure for "Revert "populate_sdk_ext: whit..." and 6 more

[Paul Eggleton]

On Saturday, 21 January 2017 2:53:37 PM NZDT Patchwork wrote:
> == Series Details ==
> 
> Series: "Revert "populate_sdk_ext: whit..." and 6 more
> Revision: 1
> URL   : https://patchwork.openembedded.org/series/4888/
> State : failure
> 
> == Summary ==
> 
> 
> Thank you for submitting this patch series to OpenEmbedded Core. This is
> an automated response. Several tests have been executed on the proposed
> series by patchtest resulting in the following failures:
> 
> 
> 
> * Patch            [1/7] Revert "populate_sdk_ext: whitelist do_package
> tasks" Issue             Patch is missing Signed-off-by
> [test_signed_off_by_presence] Suggested fix    Sign off the patch (either
> manually or with "git commit --amend -s")

Leo, we need to skip this test for patches whose shortlog starts with "Revert 
". Could you take care of that?

Thanks,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre