[PATCH v2 0/2] Check invalid characters on some recipe metadata

[PATCH v2 0/2] Check invalid characters on some recipe metadata

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

* insane.bbclass: QA check for invalid chars
* package_deb.bbclass: Opens the control file allowing UTF-8 chars
and handles a string decoding exception, reported on [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=6693

The following changes since commit a533776d6ff83b6e3e830137455b8382d002768b:

  perf: fix build breakage on kernels after 4.1 (2015-08-12 11:31:49 -0700)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib lsandov1/package_deb-handle-non-ascii-6693-v2
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lsandov1/package_deb-handle-non-ascii-6693-v2

Leonardo Sandoval (2):
  insane.bbclass: Check invalid characters (non UTF8) on recipe metadata
  package_deb.bbclass: Allow UTF-8 characters on control files

 meta/classes/insane.bbclass      | 24 +++++++++++++++++++++++-
 meta/classes/package_deb.bbclass | 18 +++++++++++-------
 2 files changed, 34 insertions(+), 8 deletions(-)

-- 
1.8.4.5

[PATCH v2 1/2] insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

Check if invalid characters are present on recipe's metadata. Fields
taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
 meta/classes/insane.bbclass | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 9c05c86..fc16540 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -11,6 +11,7 @@
 #  -Check if packages contains .debug directories or .so files
 #   where they should be in -dev or -dbg
 #  -Check if config.log contains traces to broken autoconf tests
+#  -Check invalid characters (non-utf8) on some package metadata
 #  -Ensure that binaries in base_[bindir|sbindir|libdir] do not link
 #   into exec_prefix
 #  -Check that scripts in base_[bindir|sbindir|libdir] do not reference
@@ -36,7 +37,7 @@ WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
 ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
             perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
             split-strip packages-list pkgv-undefined var-undefined \
-            version-going-backwards expanded-d \
+            version-going-backwards expanded-d invalid-chars \
             "
 
 ALL_QA = "${WARN_QA} ${ERROR_QA}"
@@ -947,6 +948,24 @@ def package_qa_check_expanded_d(path,name,d,elf,messages):
                         sane = False
     return sane
 
+def package_qa_check_encoding(keys, encode, d):
+    def check_encoding(key,enc):
+        sane = True
+        value = d.getVar(key, True)
+        if value:
+            try:
+                s = unicode(value, enc)
+            except UnicodeDecodeError as e:
+                error_msg = "%s has non %s characters" % (key,enc)
+                sane = False
+                package_qa_handle_error("invalid-chars", error_msg, d)
+        return sane
+
+    for key in keys:
+        sane = check_encoding(key, encode)
+        if not sane:
+            break
+
 # The PACKAGE FUNC to scan each package
 python do_package_qa () {
     import subprocess
@@ -956,6 +975,9 @@ python do_package_qa () {
 
     bb.build.exec_func("read_subpackage_metadata", d)
 
+    # Check non UTF-8 characters on recipe's metadata
+    package_qa_check_encoding(['DESCRIPTION', 'SUMMARY', 'LICENSE', 'SECTION'], 'utf-8', d)
+
     logdir = d.getVar('T', True)
     pkg = d.getVar('PN', True)
 
-- 
1.8.4.5

Re: [PATCH v2 1/2] insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 625 bytes --]

On Mon, Aug 17, 2015 at 12:10 AM, <
leonardo.sandoval.gonzalez@linux.intel.com> wrote:

> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
>
> Check if invalid characters are present on recipe's metadata. Fields
> taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.
>
> Signed-off-by: Leonardo Sandoval <
> leonardo.sandoval.gonzalez@linux.intel.com>
>

This commit message doesn't define what "invalid" is.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1234 bytes --]

Re: [PATCH v2 1/2] insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata

[Leonardo Sandoval]

On 08/17/2015 05:57 PM, Christopher Larson wrote:
> On Mon, Aug 17, 2015 at 12:10 AM, <
> leonardo.sandoval.gonzalez@linux.intel.com> wrote:
>
>> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
>>
>> Check if invalid characters are present on recipe's metadata. Fields
>> taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.
>>
>> Signed-off-by: Leonardo Sandoval <
>> leonardo.sandoval.gonzalez@linux.intel.com>
>>
>
> This commit message doesn't define what "invalid" is.

I will send a v3 to include what invalid means on this context. 
Basically all non-UTF8 characters are considered invalid.


>

Re: [PATCH v2 1/2] insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata

[Leonardo Sandoval]

hi Chris,

On 08/17/2015 05:57 PM, Christopher Larson wrote:
> On Mon, Aug 17, 2015 at 12:10 AM, <
> leonardo.sandoval.gonzalez@linux.intel.com> wrote:
>
>> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
>>
>> Check if invalid characters are present on recipe's metadata. Fields
>> taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.
>>
>> Signed-off-by: Leonardo Sandoval <
>> leonardo.sandoval.gonzalez@linux.intel.com>
>>
>
> This commit message doesn't define what "invalid" is.

Looking it closely, the commit message does include in parenthesis what 
is meant by 'invalid'.

Any concert about this definition?

>

Re: [PATCH v2 1/2] insane.bbclass: Check for invalid characters (non UTF8) on recipe metadata

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1155 bytes --]

On Wed, Aug 26, 2015 at 9:13 AM, Leonardo Sandoval <
leonardo.sandoval.gonzalez@linux.intel.com> wrote:

> hi Chris,
>
> On 08/17/2015 05:57 PM, Christopher Larson wrote:
>
>> On Mon, Aug 17, 2015 at 12:10 AM, <
>> leonardo.sandoval.gonzalez@linux.intel.com> wrote:
>>
>> From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
>>>
>>> Check if invalid characters are present on recipe's metadata. Fields
>>> taken into account are: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and
>>> 'SECTION'.
>>>
>>> Signed-off-by: Leonardo Sandoval <
>>> leonardo.sandoval.gonzalez@linux.intel.com>
>>>
>>>
>> This commit message doesn't define what "invalid" is.
>>
>
> Looking it closely, the commit message does include in parenthesis what is
> meant by 'invalid'.
>
> Any concert about this definition?
>

Ah, the details were in the summary, but not the description. Generally
that should be the other way around, but at least the info is there, so no
objections.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2171 bytes --]

[PATCH v2 2/2] package_deb.bbclass: Allow UTF-8 characters on control files

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

Allow UTF-8 characters on control files. Also handle an expection
in case of invalid characters (non UTF-8).

[YOCTO #6693]

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
 meta/classes/package_deb.bbclass | 18 +++++++++++-------
 1 file changed, 11 insertions(+), 7 deletions(-)

diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index 9e1ed28..398ceee 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -96,9 +96,8 @@ python do_package_deb () {
         bb.utils.mkdirhier(controldir)
         os.chmod(controldir, 0755)
         try:
-            ctrlfile = open(os.path.join(controldir, 'control'), 'w')
-            # import codecs
-            # ctrlfile = codecs.open("someFile", "w", "utf-8")
+            import codecs
+            ctrlfile = codecs.open(os.path.join(controldir, 'control'), 'w', 'utf-8')
         except OSError:
             bb.utils.unlockfile(lf)
             raise bb.build.FuncFailed("unable to open control file for writing.")
@@ -149,7 +148,7 @@ python do_package_deb () {
                 # Special behavior for description...
                 if 'DESCRIPTION' in fs:
                      summary = localdata.getVar('SUMMARY', True) or localdata.getVar('DESCRIPTION', True) or "."
-                     ctrlfile.write('Description: %s\n' % unicode(summary))
+                     ctrlfile.write('Description: %s\n' % unicode(summary,'utf-8'))
                      description = localdata.getVar('DESCRIPTION', True) or "."
                      description = textwrap.dedent(description).strip()
                      if '\\n' in description:
@@ -158,19 +157,24 @@ python do_package_deb () {
                              # We don't limit the width when manually indent, but we do
                              # need the textwrap.fill() to set the initial_indent and
                              # subsequent_indent, so set a large width
-                             ctrlfile.write('%s\n' % unicode(textwrap.fill(t, width=100000, initial_indent=' ', subsequent_indent=' ')))
+                             ctrlfile.write('%s\n' % unicode(textwrap.fill(t, width=100000, initial_indent=' ', subsequent_indent=' '),'utf-8'))
                      else:
                          # Auto indent
-                         ctrlfile.write('%s\n' % unicode(textwrap.fill(description.strip(), width=74, initial_indent=' ', subsequent_indent=' ')))
+                         ctrlfile.write('%s\n' % unicode(textwrap.fill(description.strip(), width=74, initial_indent=' ', subsequent_indent=' '),'utf-8'))
 
                 else:
-                     ctrlfile.write(unicode(c % tuple(pullData(fs, localdata))))
+                     ctrlfile.write(unicode(c % tuple(pullData(fs, localdata)),'utf-8'))
         except KeyError:
             import sys
             (type, value, traceback) = sys.exc_info()
             bb.utils.unlockfile(lf)
             ctrlfile.close()
             raise bb.build.FuncFailed("Missing field for deb generation: %s" % value)
+        except UnicodeDecodeError:
+            bb.utils.unlockfile(lf)
+            ctrlfile.close()
+            raise bb.build.FuncFailed("Non UTF-8 characters found in one of the fields")
+
         # more fields
 
         custom_fields_chunk = get_package_additional_metadata("deb", localdata)
-- 
1.8.4.5