[PATCH 0/2] Check non-ASCII characters on recipe metadata

[PATCH 0/2] Check non-ASCII characters on recipe metadata

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

* First patch creates a QA check on insane.bbclass
* Second patch handles a string encode exception, reported on [1]

[1] https://bugzilla.yoctoproject.org/show_bug.cgi?id=6693

The following changes since commit 0f7df92e3d2dbbb4c94299171d5d0287887e0d28:

  clutter-gst: update to 3.0.8 (2015-08-11 09:28:52 -0700)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib lsandov1/package_deb-handle-non-ascii-6693
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lsandov1/package_deb-handle-non-ascii-6693

Leonardo Sandoval (2):
  insane.bbclass: Check non-ASCII characters on metadata
  package_deb.bbclass: Handle exception when encoding non-ASCII
    characters

 meta/classes/insane.bbclass      | 24 +++++++++++++++++++++++-
 meta/classes/package_deb.bbclass |  4 ++++
 2 files changed, 27 insertions(+), 1 deletion(-)

-- 
1.8.4.5

[PATCH 1/2] insane.bbclass: Check non-ASCII characters on metadata

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

Check if non-ASCII characters are present on recipe's metadata. Fields
taken into account: 'DESCRIPTION', 'SUMMARY', 'LICENSE' and 'SECTION'.

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
 meta/classes/insane.bbclass | 24 +++++++++++++++++++++++-
 1 file changed, 23 insertions(+), 1 deletion(-)

diff --git a/meta/classes/insane.bbclass b/meta/classes/insane.bbclass
index 9c05c86..d560eed 100644
--- a/meta/classes/insane.bbclass
+++ b/meta/classes/insane.bbclass
@@ -11,6 +11,7 @@
 #  -Check if packages contains .debug directories or .so files
 #   where they should be in -dev or -dbg
 #  -Check if config.log contains traces to broken autoconf tests
+#  -Check non-ascii characters on some package metadata
 #  -Ensure that binaries in base_[bindir|sbindir|libdir] do not link
 #   into exec_prefix
 #  -Check that scripts in base_[bindir|sbindir|libdir] do not reference
@@ -36,7 +37,7 @@ WARN_QA ?= "ldflags useless-rpaths rpaths staticdev libdir xorg-driver-abi \
 ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
             perms dep-cmp pkgvarcheck perm-config perm-line perm-link \
             split-strip packages-list pkgv-undefined var-undefined \
-            version-going-backwards expanded-d \
+            version-going-backwards expanded-d non-ascii \
             "
 
 ALL_QA = "${WARN_QA} ${ERROR_QA}"
@@ -947,6 +948,24 @@ def package_qa_check_expanded_d(path,name,d,elf,messages):
                         sane = False
     return sane
 
+def package_qa_check_ascii_encoding(keys, d):
+    def check_ascii_encoding(key):
+        sane = True
+        value = d.getVar(key, True)
+        if value:
+            try:
+                s = unicode(value)
+            except UnicodeDecodeError as e:
+                error_msg = "%s has non-ASCII characters" % key
+                sane = False
+                package_qa_handle_error("non-ascii", error_msg, d)
+        return sane
+
+    for key in keys:
+        sane = check_ascii_encoding(key)
+        if not sane:
+            break
+
 # The PACKAGE FUNC to scan each package
 python do_package_qa () {
     import subprocess
@@ -956,6 +975,9 @@ python do_package_qa () {
 
     bb.build.exec_func("read_subpackage_metadata", d)
 
+    # Check non-ascii characters on recipe's metadata
+    package_qa_check_ascii_encoding(['DESCRIPTION', 'SUMMARY', 'LICENSE', 'SECTION'], d)
+
     logdir = d.getVar('T', True)
     pkg = d.getVar('PN', True)
 
-- 
1.8.4.5

[PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters

[leonardo.sandoval.gonzalez]

From: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>

On package creation, handle exception when encoding non-ASCII characteres.

[YOCTO #6693]

Signed-off-by: Leonardo Sandoval <leonardo.sandoval.gonzalez@linux.intel.com>
---
 meta/classes/package_deb.bbclass | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/meta/classes/package_deb.bbclass b/meta/classes/package_deb.bbclass
index 9e1ed28..374adb6 100644
--- a/meta/classes/package_deb.bbclass
+++ b/meta/classes/package_deb.bbclass
@@ -171,6 +171,10 @@ python do_package_deb () {
             bb.utils.unlockfile(lf)
             ctrlfile.close()
             raise bb.build.FuncFailed("Missing field for deb generation: %s" % value)
+        except UnicodeDecodeError:
+            bb.utils.unlockfile(lf)
+            ctrlfile.close()
+            raise bb.build.FuncFailed("Non-ASCII characters found in one of the fields")
         # more fields
 
         custom_fields_chunk = get_package_additional_metadata("deb", localdata)
-- 
1.8.4.5

Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 376 bytes --]

On 13 August 2015 at 09:58, <leonardo.sandoval.gonzalez@linux.intel.com>
wrote:

> On package creation, handle exception when encoding non-ASCII characteres.
>

Debian control files are defined to be UTF-8, so the use of an ASCII
encoding method is wrong (
https://www.debian.org/doc/debian-policy/ch-controlfields.html).

(RPM appears to assume UTF-8 too)

Ross

[-- Attachment #2: Type: text/html, Size: 940 bytes --]

Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters

[Khem Raj]

On Thu, Aug 13, 2015 at 12:05 PM, Burton, Ross <ross.burton@intel.com> wrote:
> Debian control files are defined to be UTF-8, so the use of an ASCII
> encoding method is wrong
> (https://www.debian.org/doc/debian-policy/ch-controlfields.html).
>
> (RPM appears to assume UTF-8 too)

but check it still fine isnt it.

Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters

[Mark Hatle]

On 8/13/15 2:21 PM, Khem Raj wrote:
> On Thu, Aug 13, 2015 at 12:05 PM, Burton, Ross <ross.burton@intel.com> wrote:
>> Debian control files are defined to be UTF-8, so the use of an ASCII
>> encoding method is wrong
>> (https://www.debian.org/doc/debian-policy/ch-controlfields.html).
>>
>> (RPM appears to assume UTF-8 too)
> 
> but check it still fine isnt it.
> 

RPM isn't really utf-8.. it's more single 8-bit characters...  UTF-8 (1 byte
characters) work fine.. multibyte are not promised to work.

If you need special encoding (more then 8-bit characters) then you should be
using 'po' style files to translate the 8-bit chars to localized versions...

So checking (and fixing) the strings for various things is a good idea for RPM
at least.

--Mark

Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 635 bytes --]

On 13 August 2015 at 20:31, Mark Hatle <mark.hatle@windriver.com> wrote:

> RPM isn't really utf-8.. it's more single 8-bit characters...  UTF-8 (1
> byte
> characters) work fine.. multibyte are not promised to work.
>
> If you need special encoding (more then 8-bit characters) then you should
> be
> using 'po' style files to translate the 8-bit chars to localized
> versions...
>
> So checking (and fixing) the strings for various things is a good idea for
> RPM
> at least.
>

In the common case of "maintainer's name isn't ASCII" you don't want it
translated, and assuming UTF-8 everywhere mostly works.

Ross

[-- Attachment #2: Type: text/html, Size: 1083 bytes --]

Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 186 bytes --]

On 13 August 2015 at 20:21, Khem Raj <raj.khem@gmail.com> wrote:

> but check it still fine isnt it.
>

Of course, gracefully handling encoding failures is still sensible.

Ross

[-- Attachment #2: Type: text/html, Size: 569 bytes --]

Re: [PATCH 2/2] package_deb.bbclass: Handle exception when encoding non-ASCII characters

[Leonardo Sandoval]

Ross/Raj, thanks for your comments. I will send a v2 patches, this time 
with the correct checks (invalid characters are those non-UTF8) and 
patch titles.

On 08/13/2015 02:34 PM, Burton, Ross wrote:
> On 13 August 2015 at 20:21, Khem Raj <raj.khem@gmail.com> wrote:
>
>> but check it still fine isnt it.
>>
>
> Of course, gracefully handling encoding failures is still sensible.
>
> Ross
>