[PATCH 0/2] Unicode support for Report Error and Qemu

[PATCH 0/2] Unicode support for Report Error and Qemu

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

These two patches add Unicode support for the report error bbclass
and the qemu bootlog.

[YOCTO #8225]


The following changes since commit eac61f37e36099f74485dab398b57f3812826d17:

  build-appliance-image: Update to jethro head revision (2015-10-01 17:55:11 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib mariano/bug8225p2v2
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=mariano/bug8225p2v2

Mariano Lopez (2):
  report-error.bbclass: Support Unicode reports
  oeqa/utils/qemurunner: Add support for Unicode from qemu

 meta/classes/report-error.bbclass | 11 +++++++----
 meta/lib/oeqa/utils/qemurunner.py | 15 +++++++++++----
 2 files changed, 18 insertions(+), 8 deletions(-)

-- 
1.8.4.5

[PATCH 1/2] report-error.bbclass: Support Unicode reports

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

Currently error-report doesn't manage Unicode because
the files are opened with the default codec.

This patch changes the codec of the files to UTF-8,
this way the reports will include Unicode characters.
This is useful for the qemu output when doing the
testimage task.

[YOCTO #8225]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/classes/report-error.bbclass | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/meta/classes/report-error.bbclass b/meta/classes/report-error.bbclass
index 040c29e..82b5bcd 100644
--- a/meta/classes/report-error.bbclass
+++ b/meta/classes/report-error.bbclass
@@ -9,22 +9,25 @@
 ERR_REPORT_DIR ?= "${LOG_DIR}/error-report"
 
 def errorreport_getdata(e):
+    import codecs
     logpath = e.data.getVar('ERR_REPORT_DIR', True)
     datafile = os.path.join(logpath, "error-report.txt")
-    with open(datafile) as f:
+    with codecs.open(datafile, 'r', 'utf-8') as f:
         data = f.read()
     return data
 
 def errorreport_savedata(e, newdata, file):
     import json
+    import codecs
     logpath = e.data.getVar('ERR_REPORT_DIR', True)
     datafile = os.path.join(logpath, file)
-    with open(datafile, "w") as f:
+    with codecs.open(datafile, 'w', 'utf-8') as f:
         json.dump(newdata, f, indent=4, sort_keys=True)
     return datafile
 
 python errorreport_handler () {
         import json
+        import codecs
 
         logpath = e.data.getVar('ERR_REPORT_DIR', True)
         datafile = os.path.join(logpath, "error-report.txt")
@@ -53,8 +56,8 @@ python errorreport_handler () {
             taskdata['task'] = task
             if log:
                 try:
-                    logFile = open(log, 'r')
-                    logdata = logFile.read().decode('utf-8')
+                    logFile = codecs.open(log, 'r', 'utf-8')
+                    logdata = logFile.read()
                     logFile.close()
                 except:
                     logdata = "Unable to read log file"
-- 
1.8.4.5

[PATCH 2/2] oeqa/utils/qemurunner: Add support for Unicode from qemu

[mariano.lopez]

From: Mariano Lopez <mariano.lopez@linux.intel.com>

The current state of qemurunner will drop the Unicode
characters received from qemu, this is because error
report web had problems with Unicode characters; now
that the server support Unicode, it is possible to
log all the output from qemu. So far the only Unicode
character seen is the copyright symbol.

This patch allows to get Unicode characters from the qemu
target and save the log in an UTF-8 file for latter use.

[YOCTO #8225]

Signed-off-by: Mariano Lopez <mariano.lopez@linux.intel.com>
---
 meta/lib/oeqa/utils/qemurunner.py | 15 +++++++++++----
 1 file changed, 11 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oeqa/utils/qemurunner.py b/meta/lib/oeqa/utils/qemurunner.py
index 9c878bc..6fe75b8 100644
--- a/meta/lib/oeqa/utils/qemurunner.py
+++ b/meta/lib/oeqa/utils/qemurunner.py
@@ -15,11 +15,18 @@ import select
 import errno
 import string
 import threading
+import codecs
 from oeqa.utils.dump import HostDumper
 
 import logging
 logger = logging.getLogger("BitBake.QemuRunner")
 
+# Get Unicode non printable control chars
+control_range = range(0,32)+range(127,160)
+control_chars = [unichr(x) for x in control_range
+                if unichr(x) not in string.printable]
+re_control_char = re.compile('[%s]' % re.escape("".join(control_chars)))
+
 class QemuRunner:
 
     def __init__(self, machine, rootfs, display, tmpdir, deploy_dir_image, logfile, boottime, dump_dir, dump_host_cmds):
@@ -63,9 +70,9 @@ class QemuRunner:
     def log(self, msg):
         if self.logfile:
             # It is needed to sanitize the data received from qemu
-            # because is possible to have control characters or Unicode
-            msg = "".join(filter(lambda x:x in string.printable, msg))
-            with open(self.logfile, "a") as f:
+            # because is possible to have control characters
+            msg = re_control_char.sub('', unicode(msg, 'utf-8'))
+            with codecs.open(self.logfile, "a", encoding="utf-8") as f:
                 f.write("%s" % msg)
 
     def getOutput(self, o):
@@ -176,7 +183,7 @@ class QemuRunner:
                 cmdline = p.read()
                 # It is needed to sanitize the data received
                 # because is possible to have control characters
-                cmdline = "".join(filter(lambda x:x in string.printable, cmdline))
+                cmdline = re_control_char.sub('', cmdline)
             try:
                 ips = re.findall("((?:[0-9]{1,3}\.){3}[0-9]{1,3})", cmdline.split("ip=")[1])
                 if not ips or len(ips) != 3:
-- 
1.8.4.5