[PATCH 0/7] fixes for toaster-eventreplay

[PATCH 0/7] fixes for toaster-eventreplay

[Elliot Smith]

Fix the Toaster event replay script so that it is able to run dumped
bitbake event JSON files.

For testing instructions, see
https://lists.yoctoproject.org/pipermail/toaster/2016-June/004875.html

The following changes since commit 025a2de5d67e491208fb7e76996ee0293d6fcf07
(toaster-next):

  toaster: views Fix most frequently built target in project reporting (2016-07-04 13:37:28 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib elliot/submit/ed/toaster/eventplay-9585
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/submit/ed/toaster/eventplay-9585

Ed Bartosh (6):
  eventreplay: add MockConnection.getEventHandle method
  eventprelay: implement setEventMask command
  eventreplay: fix event loading code
  eventreplay: replace MockConfigParameters with namedtuple
  eventreplay: reorganize imports
  eventreplay: rewrite the script

Elliot Smith (1):
  buildinfohelper: ensure task datetimes are timezone-aware

 bin/toaster-eventreplay      | 220 +++++++++++++++++--------------------------
 lib/bb/ui/buildinfohelper.py |   2 +-
 2 files changed, 87 insertions(+), 135 deletions(-)

--
2.7.4

[PATCH 1/7] eventreplay: add MockConnection.getEventHandle method

[Elliot Smith]

From: Ed Bartosh <ed.bartosh@linux.intel.com>

Fixed AttributeError: 'MockConnection' object has no attribute
'getEventHandle'

[YOCTO #9585]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 bin/toaster-eventreplay | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/bin/toaster-eventreplay b/bin/toaster-eventreplay
index bdddb90..14f4f00 100755
--- a/bin/toaster-eventreplay
+++ b/bin/toaster-eventreplay
@@ -86,6 +86,8 @@ class FileReadEventsServerConnection():
             """ do not do anything """
             pass
 
+        def getEventHandle(self):
+            pass
 
 
     class EventReader():
-- 
2.7.4

[PATCH 2/7] eventprelay: implement setEventMask command

[Elliot Smith]

From: Ed Bartosh <ed.bartosh@linux.intel.com>

Stored event mask list as self.eventmask for future use.
Fixed Exception: Command setEventMask not implemented.

[YOCTO #9585]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 bin/toaster-eventreplay | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/bin/toaster-eventreplay b/bin/toaster-eventreplay
index 14f4f00..03b5dde 100755
--- a/bin/toaster-eventreplay
+++ b/bin/toaster-eventreplay
@@ -53,6 +53,7 @@ class FileReadEventsServerConnection():
         """
         def __init__(self, sc):
             self._sc = sc
+            self.eventmask = []
 
         def runCommand(self, commandArray):
             """ emulates running a command on the server; only read-only commands are accepted """
@@ -79,6 +80,11 @@ class FileReadEventsServerConnection():
                     except Exception as e:
                         print(e)
                 return (dump, None)
+
+            elif command_name == 'setEventMask':
+                self.eventmask = commandArray[-1]
+                return True, None
+
             else:
                 raise Exception("Command %s not implemented" % commandArray[0])
 
-- 
2.7.4

[PATCH 3/7] eventreplay: fix event loading code

[Elliot Smith]

From: Ed Bartosh <ed.bartosh@linux.intel.com>

Event objects are represented by base64-encoded strings in
the event file and can't be loaded by existing eventreplay code.

Fixed the code of loading events from file by decoding base64 strings
into the binary form and loading them with pickle.load.

[YOCTO #9585]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 bin/toaster-eventreplay | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/bin/toaster-eventreplay b/bin/toaster-eventreplay
index 03b5dde..a107298 100755
--- a/bin/toaster-eventreplay
+++ b/bin/toaster-eventreplay
@@ -29,6 +29,7 @@
 from __future__ import print_function
 import os
 import sys, logging
+import codecs
 
 # mangle syspath to allow easy import of modules
 sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
@@ -117,7 +118,8 @@ class FileReadEventsServerConnection():
             try:
                 event_data = json.loads(line.strip())
                 event_class = _import_class(event_data['class'])
-                event_object = pickle.loads(json.loads(event_data['vars']))
+                event_str = event_data['vars'].encode('utf-8')
+                event_object = pickle.loads(codecs.decode(event_str, 'base64'))
             except ValueError as e:
                 print("Failed loading ", line)
                 raise e
-- 
2.7.4

[PATCH 4/7] eventreplay: replace MockConfigParameters with namedtuple

[Elliot Smith]

From: Ed Bartosh <ed.bartosh@linux.intel.com>

class MockConfigParameters has only one attribute and only __init__
method. Replacing it with namedtuple makes code less nested and more
readable.

[YOCTO #9585]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 bin/toaster-eventreplay | 14 ++------------
 1 file changed, 2 insertions(+), 12 deletions(-)

diff --git a/bin/toaster-eventreplay b/bin/toaster-eventreplay
index a107298..b9ab79e 100755
--- a/bin/toaster-eventreplay
+++ b/bin/toaster-eventreplay
@@ -30,6 +30,7 @@ from __future__ import print_function
 import os
 import sys, logging
 import codecs
+from collections import namedtuple
 
 # mangle syspath to allow easy import of modules
 sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
@@ -159,17 +160,6 @@ class FileReadEventsServerConnection():
 
         self.events = FileReadEventsServerConnection.EventReader(self)
 
-
-
-
-
-class MockConfigParameters():
-    """ stand-in for cookerdata.ConfigParameters; as we don't really config a cooker, this
-        serves just to supply needed interfaces for the toaster ui to work """
-    def __init__(self):
-        self.observe_only = True            # we can only read files
-
-
 # run toaster ui on our mock bitbake class
 if __name__ == "__main__":
     if len(sys.argv) < 2:
@@ -178,7 +168,7 @@ if __name__ == "__main__":
 
     file_name = sys.argv[-1]
     mock_connection = FileReadEventsServerConnection(file_name)
-    configParams = MockConfigParameters()
+    configParams = namedtuple('ConfigParams', ['observe_only'])(True)
 
     # run the main program and set exit code to the returned value
     sys.exit(toasterui.main(mock_connection.connection, mock_connection.events, configParams))
-- 
2.7.4

[PATCH 5/7] eventreplay: reorganize imports

[Elliot Smith]

From: Ed Bartosh <ed.bartosh@linux.intel.com>

Cleaned up module imports:
- Removed unused imports
- Removed import of print_function
- Removed duplicated imports
- Splitted importing bb.lib to 2 lines

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 bin/toaster-eventreplay | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/bin/toaster-eventreplay b/bin/toaster-eventreplay
index b9ab79e..7de3967 100755
--- a/bin/toaster-eventreplay
+++ b/bin/toaster-eventreplay
@@ -26,24 +26,19 @@
 # as a build eventlog, and the ToasterUI is used to process events in the file
 # and log data in the database
 
-from __future__ import print_function
-import os
-import sys, logging
+import sys
+import json
+import pickle
 import codecs
+
 from collections import namedtuple
 
 # mangle syspath to allow easy import of modules
-sys.path.insert(0, os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
-                                'lib'))
-
+from os.path import join, dirname, abspath
+sys.path.insert(0, join(dirname(dirname(abspath(__file__))), 'lib'))
 
 import bb.cooker
 from bb.ui import toasterui
-import sys
-import logging
-
-import json, pickle
-
 
 class FileReadEventsServerConnection():
     """  Emulates a connection to a bitbake server that feeds
-- 
2.7.4

[PATCH 6/7] eventreplay: rewrite the script

[Elliot Smith]

From: Ed Bartosh <ed.bartosh@linux.intel.com>

Rewritten toaster-eventreplay to make code working as expected,
more compact and readable.

[YOCTO #9585]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 bin/toaster-eventreplay | 207 +++++++++++++++++++-----------------------------
 1 file changed, 82 insertions(+), 125 deletions(-)

diff --git a/bin/toaster-eventreplay b/bin/toaster-eventreplay
index 7de3967..80967a0 100755
--- a/bin/toaster-eventreplay
+++ b/bin/toaster-eventreplay
@@ -21,11 +21,13 @@
 # with this program; if not, write to the Free Software Foundation, Inc.,
 # 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
 
+"""
+This command takes a filename as a single parameter. The filename is read
+as a build eventlog, and the ToasterUI is used to process events in the file
+and log data in the database
+"""
 
-# This command takes a filename as a single parameter. The filename is read
-# as a build eventlog, and the ToasterUI is used to process events in the file
-# and log data in the database
-
+import os
 import sys
 import json
 import pickle
@@ -40,130 +42,85 @@ sys.path.insert(0, join(dirname(dirname(abspath(__file__))), 'lib'))
 import bb.cooker
 from bb.ui import toasterui
 
-class FileReadEventsServerConnection():
-    """  Emulates a connection to a bitbake server that feeds
-        events coming actually read from a saved log file.
-    """
-
-    class MockConnection():
-        """ fill-in for the proxy to the server. we just return generic data
+class EventPlayer:
+    """Emulate a connection to a bitbake server."""
+
+    def __init__(self, eventfile, variables):
+        self.eventfile = eventfile
+        self.variables = variables
+        self.eventmask = []
+
+    def waitEvent(self, _timeout):
+        """Read event from the file."""
+        line = self.eventfile.readline().strip()
+        if not line:
+            return
+        try:
+            event_str = json.loads(line)['vars'].encode('utf-8')
+            event = pickle.loads(codecs.decode(event_str, 'base64'))
+            event_name = "%s.%s" % (event.__module__, event.__class__.__name__)
+            if event_name not in self.eventmask:
+                return
+            return event
+        except ValueError as err:
+            print("Failed loading ", line)
+            raise err
+
+    def runCommand(self, command_line):
+        """Emulate running a command on the server."""
+        name = command_line[0]
+
+        if name == "getVariable":
+            var_name = command_line[1]
+            variable = self.variables.get(var_name)
+            if variable:
+                return variable['v'], None
+            return None, "Missing variable %s" % var_name
+
+        elif name == "getAllKeysWithFlags":
+            dump = {}
+            flaglist = command_line[1]
+            for key, val in self.variables.items():
+                try:
+                    if not key.startswith("__"):
+                        dump[key] = {
+                            'v': val['v'],
+                            'history' : val['history'],
+                        }
+                        for flag in flaglist:
+                            dump[key][flag] = val[flag]
+                except Exception as err:
+                    print(err)
+            return (dump, None)
+
+        elif name == 'setEventMask':
+            self.eventmask = command_line[-1]
+            return True, None
+
+        else:
+            raise Exception("Command %s not implemented" % command_line[0])
+
+    def getEventHandle(self):
         """
-        def __init__(self, sc):
-            self._sc = sc
-            self.eventmask = []
-
-        def runCommand(self, commandArray):
-            """ emulates running a command on the server; only read-only commands are accepted """
-            command_name = commandArray[0]
-
-            if command_name == "getVariable":
-                if commandArray[1] in self._sc._variables:
-                    return (self._sc._variables[commandArray[1]]['v'], None)
-                return (None, "Missing variable")
-
-            elif command_name == "getAllKeysWithFlags":
-                dump = {}
-                flaglist = commandArray[1]
-                for k in self._sc._variables.keys():
-                    try:
-                        if not k.startswith("__"):
-                            v = self._sc._variables[k]['v']
-                            dump[k] = {
-                                'v' : v ,
-                                'history' : self._sc._variables[k]['history'],
-                            }
-                            for d in flaglist:
-                                dump[k][d] = self._sc._variables[k][d]
-                    except Exception as e:
-                        print(e)
-                return (dump, None)
-
-            elif command_name == 'setEventMask':
-                self.eventmask = commandArray[-1]
-                return True, None
-
-            else:
-                raise Exception("Command %s not implemented" % commandArray[0])
-
-        def terminateServer(self):
-            """ do not do anything """
-            pass
-
-        def getEventHandle(self):
-            pass
-
-
-    class EventReader():
-        def __init__(self, sc):
-            self._sc = sc
-            self.firstraise = 0
-
-        def _create_event(self, line):
-            def _import_class(name):
-                assert len(name) > 0
-                assert "." in name, name
-
-                components = name.strip().split(".")
-                modulename = ".".join(components[:-1])
-                moduleklass = components[-1]
-
-                module = __import__(modulename, fromlist=[str(moduleklass)])
-                return getattr(module, moduleklass)
-
-            # we build a toaster event out of current event log line
-            try:
-                event_data = json.loads(line.strip())
-                event_class = _import_class(event_data['class'])
-                event_str = event_data['vars'].encode('utf-8')
-                event_object = pickle.loads(codecs.decode(event_str, 'base64'))
-            except ValueError as e:
-                print("Failed loading ", line)
-                raise e
-
-            if not isinstance(event_object, event_class):
-                raise Exception("Error loading objects %s class %s ", event_object, event_class)
-
-            return event_object
-
-        def waitEvent(self, timeout):
-
-            nextline = self._sc._eventfile.readline()
-            if len(nextline) == 0:
-                # the build data ended, while toasterui still waits for events.
-                # this happens when the server was abruptly stopped, so we simulate this
-                self.firstraise += 1
-                if self.firstraise == 1:
-                    raise KeyboardInterrupt()
-                else:
-                    return None
-            else:
-                self._sc.lineno += 1
-            return self._create_event(nextline)
-
-
-    def _readVariables(self, variableline):
-        self._variables = json.loads(variableline.strip())['allvariables']
-
-
-    def __init__(self, file_name):
-        self.connection = FileReadEventsServerConnection.MockConnection(self)
-        self._eventfile = open(file_name, "r")
-
-        # we expect to have the variable dump at the start of the file
-        self.lineno = 1
-        self._readVariables(self._eventfile.readline())
-
-        self.events = FileReadEventsServerConnection.EventReader(self)
+        This method is called by toasterui.
+        The return value is passed to self.runCommand but not used there.
+        """
+        pass
+
+def main(argv):
+    with open(argv[-1]) as eventfile:
+        # load variables from the first line
+        variables = json.loads(eventfile.readline().strip())['allvariables']
+
+        params = namedtuple('ConfigParams', ['observe_only'])(True)
+        player = EventPlayer(eventfile, variables)
+
+        return toasterui.main(player, player, params)
 
 # run toaster ui on our mock bitbake class
 if __name__ == "__main__":
-    if len(sys.argv) < 2:
-        print("Usage: %s event.log " % sys.argv[0])
+    if len(sys.argv) != 2:
+        print("Usage: %s <event file>" % os.path.basename(sys.argv[0]))
         sys.exit(1)
 
-    file_name = sys.argv[-1]
-    mock_connection = FileReadEventsServerConnection(file_name)
-    configParams = namedtuple('ConfigParams', ['observe_only'])(True)
-
-    # run the main program and set exit code to the returned value
-    sys.exit(toasterui.main(mock_connection.connection, mock_connection.events, configParams))
+    sys.exit(main(sys.argv))
-- 
2.7.4

[PATCH 7/7] buildinfohelper: ensure task datetimes are timezone-aware

[Elliot Smith]

When using toaster-eventreplay to run a bitbake event file
through toasterui/buildinfohelper, errors occur when the
tasks are updated with buildstats info:

  RuntimeWarning: DateTimeField Task.started received a naive
  datetime (2016-07-06 09:15:22.070000) while time zone support
  is active.

This is because a method in buildinfohelper returns a naive
datetime, but Django is expecting timezone-aware datetimes.

Ensure that datetimes used to set the started/ended times on
tasks are converted to timezone-aware datetimes.

Signed-off-by: Elliot Smith <elliot.smith@intel.com>
---
 lib/bb/ui/buildinfohelper.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/ui/buildinfohelper.py b/lib/bb/ui/buildinfohelper.py
index d5ba629..447670c 100644
--- a/lib/bb/ui/buildinfohelper.py
+++ b/lib/bb/ui/buildinfohelper.py
@@ -125,7 +125,7 @@ class ORMWrapper(object):
         """
         Convert timestamp in seconds to Python datetime
         """
-        return datetime(1970, 1, 1) + timedelta(seconds=secs)
+        return timezone.make_aware(datetime(1970, 1, 1) + timedelta(seconds=secs))
 
     # pylint: disable=no-self-use
     # we disable detection of no self use in functions because the methods actually work on the object
-- 
2.7.4