[PATCH 0/2][Image Creator] Hob Cache for speed up switching

[PATCH 0/2][Image Creator] Hob Cache for speed up switching

[Ke Liping]

From: Liping Ke <liping.ke@intel.com>

The two patches are the implmentation of Hob Cache for accelarating
UI switching. If one configuration has been saved before, when user
switch back to the same configuration combination, we will use the
cache content instead of re-triggering parsing/loading phase.

The following changes since commit f974868cadf74d0cf8380b2c6a4152d2fa4590b5:
  Saul Wold (1):
        libx11: ensure nativesdk uses correct DEPENDS and XCB flags

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib lke/hob_cache
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=lke/hob_cache

Liping Ke (2):
  Use Pref as the centralized place for storing Hob attributes
  Cache target dependency tree for Hob

 bitbake/lib/bb/ui/crumbs/hobeventhandler.py |   19 ++++++++++++++++---
 bitbake/lib/bb/ui/crumbs/hobprefs.py        |    3 ++-
 bitbake/lib/bb/ui/crumbs/tasklistmodel.py   |    5 +++++
 bitbake/lib/bb/ui/hob.py                    |   24 +++++++++++++++++++++---
 4 files changed, 44 insertions(+), 7 deletions(-)

[PATCH 1/2][Image Creator] Hob Cache for speed up switching

[Ke Liping]

From: Liping Ke <liping.ke@intel.com>

This patch is the centralized place for storing Hob attributes.
Most of informations in Prefs will be used when caching user-select
packages information. Those information will be the cache hash-key
which is useful in deciding whether the configuration combination
value has been cached or not

Signed-off-by: Liping Ke <liping.ke@intel.com>
---
 bitbake/lib/bb/ui/crumbs/hobeventhandler.py |    6 +++---
 bitbake/lib/bb/ui/crumbs/hobprefs.py        |    3 ++-
 bitbake/lib/bb/ui/hob.py                    |    6 ++++--
 3 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/bitbake/lib/bb/ui/crumbs/hobeventhandler.py b/bitbake/lib/bb/ui/crumbs/hobeventhandler.py
index fa79e0c..8c8d9e2 100644
--- a/bitbake/lib/bb/ui/crumbs/hobeventhandler.py
+++ b/bitbake/lib/bb/ui/crumbs/hobeventhandler.py
@@ -99,7 +99,7 @@ class HobHandler(gobject.GObject):
             args.extend(argument)
             self.server.runCommand(args)
 
-    def handle_event(self, event, running_build, pbar):
+    def handle_event(self, event, running_build, pbar, hobprefs):
         if not event:
 	    return
 
@@ -161,11 +161,11 @@ class HobHandler(gobject.GObject):
             pbar.set_fraction(1.0)
         return
 
-    def event_handle_idle_func (self, eventHandler, running_build, pbar):
+    def event_handle_idle_func (self, eventHandler, running_build, pbar, hobrefs):
         # Consume as many messages as we can in the time available to us
         event = eventHandler.getEvent()
         while event:
-            self.handle_event(event, running_build, pbar)
+            self.handle_event(event, running_build, pbar, hobrefs)
             event = eventHandler.getEvent()
         return True
 
diff --git a/bitbake/lib/bb/ui/crumbs/hobprefs.py b/bitbake/lib/bb/ui/crumbs/hobprefs.py
index f186410..05e8170 100644
--- a/bitbake/lib/bb/ui/crumbs/hobprefs.py
+++ b/bitbake/lib/bb/ui/crumbs/hobprefs.py
@@ -143,7 +143,7 @@ class HobPrefs(gtk.Dialog):
             glib.idle_add(self.handler.reload_data)
 
     def __init__(self, configurator, handler, curr_sdk_mach, curr_distro, pclass,
-                 cpu_cnt, pmake, bbthread, image_types):
+                 cpu_cnt, pmake, bbthread, image_types, curr_mach):
         """
         """
         gtk.Dialog.__init__(self, "Preferences", None,
@@ -159,6 +159,7 @@ class HobPrefs(gtk.Dialog):
         self.configurator = configurator
 
         self.curr_sdk_mach = curr_sdk_mach
+        self.curr_mach = curr_mach
         self.curr_distro = curr_distro
         self.curr_package_format = pclass
         self.curr_output_type = None
diff --git a/bitbake/lib/bb/ui/hob.py b/bitbake/lib/bb/ui/hob.py
index 06d936e..76cb294 100644
--- a/bitbake/lib/bb/ui/hob.py
+++ b/bitbake/lib/bb/ui/hob.py
@@ -160,6 +160,7 @@ class MainWindow (gtk.Window):
         mach = combo.get_active_text()
         if mach != self.curr_mach:
             self.curr_mach = mach
+            self.prefs.curr_mach = mach
             # Flush this straight to the file as MACHINE is changed
             # independently of other 'Preferences'
             self.configurator.setLocalConfVar('MACHINE', mach)
@@ -887,7 +888,7 @@ def main (server, eventHandler):
     pkg, sep, pclass = pclasses[0].rpartition("_")
 
     prefs = HobPrefs(configurator, handler, sdk_mach, distro, pclass, cpu_cnt,
-                     pmake, bbthread, image_types)
+                     pmake, bbthread, image_types, mach)
     layers = LayerEditor(configurator, None)
     window = MainWindow(taskmodel, handler, configurator, prefs, layers, mach)
     prefs.set_parent_window(window)
@@ -918,7 +919,8 @@ def main (server, eventHandler):
                          handler.event_handle_idle_func,
                          eventHandler,
                          window.build,
-                         window.progress)
+                         window.progress,
+                         prefs)
 
     try:
         gtk.main()
-- 
1.7.0.4

[PATCH 2/2][Image Creator] Hob Cache for speed up switching

[Ke Liping]

From: Liping Ke <liping.ke@intel.com>

This patch is to cache target dependency tree for Hob (cache impl).
Hob User might switch between different configuration combinations
(machine, sdk-machine, distro, etc). Each dependency re-calcuation
might be time consuming. We use a hashset to cache those result for
avoiding recalc. Thus we can speed up UI data loading speed in
some cases.

Signed-off-by: Liping Ke <liping.ke@intel.com>
---
 bitbake/lib/bb/ui/crumbs/hobeventhandler.py |   13 +++++++++++++
 bitbake/lib/bb/ui/crumbs/tasklistmodel.py   |    5 +++++
 bitbake/lib/bb/ui/hob.py                    |   18 +++++++++++++++++-
 3 files changed, 35 insertions(+), 1 deletions(-)

diff --git a/bitbake/lib/bb/ui/crumbs/hobeventhandler.py b/bitbake/lib/bb/ui/crumbs/hobeventhandler.py
index 8c8d9e2..c6d3bd5 100644
--- a/bitbake/lib/bb/ui/crumbs/hobeventhandler.py
+++ b/bitbake/lib/bb/ui/crumbs/hobeventhandler.py
@@ -110,6 +110,19 @@ class HobHandler(gobject.GObject):
             self.emit("data-generated")
             self.generating = False
             if event._model:
+                # Currently, use these three property as the key, in the future
+                # more elements might be needed
+                mach = hobprefs.curr_mach
+                if mach == None:
+                    mach = ""
+                sdk_mach = hobprefs.curr_sdk_mach
+                if sdk_mach == None:
+                    sdk_mach = ""
+                distro = hobprefs.curr_distro
+                if distro == None:
+                    distro = ""
+                hash_key = mach + "," + sdk_mach + "," + distro
+                self.model.targetTreeCache[hash_key] = event._model 
                 self.model.populate(event._model)
         elif isinstance(event, bb.event.ConfigFilesFound):
             var = event._variable
diff --git a/bitbake/lib/bb/ui/crumbs/tasklistmodel.py b/bitbake/lib/bb/ui/crumbs/tasklistmodel.py
index 5e979b7..bcd6328 100644
--- a/bitbake/lib/bb/ui/crumbs/tasklistmodel.py
+++ b/bitbake/lib/bb/ui/crumbs/tasklistmodel.py
@@ -102,6 +102,11 @@ class TaskListModel(gtk.ListStore):
         self.packages = None
         self.images = None
         self.selected_image = None
+        # This var is used for caching target package dependency data. This
+        # data will be impacted by layers, distros, machine, sdk_machine value
+        # which is now all saved in HobPrefs data. For the hashset cache, it's
+        # obvious that the key will be those properties combination.
+        self.targetTreeCache = {}
         
         gtk.ListStore.__init__ (self,
                                 gobject.TYPE_STRING,
diff --git a/bitbake/lib/bb/ui/hob.py b/bitbake/lib/bb/ui/hob.py
index 76cb294..5f6fdf5 100644
--- a/bitbake/lib/bb/ui/hob.py
+++ b/bitbake/lib/bb/ui/hob.py
@@ -166,7 +166,23 @@ class MainWindow (gtk.Window):
             self.configurator.setLocalConfVar('MACHINE', mach)
             self.configurator.writeLocalConf()
             handler.set_machine(mach)
-            handler.reload_data()
+            mach = self.prefs.curr_mach
+            if mach == None:
+                mach = ""
+            sdk_mach = self.prefs.curr_sdk_mach
+            if sdk_mach == None:
+                sdk_mach = ""
+            distro = self.prefs.curr_distro
+            if distro == None:
+                distro = ""
+            hash_key = mach + "," + sdk_mach + "," + distro
+            # Only when this target tree has not been cached, need we reload
+            # the target tree data
+            if (self.model.targetTreeCache.has_key(hash_key) == False):
+                handler.reload_data()
+            # Otherwise, simply reload the cache data
+            else:
+                handler.model.populate(self.model.targetTreeCache[hash_key])
 
     def update_machines(self, handler, machines):
         active = 0
-- 
1.7.0.4

Re: [PATCH 0/2][Image Creator] Hob Cache for speed up switching

[Joshua Lock]

On Mon, 2011-07-11 at 15:31 +0800, Ke Liping wrote:
> From: Liping Ke <liping.ke@intel.com>
> 
> The two patches are the implmentation of Hob Cache for accelarating
> UI switching. If one configuration has been saved before, when user
> switch back to the same configuration combination, we will use the
> cache content instead of re-triggering parsing/loading phase.

I'm still not certain about merging this change, do you feel it's still
worth it once we've merged your change to reduce the tree generation
time?

If so we need to revise the second patch to invalidate the cached value
correctly. sdk-machine shouldn't affect the available packages but which
layers are enabled certainly will.

Cheers,
Joshua
-- 
Joshua Lock
        Yocto Project "Johannes factotum"
        Intel Open Source Technology Centre

Re: [PATCH 0/2][Image Creator] Hob Cache for speed up switching

[Ke, Liping]

Hi, Josh

Yes, definitely this patch's importance is much less than the previous one. But if more users are using hob, the cache would be very useful.
Because when using UI, users tend to switch between all combination of menu frequently. 
Even with the improved the speed, we still need around 7s. It would be nice if he switches back the UI is loaded immediately without waiting time.

Yes. Definitely we need to make sure the hash key was correct.
Currently I am not sure which elements should be the final correct one (the UI might changes in the future).
So my suggestion is that: we hold the patch. In the near future, when Hob is very mature and nice, we can decide the combination key.
Then we rebase the patch and resend the patch. How do you think about it?

Thanks a lot for your help!
criping

> -----Original Message-----
> From: yocto-bounces@yoctoproject.org
> [mailto:yocto-bounces@yoctoproject.org] On Behalf Of Joshua Lock
> Sent: Wednesday, July 20, 2011 8:42 AM
> To: yocto@yoctoproject.org
> Subject: Re: [yocto] [PATCH 0/2][Image Creator] Hob Cache for speed up
> switching
> 
> On Mon, 2011-07-11 at 15:31 +0800, Ke Liping wrote:
> > From: Liping Ke <liping.ke@intel.com>
> >
> > The two patches are the implmentation of Hob Cache for accelarating
> > UI switching. If one configuration has been saved before, when user
> > switch back to the same configuration combination, we will use the
> > cache content instead of re-triggering parsing/loading phase.
> 
> I'm still not certain about merging this change, do you feel it's still
> worth it once we've merged your change to reduce the tree generation
> time?
> 
> If so we need to revise the second patch to invalidate the cached value
> correctly. sdk-machine shouldn't affect the available packages but which
> layers are enabled certainly will.
> 
> Cheers,
> Joshua
> --
> Joshua Lock
>         Yocto Project "Johannes factotum"
>         Intel Open Source Technology Centre
> 
> _______________________________________________
> yocto mailing list
> yocto@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/yocto

Re: [PATCH 0/2][Image Creator] Hob Cache for speed up switching

[Joshua Lock]

On Thu, 2011-07-21 at 13:39 +0800, Ke, Liping wrote:
> Hi, Josh
> 
> Yes, definitely this patch's importance is much less than the previous one. But if more users are using hob, the cache would be very useful.
> Because when using UI, users tend to switch between all combination of menu frequently. 
> Even with the improved the speed, we still need around 7s. It would be nice if he switches back the UI is loaded immediately without waiting time.
> 
> Yes. Definitely we need to make sure the hash key was correct.
> Currently I am not sure which elements should be the final correct one (the UI might changes in the future).
> So my suggestion is that: we hold the patch. In the near future, when Hob is very mature and nice, we can decide the combination key.
> Then we rebase the patch and resend the patch. How do you think about it?

I agree on reviewing the status later.

Thanks,
Joshua
-- 
Joshua Lock
        Yocto Project "Johannes Factotum"
        Intel Open Source Technology Centre