[PATCH v3 0/3] Fix performance issues for bitbake-layers show-recipes (Saves 95% time)

[PATCH v3 0/3] Fix performance issues for bitbake-layers show-recipes (Saves 95% time)

[Robert Yang]

* V3
  - Define self.collection_res in an init function

* V2
  - Save prioritized BBFILES to BBFILES_PRIORITIZED

  - "bitbake-layers show-recipes" requires 14 min to run which was too slow,
    these 3 patches can make it run in 40s which can save 95% time.

  - It becomes slow since the following commit:
    (85e03a64: tinfoil: Simplify remote datastore connections)

* V1
  - Initial version

The following changes since commit c25309ecde4e7ff81fb74a74fe06991dcdbbbfc8:

  bitbake: cooker: Ensure parser worker signal handlers are default (2020-09-08 10:18:02 +0100)

are available in the Git repository at:

  git://git.yoctoproject.org/poky-contrib rbt/bblayers
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=rbt/bblayers

Robert Yang (3):
  cooker.py: Save prioritized BBFILES to BBFILES_PRIORITIZED
  utils.py: get_file_layer(): Exit the loop when file is matched
  utils.py: get_file_layer(): Improve performance

 bitbake/lib/bb/cooker.py      |  2 +-
 bitbake/lib/bb/utils.py       | 17 ++++++++++++-----
 bitbake/lib/bblayers/query.py | 12 ++++++++++--
 3 files changed, 23 insertions(+), 8 deletions(-)

-- 
2.17.1

[PATCH v3 1/3] cooker.py: Save prioritized BBFILES to BBFILES_PRIORITIZED

[Robert Yang]

The original code saved BBFILES back to BBFILES without any changes which isn't
usefule, so remove that line. Now save prioritized BBFILES to
BBFILES_PRIORITIZED which can accelerate the query a lot for the one which
relies on it such as bb.utils.get_file_layer().

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/cooker.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index f3ff4274ad..e45755206e 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1761,10 +1761,10 @@ class CookerCollectFiles(object):
         collectlog.debug(1, "collecting .bb files")
 
         files = (config.getVar( "BBFILES") or "").split()
-        config.setVar("BBFILES", " ".join(files))
 
         # Sort files by priority
         files.sort( key=lambda fileitem: self.calc_bbfile_priority(fileitem)[0] )
+        config.setVar("BBFILES_PRIORITIZED", " ".join(files))
 
         if not len(files):
             files = self.get_bbfiles()
-- 
2.17.1

[PATCH v3 2/3] utils.py: get_file_layer(): Exit the loop when file is matched

[Robert Yang]

This can make "$ bitbake-layers show-recipes" save about 60% time (14min ->
6min) in my build (more than 3000 recipes)

The command "bitbake-layers show-recipes" calls bb.utils.get_file_layer() with
each recipe, and get_file_layer() compare the file with each item in BBFILES
which makes it very time consuming when there are a lot of recipes and items in
BBFILES. So use BBFILES_PRIORITIZED and exit when file is matched, it doesn't
make sense to go on the loop when file is matched.

And use fnmatchcase to replace of fnmatch since the comparison should be
case-sensitive.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/utils.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index e51b8ca508..5722fced97 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -1477,12 +1477,13 @@ def get_file_layer(filename, d):
         return match
 
     result = None
-    bbfiles = (d.getVar('BBFILES') or '').split()
+    bbfiles = (d.getVar('BBFILES_PRIORITIZED') or '').split()
     bbfilesmatch = False
     for bbfilesentry in bbfiles:
-        if fnmatch.fnmatch(filename, bbfilesentry):
+        if fnmatch.fnmatchcase(filename, bbfilesentry):
             bbfilesmatch = True
             result = path_to_layer(bbfilesentry)
+            break
 
     if not bbfilesmatch:
         # Probably a bbclass
-- 
2.17.1

[PATCH v3 3/3] utils.py: get_file_layer(): Improve performance

[Robert Yang]

The following code costs a lot of time when there are lot of layers and recipes:

     for collection in collections:
         collection_res[collection] = d.getVar('BBFILE_PATTERN_%s' % collection) or ''

My build has more than 100 layers and 3000 recipes, which calls d.getVar() 300K
(3000 * 100) times and makes 'bitbake-layers show-recipes' very slow, add a
keyword argument to get_file_layer() can fix the problem, it can save about 90%
time in my build (6min -> 40s).

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/utils.py       | 12 +++++++++---
 bitbake/lib/bblayers/query.py | 12 ++++++++++--
 2 files changed, 19 insertions(+), 5 deletions(-)

diff --git a/bitbake/lib/bb/utils.py b/bitbake/lib/bb/utils.py
index 5722fced97..fc78c5436b 100644
--- a/bitbake/lib/bb/utils.py
+++ b/bitbake/lib/bb/utils.py
@@ -1458,14 +1458,20 @@ def edit_bblayers_conf(bblayers_conf, add, remove, edit_cb=None):
 
     return (notadded, notremoved)
 
-
-def get_file_layer(filename, d):
-    """Determine the collection (as defined by a layer's layer.conf file) containing the specified file"""
+def get_collection_res(d):
     collections = (d.getVar('BBFILE_COLLECTIONS') or '').split()
     collection_res = {}
     for collection in collections:
         collection_res[collection] = d.getVar('BBFILE_PATTERN_%s' % collection) or ''
 
+    return collection_res
+
+
+def get_file_layer(filename, d, collection_res={}):
+    """Determine the collection (as defined by a layer's layer.conf file) containing the specified file"""
+    if not collection_res:
+        collection_res = get_collection_res(d)
+
     def path_to_layer(path):
         # Use longest path so we handle nested layers
         matchlen = 0
diff --git a/bitbake/lib/bblayers/query.py b/bitbake/lib/bblayers/query.py
index ee2db0efed..f5e3c84747 100644
--- a/bitbake/lib/bblayers/query.py
+++ b/bitbake/lib/bblayers/query.py
@@ -21,6 +21,10 @@ def plugin_init(plugins):
 
 
 class QueryPlugin(LayerPlugin):
+    def __init__(self):
+        super(QueryPlugin, self).__init__()
+        self.collection_res = {}
+
     def do_show_layers(self, args):
         """show current configured layers."""
         logger.plain("%s  %s  %s" % ("layer".ljust(20), "path".ljust(40), "priority"))
@@ -222,7 +226,6 @@ skipped recipes will also be listed, with a " (skipped)" suffix.
                             multilayer = True
                         if prov[0] != pref[0]:
                             same_ver = False
-
                     if (multilayer or not show_overlayed_only) and (same_ver or not show_same_ver_only):
                         if not items_listed:
                             logger.plain('=== %s ===' % title)
@@ -243,8 +246,13 @@ skipped recipes will also be listed, with a " (skipped)" suffix.
         else:
             return '?'
 
+    def get_collection_res(self):
+        if not self.collection_res:
+            self.collection_res = bb.utils.get_collection_res(self.tinfoil.config_data)
+        return self.collection_res
+
     def get_file_layerdir(self, filename):
-        layer = bb.utils.get_file_layer(filename, self.tinfoil.config_data)
+        layer = bb.utils.get_file_layer(filename, self.tinfoil.config_data, self.get_collection_res())
         return self.bbfile_collections.get(layer, None)
 
     def remove_layer_prefix(self, f):
-- 
2.17.1