[PATCH 0/4] Load devtool and reciptool plugins in a well-defined manner

[PATCH 0/4] Load devtool and reciptool plugins in a well-defined manner

[Ola x Nilsson]

From: Ola x Nilsson <ola.x.nilsson@axis.com>

Make devtool and recipetool load plugins in a well-defined and expected manner.

devtool and reciptool search for pluings first in scripts/lib/<tool>
and then in BBPATH.  Each found plugin is loaded, which means that any
plugin found later in BBPATH shadows previously loaded plugins.
plugin_init is loaded for all plugins, no matter whether they are
later replaced.  register_plugin is called once for each time a plugin
with a certain name is found, but it is always called on the last
loaded plugin.

Change the scriptutils.load_plugins function to never load a plugin
file if one with the same filename has already been loaded.  This
means that the first found plugin wins.

Change the load order to first search BBPATH and then
scripts/lib/tools.  This means that plugins in scripts/lib/<tool> can
be shadowed by plugins in layers.

Ola x Nilsson (4):
  devtool: selftest: add test for devtool plugin loading
  recipetool: selftest: Add test for recipetool plugin loading
  devtool: Load plugins in a well defined order
  recipetool: Load plugins in a well defined order

 meta-selftest/lib/devtool/bbpath.py    | 44 ++++++++++++++++++++++++++++++++++
 meta-selftest/lib/recipetool/bbpath.py | 41 +++++++++++++++++++++++++++++++
 meta/lib/oeqa/selftest/devtool.py      | 43 +++++++++++++++++++++++++++++++++
 meta/lib/oeqa/selftest/recipetool.py   | 44 ++++++++++++++++++++++++++++++++++
 scripts/devtool                        |  3 ++-
 scripts/lib/scriptutils.py             |  8 +++++--
 scripts/recipetool                     |  4 ++--
 7 files changed, 182 insertions(+), 5 deletions(-)
 create mode 100644 meta-selftest/lib/devtool/bbpath.py
 create mode 100644 meta-selftest/lib/recipetool/bbpath.py

-- 
2.1.4

[PATCH 1/4] devtool: selftest: add test for devtool plugin loading

[Ola x Nilsson]

From: Ola x Nilsson <ola.x.nilsson@axis.com>

Test that devtool plugins are loaded in a well defined order.

Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
---
 meta-selftest/lib/devtool/bbpath.py | 44 +++++++++++++++++++++++++++++++++++++
 meta/lib/oeqa/selftest/devtool.py   | 43 ++++++++++++++++++++++++++++++++++++
 2 files changed, 87 insertions(+)
 create mode 100644 meta-selftest/lib/devtool/bbpath.py

diff --git a/meta-selftest/lib/devtool/bbpath.py b/meta-selftest/lib/devtool/bbpath.py
new file mode 100644
index 0000000..5e8ffb3
--- /dev/null
+++ b/meta-selftest/lib/devtool/bbpath.py
@@ -0,0 +1,44 @@
+import argparse
+
+already_loaded = False
+kept_context = None
+
+def plugin_name(filename):
+    return os.path.splitext(os.path.basename(filename))[0]
+
+def plugin_init(plugins):
+    global already_loaded
+    already_loaded = plugin_name(__file__) in (plugin_name(p.__name__) for p in plugins)
+
+def print_name(args, config, basepath, workspace):
+    print (__file__)
+
+def print_bbdir(args, config, basepath, workspace):
+    print (__file__.replace('/lib/devtool/bbpath.py',''))
+
+def print_registered(args, config, basepath, workspace):
+    global kept_context
+    print(kept_context.loaded)
+
+def multiloaded(args, config, basepath, workspace):
+    global already_loaded
+    print("yes" if already_loaded else "no")
+
+def register_commands(subparsers, context):
+    global kept_context
+    kept_context = context
+    if 'loaded' in context.__dict__:
+        context.loaded += 1
+    else:
+        context.loaded = 1
+
+    def addparser(name, helptxt, func):
+        parser = subparsers.add_parser(name, help=helptxt,
+                                       formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+        parser.set_defaults(func=func)
+        return parser
+
+    addparser('pluginfile', 'Print the filename of this plugin', print_name)
+    addparser('bbdir', 'Print the BBPATH directory of this plugin', print_bbdir)
+    addparser('count', 'How many times have this plugin been registered.', print_registered)
+    addparser('multiloaded', 'How many times have this plugin been initialized', multiloaded)
diff --git a/meta/lib/oeqa/selftest/devtool.py b/meta/lib/oeqa/selftest/devtool.py
index e992dcf..cd9077f 100644
--- a/meta/lib/oeqa/selftest/devtool.py
+++ b/meta/lib/oeqa/selftest/devtool.py
@@ -1191,6 +1191,49 @@ class DevtoolTests(DevtoolBase):
         result = runCmd("devtool --quiet selftest-reverse \"%s\"" % s)
         self.assertEqual(result.output, s[::-1])
 
+    def _copy_file_with_cleanup(self, srcfile, basedstdir, *paths):
+        dstdir = basedstdir
+        self.assertTrue(os.path.exists(dstdir))
+        for p in paths:
+            dstdir = os.path.join(dstdir, p)
+            if not os.path.exists(dstdir):
+                os.makedirs(dstdir)
+                self.track_for_cleanup(dstdir)
+        dstfile = os.path.join(dstdir, os.path.basename(srcfile))
+        if srcfile != dstfile:
+            shutil.copy(srcfile, dstfile)
+            self.track_for_cleanup(dstfile)
+
+    def test_devtool_load_plugin(self):
+        """Test that devtool loads only the first found plugin in BBPATH."""
+
+        self.track_for_cleanup(self.workspacedir)
+        self.add_command_to_tearDown('bitbake-layers remove-layer */workspace')
+
+        devtool = runCmd("which devtool")
+        fromname = runCmd("devtool --quiet pluginfile")
+        srcfile = fromname.output
+        bbpath = get_bb_var('BBPATH')
+        searchpath = bbpath.split(':') + [os.path.dirname(devtool.output)]
+        plugincontent = []
+        with open(srcfile) as fh:
+            plugincontent = fh.readlines()
+        try:
+            self.assertIn('meta-selftest', srcfile, 'wrong bbpath plugin found')
+            for path in searchpath:
+                self._copy_file_with_cleanup(srcfile, path, 'lib', 'devtool')
+            result = runCmd("devtool --quiet count")
+            self.assertEqual(result.output, '1')
+            result = runCmd("devtool --quiet multiloaded")
+            self.assertEqual(result.output, "no")
+            for path in searchpath:
+                result = runCmd("devtool --quiet bbdir")
+                self.assertEqual(result.output, path)
+                os.unlink(os.path.join(result.output, 'lib', 'devtool', 'bbpath.py'))
+        finally:
+            with open(srcfile, 'w') as fh:
+                fh.writelines(plugincontent)
+
     def _setup_test_devtool_finish_upgrade(self):
         # Check preconditions
         self.assertTrue(not os.path.exists(self.workspacedir), 'This test cannot be run with a workspace directory under the build directory')
-- 
2.1.4

[PATCH 2/4] recipetool: selftest: Add test for recipetool plugin loading

[Ola x Nilsson]

From: Ola x Nilsson <ola.x.nilsson@axis.com>

Test that recipetool plugins are loaded in a well defined order.

Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
---
 meta-selftest/lib/recipetool/bbpath.py | 41 +++++++++++++++++++++++++++++++
 meta/lib/oeqa/selftest/recipetool.py   | 44 ++++++++++++++++++++++++++++++++++
 2 files changed, 85 insertions(+)
 create mode 100644 meta-selftest/lib/recipetool/bbpath.py

diff --git a/meta-selftest/lib/recipetool/bbpath.py b/meta-selftest/lib/recipetool/bbpath.py
new file mode 100644
index 0000000..783b2dc
--- /dev/null
+++ b/meta-selftest/lib/recipetool/bbpath.py
@@ -0,0 +1,41 @@
+import argparse
+
+already_loaded = False
+register_count = 0
+
+def plugin_name(filename):
+    return os.path.splitext(os.path.basename(filename))[0]
+
+def plugin_init(plugins):
+    global already_loaded
+    already_loaded = plugin_name(__file__) in (plugin_name(p.__name__) for p in plugins)
+
+def print_name(opts):
+    print (__file__)
+
+def print_bbdir(opts):
+    print (__file__.replace('/lib/recipetool/bbpath.py',''))
+
+def print_registered(opts):
+    #global kept_context
+    #print(kept_context.loaded)
+    print ("1")
+
+def multiloaded(opts):
+    global already_loaded
+    print("yes" if already_loaded else "no")
+
+def register_commands(subparsers):
+    global register_count
+    register_count += 1
+
+    def addparser(name, helptxt, func):
+        parser = subparsers.add_parser(name, help=helptxt,
+                                       formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+        parser.set_defaults(func=func)
+        return parser
+
+    addparser('pluginfile', 'Print the filename of this plugin', print_name)
+    addparser('bbdir', 'Print the BBPATH directory of this plugin', print_bbdir)
+    addparser('count', 'How many times have this plugin been registered.', print_registered)
+    addparser('multiloaded', 'How many times have this plugin been initialized', multiloaded)
diff --git a/meta/lib/oeqa/selftest/recipetool.py b/meta/lib/oeqa/selftest/recipetool.py
index db1f8de..bcc2b46 100644
--- a/meta/lib/oeqa/selftest/recipetool.py
+++ b/meta/lib/oeqa/selftest/recipetool.py
@@ -1,5 +1,6 @@
 import os
 import logging
+import shutil
 import tempfile
 import urllib.parse
 
@@ -485,6 +486,47 @@ class RecipetoolTests(RecipetoolBase):
         inherits = ['pkgconfig', 'autotools']
         self._test_recipe_contents(recipefile, checkvars, inherits)
 
+    def _copy_file_with_cleanup(self, srcfile, basedstdir, *paths):
+        dstdir = basedstdir
+        self.assertTrue(os.path.exists(dstdir))
+        for p in paths:
+            dstdir = os.path.join(dstdir, p)
+            if not os.path.exists(dstdir):
+                os.makedirs(dstdir)
+                self.track_for_cleanup(dstdir)
+        dstfile = os.path.join(dstdir, os.path.basename(srcfile))
+        if srcfile != dstfile:
+            shutil.copy(srcfile, dstfile)
+            self.track_for_cleanup(dstfile)
+
+    def test_recipetool_load_plugin(self):
+        """Test that recipetool loads only the first found plugin in BBPATH."""
+
+        recipetool = runCmd("which recipetool")
+        fromname = runCmd("recipetool --quiet pluginfile")
+        srcfile = fromname.output
+        bbpath = get_bb_var('BBPATH')
+        searchpath = bbpath.split(':') + [os.path.dirname(recipetool.output)]
+        plugincontent = []
+        with open(srcfile) as fh:
+            plugincontent = fh.readlines()
+        try:
+            self.assertIn('meta-selftest', srcfile, 'wrong bbpath plugin found')
+            for path in searchpath:
+                self._copy_file_with_cleanup(srcfile, path, 'lib', 'recipetool')
+            result = runCmd("recipetool --quiet count")
+            self.assertEqual(result.output, '1')
+            result = runCmd("recipetool --quiet multiloaded")
+            self.assertEqual(result.output, "no")
+            for path in searchpath:
+                result = runCmd("recipetool --quiet bbdir")
+                self.assertEqual(result.output, path)
+                os.unlink(os.path.join(result.output, 'lib', 'recipetool', 'bbpath.py'))
+        finally:
+            with open(srcfile, 'w') as fh:
+                fh.writelines(plugincontent)
+
+
 class RecipetoolAppendsrcBase(RecipetoolBase):
     def _try_recipetool_appendsrcfile(self, testrecipe, newfile, destfile, options, expectedlines, expectedfiles):
         cmd = 'recipetool appendsrcfile %s %s %s %s %s' % (options, self.templayerdir, testrecipe, newfile, destfile)
@@ -574,6 +616,8 @@ class RecipetoolAppendsrcBase(RecipetoolBase):
         self.assertIn(filesdir, filesextrapaths)
 
 
+
+
 class RecipetoolAppendsrcTests(RecipetoolAppendsrcBase):
 
     @testcase(1273)
-- 
2.1.4

[PATCH 3/4] devtool: Load plugins in a well defined order

[Ola x Nilsson]

From: Ola x Nilsson <ola.x.nilsson@axis.com>

To allow devtool plugins in one layer to shadow another in a well
defined way, first search BBPATH/lib/devtool directories and then
scripts/lib/devool and load only the first found.

The previous search and load loop would load all found plugins with the
ones found later replacing any found before.

Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
---
 scripts/devtool            | 3 ++-
 scripts/lib/scriptutils.py | 8 ++++++--
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/scripts/devtool b/scripts/devtool
index 0c32c50..2197493 100755
--- a/scripts/devtool
+++ b/scripts/devtool
@@ -294,7 +294,8 @@ def main():
         finally:
             tinfoil.shutdown()
 
-    for path in [scripts_path] + global_args.bbpath.split(':'):
+    # Search BBPATH first to allow layers to override plugins in scripts_path
+    for path in global_args.bbpath.split(':') + [scripts_path]:
         pluginpath = os.path.join(path, 'lib', 'devtool')
         scriptutils.load_plugins(logger, plugins, pluginpath)
 
diff --git a/scripts/lib/scriptutils.py b/scripts/lib/scriptutils.py
index 5ccc027..27d82b6 100644
--- a/scripts/lib/scriptutils.py
+++ b/scripts/lib/scriptutils.py
@@ -52,10 +52,14 @@ def load_plugins(logger, plugins, pluginpath):
             if fp:
                 fp.close()
 
+    def plugin_name(filename):
+        return os.path.splitext(os.path.basename(filename))[0]
+
+    known_plugins = [plugin_name(p.__name__) for p in plugins]
     logger.debug('Loading plugins from %s...' % pluginpath)
     for fn in glob.glob(os.path.join(pluginpath, '*.py')):
-        name = os.path.splitext(os.path.basename(fn))[0]
-        if name != '__init__':
+        name = plugin_name(fn)
+        if name != '__init__' and name not in known_plugins:
             plugin = load_plugin(name)
             if hasattr(plugin, 'plugin_init'):
                 plugin.plugin_init(plugins)
-- 
2.1.4

[PATCH 4/4] recipetool: Load plugins in a well defined order

[Ola x Nilsson]

From: Ola x Nilsson <ola.x.nilsson@axis.com>

To allow recipetool plugins in one layer to shadow another in a well
defined way, first search BBPATH/lib/recipetool directories and then
scripts/lib/recipetool and load only the first found.

The previous search and load loop would load all found plugins with the
ones found later replacing any found before.

Signed-off-by: Ola x Nilsson <ola.x.nilsson@axis.com>
---
 scripts/recipetool | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/recipetool b/scripts/recipetool
index 1052cd2..00c9007 100755
--- a/scripts/recipetool
+++ b/scripts/recipetool
@@ -78,8 +78,8 @@ def main():
 
     tinfoil = tinfoil_init(False)
     try:
-        for path in ([scripts_path] +
-                    tinfoil.config_data.getVar('BBPATH', True).split(':')):
+        for path in (tinfoil.config_data.getVar('BBPATH', True).split(':')
+                     + [scripts_path]):
             pluginpath = os.path.join(path, 'lib', 'recipetool')
             scriptutils.load_plugins(logger, plugins, pluginpath)
 
-- 
2.1.4

Re: [PATCH 0/4] Load devtool and reciptool plugins in a well-defined manner

[Ola x Nilsson]

Ping?

-- 
Ola x Nilsson

> -----Original Message-----
> From: Ola x Nilsson [mailto:olani@axis.com]
> Sent: den 25 oktober 2016 13:04
> To: Ola x Nilsson; openembedded-core@lists.openembedded.org
> Subject: [PATCH 0/4] Load devtool and reciptool plugins in a well-defined
> manner
> 
> From: Ola x Nilsson <ola.x.nilsson@axis.com>
> 
> Make devtool and recipetool load plugins in a well-defined and expected
> manner.
> 
> devtool and reciptool search for pluings first in scripts/lib/<tool> and then in
> BBPATH.  Each found plugin is loaded, which means that any plugin found
> later in BBPATH shadows previously loaded plugins.
> plugin_init is loaded for all plugins, no matter whether they are later
> replaced.  register_plugin is called once for each time a plugin with a certain
> name is found, but it is always called on the last loaded plugin.
> 
> Change the scriptutils.load_plugins function to never load a plugin file if one
> with the same filename has already been loaded.  This means that the first
> found plugin wins.
> 
> Change the load order to first search BBPATH and then scripts/lib/tools.  This
> means that plugins in scripts/lib/<tool> can be shadowed by plugins in layers.
> 
> Ola x Nilsson (4):
>   devtool: selftest: add test for devtool plugin loading
>   recipetool: selftest: Add test for recipetool plugin loading
>   devtool: Load plugins in a well defined order
>   recipetool: Load plugins in a well defined order
> 
>  meta-selftest/lib/devtool/bbpath.py    | 44
> ++++++++++++++++++++++++++++++++++
>  meta-selftest/lib/recipetool/bbpath.py | 41
> +++++++++++++++++++++++++++++++
>  meta/lib/oeqa/selftest/devtool.py      | 43
> +++++++++++++++++++++++++++++++++
>  meta/lib/oeqa/selftest/recipetool.py   | 44
> ++++++++++++++++++++++++++++++++++
>  scripts/devtool                        |  3 ++-
>  scripts/lib/scriptutils.py             |  8 +++++--
>  scripts/recipetool                     |  4 ++--
>  7 files changed, 182 insertions(+), 5 deletions(-)  create mode 100644 meta-
> selftest/lib/devtool/bbpath.py
>  create mode 100644 meta-selftest/lib/recipetool/bbpath.py
> 
> --
> 2.1.4

Re: [PATCH 0/4] Load devtool and reciptool plugins in a well-defined manner

[Paul Eggleton]

On Tue, 06 Dec 2016 07:37:16 Ola x Nilsson wrote:
> Ping?

I guess Ross / Richard were waiting for my ack - sorry about that.

Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com>

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/4] Load devtool and reciptool plugins in a well-defined manner

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 677 bytes --]

Sorry, slipped down in my inbox.  Thanks for acking Paul, these are in
staging now.

Ross

On 7 December 2016 at 21:16, Paul Eggleton <paul.eggleton@linux.intel.com>
wrote:

> On Tue, 06 Dec 2016 07:37:16 Ola x Nilsson wrote:
> > Ping?
>
> I guess Ross / Richard were waiting for my ack - sorry about that.
>
> Acked-by: Paul Eggleton <paul.eggleton@linux.intel.com>
>
> Cheers,
> Paul
>
> --
>
> Paul Eggleton
> Intel Open Source Technology Centre
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 1402 bytes --]