[PATCH 0/2] wic: looking by plugins and .wks file in all layers

[PATCH 0/2] wic: looking by plugins and .wks file in all layers

[João Henrique Ferreira de Freitas]

Hi,

This introduces a little wic usability enhanced. Now wic has the
ability to search in all layers looking by .wks files and plugins.

The aim is to provide the layers a way to keep .wks and plugins that
they use. Without this patch the user need to inform the full path of .wks, or put it
in scripts directory. Or, in case of plugins, only putting in scripts directory.

Example:

In .wks case, inside my custom layer I have the following files: 

scripts/lib/image/canned-wks/
 otns-directdisk-devel.wks
 otns-directdisk.wks
 otns-directsdcard-devel.wks
 otns-directsdcard.wks
 otns-nobml-directdisk-devel.wks
 otns-nobml-directsdcard-devel.wks

So, I do:

 $ wic list images
  otns-directdisk-devel                         Create a 'pcbios' direct disk image
  otns-directdisk                               Create a 'pcbios' direct disk image
  otns-nobml-directdisk-devel                   Create a 'pcbios' direct disk image
  otns-nobml-directsdcard-devel                 Create a 'pcbios' direct disk image
  otns-directsdcard                             Create a 'pcbios' direct disk image
  otns-directsdcard-devel                       Create a 'pcbios' direct disk image
  mkefidisk                                     Create an EFI disk image
  directdisk                                    Create a 'pcbios' direct disk image

And to plugins case, in another layer, I have:

scripts/lib/mic/plugins/source/
 bootimg-pcbios-myplugin.py


To sum up, this enhanced wic because users could keep .wks and plugins inside their own layers.

I don't know if using the 'scripts/lib/' is the best to keep files in layers. Any suggestion?

Thanks.

João Henrique Ferreira de Freitas (2):
  wic: add support to look in all layers and get .wks file
  wic: add support to look in all layers and get plugins

 scripts/lib/image/engine.py | 72 +++++++++++++++++++++++++++------------------
 scripts/lib/mic/plugin.py   | 28 ++++++++++++++----
 scripts/wic                 |  6 ++++
 3 files changed, 72 insertions(+), 34 deletions(-)

-- 
1.8.3.2

[PATCH 1/2] wic: add support to look in all layers and get .wks file

[João Henrique Ferreira de Freitas]

.wks file are looked in 'scripts/lib/image/canned-wks' directory on all
BBLAYERS variable returned by bitbake environment. If found, it will
be used.

The user could create your own .wks and keep it inside its layers. For
now the path must be <layer-dir>/scripts/lib/image/canned-wks.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/image/engine.py | 72 +++++++++++++++++++++++++++------------------
 scripts/wic                 |  6 ++++
 2 files changed, 49 insertions(+), 29 deletions(-)

diff --git a/scripts/lib/image/engine.py b/scripts/lib/image/engine.py
index 311737a..3bda1bf 100644
--- a/scripts/lib/image/engine.py
+++ b/scripts/lib/image/engine.py
@@ -90,6 +90,20 @@ def find_artifacts(image_name):
 
 
 CANNED_IMAGE_DIR = "lib/image/canned-wks" # relative to scripts
+SCRIPTS_CANNED_IMAGE_DIR = "scripts/" + CANNED_IMAGE_DIR
+
+def build_canned_image_list(dl):
+    layers_path = get_bitbake_var("BBLAYERS")
+    canned_wks_layer_dirs = []
+
+    for layer_path in layers_path.split():
+        path = os.path.join(layer_path, SCRIPTS_CANNED_IMAGE_DIR)
+        canned_wks_layer_dirs.append(path)
+
+    path = os.path.join(dl, CANNED_IMAGE_DIR)
+    canned_wks_layer_dirs.append(path)
+
+    return canned_wks_layer_dirs
 
 def find_canned_image(scripts_path, wks_file):
     """
@@ -97,15 +111,16 @@ def find_canned_image(scripts_path, wks_file):
 
     Return False if not found
     """
-    canned_wks_dir = os.path.join(scripts_path, CANNED_IMAGE_DIR)
-
-    for root, dirs, files in os.walk(canned_wks_dir):
-        for file in files:
-            if file.endswith("~") or file.endswith("#"):
-                continue
-            if file.endswith(".wks") and wks_file + ".wks" == file:
-                fullpath = os.path.join(canned_wks_dir, file)
-                return fullpath
+    layers_canned_wks_dir = build_canned_image_list(scripts_path)
+
+    for canned_wks_dir in layers_canned_wks_dir:
+        for root, dirs, files in os.walk(canned_wks_dir):
+            for file in files:
+                if file.endswith("~") or file.endswith("#"):
+                    continue
+                if file.endswith(".wks") and wks_file + ".wks" == file:
+                    fullpath = os.path.join(canned_wks_dir, file)
+                    return fullpath
     return None
 
 
@@ -113,32 +128,31 @@ def list_canned_images(scripts_path):
     """
     List the .wks files in the canned image dir, minus the extension.
     """
-    canned_wks_dir = os.path.join(scripts_path, CANNED_IMAGE_DIR)
-
-    for root, dirs, files in os.walk(canned_wks_dir):
-        for file in files:
-            if file.endswith("~") or file.endswith("#"):
-                continue
-            if file.endswith(".wks"):
-                fullpath = os.path.join(canned_wks_dir, file)
-                f = open(fullpath, "r")
-                lines = f.readlines()
-                for line in lines:
-                    desc = ""
-                    idx = line.find("short-description:")
-                    if idx != -1:
-                        desc = line[idx + len("short-description:"):].strip()
-                        break
-                basename = os.path.splitext(file)[0]
-                print "  %s\t\t%s" % (basename, desc)
+    layers_canned_wks_dir = build_canned_image_list(scripts_path)
+
+    for canned_wks_dir in layers_canned_wks_dir:
+        for root, dirs, files in os.walk(canned_wks_dir):
+            for file in files:
+                if file.endswith("~") or file.endswith("#"):
+                    continue
+                if file.endswith(".wks"):
+                    fullpath = os.path.join(canned_wks_dir, file)
+                    f = open(fullpath, "r")
+                    lines = f.readlines()
+                    for line in lines:
+                        desc = ""
+                        idx = line.find("short-description:")
+                        if idx != -1:
+                            desc = line[idx + len("short-description:"):].strip()
+                            break
+                    basename = os.path.splitext(file)[0]
+                    print "  %s\t\t%s" % (basename.ljust(30), desc)
 
 
 def list_canned_image_help(scripts_path, fullpath):
     """
     List the help and params in the specified canned image.
     """
-    canned_wks_dir = os.path.join(scripts_path, CANNED_IMAGE_DIR)
-
     f = open(fullpath, "r")
     lines = f.readlines()
     found = False
diff --git a/scripts/wic b/scripts/wic
index 4423340..2d3fd09 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -214,6 +214,12 @@ def wic_list_subcommand(args, usage_str):
 
     (options, args) = parser.parse_args(args)
 
+    bitbake_env_lines = find_bitbake_env_lines(None)
+    if not bitbake_env_lines:
+        print "Couldn't get bitbake environment, exiting."
+        sys.exit(1)
+    set_bitbake_env_lines(bitbake_env_lines)
+
     if not wic_list(args, scripts_path, options.properties_file):
         logging.error("Bad list arguments, exiting\n")
         parser.print_help()
-- 
1.8.3.2

[PATCH 2/2] wic: add support to look in all layers and get plugins

[João Henrique Ferreira de Freitas]

Plugins are looked in 'scripts/lib/mic/plugins/[type]/' directory on all
BBLAYERS variable returned by bitbake environment. If found, it will
be load at runtime.

The user could create your own plugin and keep it inside its layers. For
now the path must be <layer-dir>/scripts/lib/mic/plugins/[type]/. Where
'type' could be 'imager' or 'source'.

Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
---
 scripts/lib/mic/plugin.py | 28 +++++++++++++++++++++++-----
 1 file changed, 23 insertions(+), 5 deletions(-)

diff --git a/scripts/lib/mic/plugin.py b/scripts/lib/mic/plugin.py
index df03c15..bec33d6 100644
--- a/scripts/lib/mic/plugin.py
+++ b/scripts/lib/mic/plugin.py
@@ -20,12 +20,14 @@ import os, sys
 from mic import msger
 from mic import pluginbase
 from mic.utils import errors
-
+from mic.utils.oe.misc import *
 
 __ALL__ = ['PluginMgr', 'pluginmgr']
 
 PLUGIN_TYPES = ["imager", "source"] # TODO  "hook"
 
+PLUGIN_DIR = "/lib/mic/plugins" # relative to scripts
+SCRIPTS_PLUGIN_DIR = "scripts" + PLUGIN_DIR
 
 class PluginMgr(object):
     plugin_dirs = {}
@@ -42,8 +44,23 @@ class PluginMgr(object):
         mic_path = os.path.dirname(__file__)
         eos = mic_path.find('scripts') + len('scripts')
         scripts_path = mic_path[:eos]
+        self.scripts_path = scripts_path
+        self.plugin_dir = scripts_path + PLUGIN_DIR
+        self.layers_path = None
+
+    def _build_plugin_dir_list(self, dl, ptype):
+        if self.layers_path is None:
+            self.layers_path = get_bitbake_var("BBLAYERS")
+        layer_dirs = []
+
+        for layer_path in self.layers_path.split():
+            path = os.path.join(layer_path, SCRIPTS_PLUGIN_DIR, ptype)
+            layer_dirs.append(path)
 
-        self.plugin_dir = scripts_path + "/lib/mic/plugins"
+            path = os.path.join(dl, ptype)
+            layer_dirs.append(path)
+
+        return layer_dirs
 
     def append_dirs(self, dirs):
         for path in dirs:
@@ -56,7 +73,7 @@ class PluginMgr(object):
         path = os.path.abspath(os.path.expanduser(path))
 
         if not os.path.isdir(path):
-            msger.warning("Plugin dir is not a directory or does not exist: %s"\
+            msger.debug("Plugin dir is not a directory or does not exist: %s"\
                           % path)
             return
 
@@ -93,8 +110,9 @@ class PluginMgr(object):
         if ptype not in PLUGIN_TYPES:
             raise errors.CreatorError('%s is not valid plugin type' % ptype)
 
-        self._add_plugindir(os.path.join(self.plugin_dir, ptype))
-        self._load_all()
+        plugins_dir = self._build_plugin_dir_list(self.plugin_dir, ptype)
+
+        self.append_dirs(plugins_dir)
 
         return pluginbase.get_plugins(ptype)
 
-- 
1.8.3.2

Re: [PATCH 0/2] wic: looking by plugins and .wks file in all layers

[Tom Zanussi]

On Wed, 2014-05-14 at 22:37 -0300, João Henrique Ferreira de Freitas
wrote:
> Hi,
> 
> This introduces a little wic usability enhanced. Now wic has the
> ability to search in all layers looking by .wks files and plugins.
> 
> The aim is to provide the layers a way to keep .wks and plugins that
> they use. Without this patch the user need to inform the full path of .wks, or put it
> in scripts directory. Or, in case of plugins, only putting in scripts directory.
> 
> Example:
> 
> In .wks case, inside my custom layer I have the following files: 
> 
> scripts/lib/image/canned-wks/
>  otns-directdisk-devel.wks
>  otns-directdisk.wks
>  otns-directsdcard-devel.wks
>  otns-directsdcard.wks
>  otns-nobml-directdisk-devel.wks
>  otns-nobml-directsdcard-devel.wks
> 
> So, I do:
> 
>  $ wic list images
>   otns-directdisk-devel                         Create a 'pcbios' direct disk image
>   otns-directdisk                               Create a 'pcbios' direct disk image
>   otns-nobml-directdisk-devel                   Create a 'pcbios' direct disk image
>   otns-nobml-directsdcard-devel                 Create a 'pcbios' direct disk image
>   otns-directsdcard                             Create a 'pcbios' direct disk image
>   otns-directsdcard-devel                       Create a 'pcbios' direct disk image
>   mkefidisk                                     Create an EFI disk image
>   directdisk                                    Create a 'pcbios' direct disk image
> 
> And to plugins case, in another layer, I have:
> 
> scripts/lib/mic/plugins/source/
>  bootimg-pcbios-myplugin.py
> 
> 
> To sum up, this enhanced wic because users could keep .wks and plugins inside their own layers.
> 

Very nice enhancement, makes a lot of sense.

Acked-by: Tom Zanussi <tom.zanussi@linux.intel.com>

> I don't know if using the 'scripts/lib/' is the best to keep files in layers. Any suggestion?
> 

I think it's fine and is consistent with the current path.

Tom

> Thanks.
> 
> João Henrique Ferreira de Freitas (2):
>   wic: add support to look in all layers and get .wks file
>   wic: add support to look in all layers and get plugins
> 
>  scripts/lib/image/engine.py | 72 +++++++++++++++++++++++++++------------------
>  scripts/lib/mic/plugin.py   | 28 ++++++++++++++----
>  scripts/wic                 |  6 ++++
>  3 files changed, 72 insertions(+), 34 deletions(-)
> 

Re: [PATCH 2/2] wic: add support to look in all layers and get plugins

[Adrian Calianu]

Hi,

	A late review but I hope to be a pertinent one.
	

> -----Original Message-----
> From: openembedded-core-bounces@lists.openembedded.org
> [mailto:openembedded-core-bounces@lists.openembedded.org] On Behalf
> Of Joao Henrique Ferreira de Freitas
> Sent: Thursday, May 15, 2014 4:37 AM
> To: openembedded-core@lists.openembedded.org
> Cc: tom.zanussi@linux.intel.com
> Subject: [OE-core] [PATCH 2/2] wic: add support to look in all layers and get
> plugins
> 
> Plugins are looked in 'scripts/lib/mic/plugins/[type]/' directory on all
> BBLAYERS variable returned by bitbake environment. If found, it will be load
> at runtime.
> 
> The user could create your own plugin and keep it inside its layers. For now
> the path must be <layer-dir>/scripts/lib/mic/plugins/[type]/. Where 'type'
> could be 'imager' or 'source'.
> 
> Signed-off-by: João Henrique Ferreira de Freitas <joaohf@gmail.com>
> ---
>  scripts/lib/mic/plugin.py | 28 +++++++++++++++++++++++-----
>  1 file changed, 23 insertions(+), 5 deletions(-)
> 
> diff --git a/scripts/lib/mic/plugin.py b/scripts/lib/mic/plugin.py index
> df03c15..bec33d6 100644
> --- a/scripts/lib/mic/plugin.py
> +++ b/scripts/lib/mic/plugin.py
> @@ -20,12 +20,14 @@ import os, sys
>  from mic import msger
>  from mic import pluginbase
>  from mic.utils import errors
> -
> +from mic.utils.oe.misc import *
> 
>  __ALL__ = ['PluginMgr', 'pluginmgr']
> 
>  PLUGIN_TYPES = ["imager", "source"] # TODO  "hook"
> 
> +PLUGIN_DIR = "/lib/mic/plugins" # relative to scripts
> +SCRIPTS_PLUGIN_DIR = "scripts" + PLUGIN_DIR
> 
>  class PluginMgr(object):
>      plugin_dirs = {}
> @@ -42,8 +44,23 @@ class PluginMgr(object):
>          mic_path = os.path.dirname(__file__)
>          eos = mic_path.find('scripts') + len('scripts')
>          scripts_path = mic_path[:eos]
> +        self.scripts_path = scripts_path
> +        self.plugin_dir = scripts_path + PLUGIN_DIR
> +        self.layers_path = None
> +
> +    def _build_plugin_dir_list(self, dl, ptype):
> +        if self.layers_path is None:
> +            self.layers_path = get_bitbake_var("BBLAYERS")
> +        layer_dirs = []
> +
> +        for layer_path in self.layers_path.split():
> +            path = os.path.join(layer_path, SCRIPTS_PLUGIN_DIR, ptype)
> +            layer_dirs.append(path)
> 
> -        self.plugin_dir = scripts_path + "/lib/mic/plugins"
> +            path = os.path.join(dl, ptype)
> +            layer_dirs.append(path)
[Adrian Calianu] 
1) It looks like a bad indentation, here, adding 'dl' path for each found BBLAYER. Probably those two lines from above should not be part of 'for' statement.
2) If BBLAYERS variable is empty/None then the "self.layers_path.split()" will generate an error:
AttributeError: 'NoneType' object has no attribute 'split'

I don't think is good idea to force users to have BBLAYER variable set since will broke the raw mode of the wic.

> +
> +        return layer_dirs
> 
>      def append_dirs(self, dirs):
>          for path in dirs:
> @@ -56,7 +73,7 @@ class PluginMgr(object):
>          path = os.path.abspath(os.path.expanduser(path))
> 
>          if not os.path.isdir(path):
> -            msger.warning("Plugin dir is not a directory or does not exist: %s"\
> +            msger.debug("Plugin dir is not a directory or does not
> + exist: %s"\
>                            % path)
>              return
> 
> @@ -93,8 +110,9 @@ class PluginMgr(object):
>          if ptype not in PLUGIN_TYPES:
>              raise errors.CreatorError('%s is not valid plugin type' % ptype)
> 
> -        self._add_plugindir(os.path.join(self.plugin_dir, ptype))
> -        self._load_all()
> +        plugins_dir = self._build_plugin_dir_list(self.plugin_dir,
> + ptype)
> +
> +        self.append_dirs(plugins_dir)
> 
>          return pluginbase.get_plugins(ptype)
> 
> --
> 1.8.3.2
> 
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH 2/2] wic: add support to look in all layers and get plugins

[João Henrique Ferreira de Freitas]

Hi,

Thanks to point this.

> [Adrian Calianu]
> 1) It looks like a bad indentation, here, adding 'dl' path for each found BBLAYER. Probably those two lines from above should not be part of 'for' statement.

Yes, it is.

> 2) If BBLAYERS variable is empty/None then the "self.layers_path.split()" will generate an error:
> AttributeError: 'NoneType' object has no attribute 'split'

I will protect it. So if BBLAYERS is empty/None the default will be 
used. Then the raw mode of the wic works too.

-- 
João Henrique Ferreira de Freitas - joaohf_at_gmail.com
Campinas-SP-Brasil