[PATCH 1/5] wic: Add command to list available source plugins

[PATCH 1/5] wic: Add command to list available source plugins

[Tom Zanussi]

Add a 'wic list source-plugins' command enabling users to get a list
of valid partition --sources.  This is useful not only for determining
sources to use in .wks partition statements, but also for making sense
of errors in .wks partition processing.

Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
---
 scripts/lib/image/engine.py | 13 +++++++++++++
 scripts/lib/mic/plugin.py   | 16 ++++++++++++++++
 2 files changed, 29 insertions(+)

diff --git a/scripts/lib/image/engine.py b/scripts/lib/image/engine.py
index 1256236..b850bb9 100644
--- a/scripts/lib/image/engine.py
+++ b/scripts/lib/image/engine.py
@@ -174,6 +174,16 @@ def list_canned_image_help(scripts_path, fullpath):
             break
 
 
+def list_source_plugins():
+    """
+    List the available source plugins i.e. plugins available for --source.
+    """
+    plugins = pluginmgr.get_source_plugins()
+
+    for plugin in plugins:
+        print "  %s" % plugin
+
+
 def wic_create(args, wks_file, rootfs_dir, bootimg_dir, kernel_dir,
                native_sysroot, hdddir, staging_data_dir, scripts_path,
                image_output_dir, debug, properties_file, properties=None):
@@ -258,6 +268,9 @@ def wic_list(args, scripts_path, properties_file):
         if args[0] == "images":
             list_canned_images(scripts_path)
             return True
+        elif args[0] == "source-plugins":
+            list_source_plugins()
+            return True
         elif args[0] == "properties":
             return True
         else:
diff --git a/scripts/lib/mic/plugin.py b/scripts/lib/mic/plugin.py
index 585fd6d..f836950 100644
--- a/scripts/lib/mic/plugin.py
+++ b/scripts/lib/mic/plugin.py
@@ -117,6 +117,22 @@ class PluginMgr(object):
 
         return pluginbase.get_plugins(ptype)
 
+    def get_source_plugins(self):
+        """
+        Return list of available source plugins.
+        """
+        plugins_dir = self._build_plugin_dir_list(self.plugin_dir, 'source')
+
+        self.append_dirs(plugins_dir)
+
+        plugins = []
+
+        for _source_name, klass in self.get_plugins('source').iteritems():
+            plugins.append(_source_name)
+
+        return plugins
+
+
     def get_source_plugin_methods(self, source_name, methods):
         """
         The methods param is a dict with the method names to find.  On
-- 
1.8.3.1

[PATCH 2/5] wic: Add help text for 'wic list source-plugins'

[Tom Zanussi]

Add both short and long text for the new 'wic list source-plugins'
command.

Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
---
 scripts/lib/image/help.py | 20 +++++++++++++++++---
 1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/scripts/lib/image/help.py b/scripts/lib/image/help.py
index cb3112c..6de7ab0 100644
--- a/scripts/lib/image/help.py
+++ b/scripts/lib/image/help.py
@@ -199,6 +199,7 @@ wic_list_usage = """
 
  usage: wic list images
         wic list <image> help
+        wic list source-plugins
         wic list properties
         wic list properties <wks file>
         wic list property <property>
@@ -214,11 +215,14 @@ wic_list_usage = """
  The second form lists the detailed help information for a specific
  'canned' image.
 
- The third form enumerates all the possible values that exist and can
+ The third form enumerates all the available --sources (source
+ plugins).
+
+ The fourth form enumerates all the possible values that exist and can
  be specified in an OE kickstart (wks) file.
 
- The fourth form enumerates all the possible options that exist for
- the set of properties specified in a given OE kickstart (ks) file.
+ The fifth form enumerates all the possible options that exist for the
+ set of properties specified in a given OE kickstart (ks) file.
 
  The final form enumerates all the possible values that exist and can
  be specified for any given OE kickstart (wks) property.
@@ -234,6 +238,7 @@ NAME
 SYNOPSIS
     wic list images
     wic list <image> help
+    wic list source-plugins
     wic list properties
     wic list properties <wks file>
     wic list property <property>
@@ -255,6 +260,15 @@ DESCRIPTION
     The second form lists the detailed help information for a specific
     'canned' image.
 
+    The third form enumerates all the available --sources (source
+    plugins).  The contents of a given partition are driven by code
+    defined in 'source plugins'.  Users specify a specific plugin via
+    the --source parameter of the partition .wks command.  Normally
+    this is the 'rootfs' plugin but can be any of the more specialized
+    sources listed by the 'list source-plugins' command.  Users can
+    also add their own source plugins - see 'wic help plugins' for
+    details.
+
     The third form enumerates all the possible values that exist and
     can be specified in a OE kickstart (wks) file.  The output of this
     can be used by the third form to print the description and
-- 
1.8.3.1

[PATCH 3/5] wic: Print error if a partition specifies an invalid --source

[Tom Zanussi]

If a partition specified in a .wks file specifies a nonexistent
--source, print an error to that effect and exit.  The error text also
points the user to a command listing valid sources, and help on adding
a new source plugin.

Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
---
 scripts/lib/mic/kickstart/custom_commands/partition.py | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/scripts/lib/mic/kickstart/custom_commands/partition.py b/scripts/lib/mic/kickstart/custom_commands/partition.py
index 6b575c0..75ad6ad 100644
--- a/scripts/lib/mic/kickstart/custom_commands/partition.py
+++ b/scripts/lib/mic/kickstart/custom_commands/partition.py
@@ -130,6 +130,11 @@ class Wic_PartData(Mic_PartData):
                                              native_sysroot)
             return
 
+        plugins = pluginmgr.get_source_plugins()
+
+        if self.source not in plugins:
+            msger.error("The '%s' --source specified for %s doesn't exist.\n\tSee 'wic list source-plugins' for a list of available --sources.\n\tSee 'wic help source-plugins' for details on adding a new source plugin." % (self.source, self.mountpoint))
+
         self._source_methods = pluginmgr.get_source_plugin_methods(self.source, partition_methods)
         self._source_methods["do_configure_partition"](self, cr, cr_workdir,
                                                        oe_builddir,
-- 
1.8.3.1

[PATCH 0/5] wic: Source plugin updates

[Tom Zanussi]

This patchset adds a new 'list plugins' subcommand along with some
better error handling and messaging when users try to use
nonexistent plugins.  It also adds a new hook for general purpose
help topics and adds a 'plugins' topic as the first entry.

Addresses problems exposed in YOCTO #6481.

The following changes since commit 6879c3c903ec7d08cf0240281c83d85b4a42b6db:

  maintainers.inc: add info for libical (2014-07-08 11:20:14 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib.git tzanussi/wic-plugin-updates
  http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=tzanussi/wic-plugin-updates

Tom Zanussi (5):
  wic: Add command to list available source plugins
  wic: Add help text for 'wic list source-plugins'
  wic: Print error if a partition specifies an invalid --source
  wic: Add dummy subcommand and usage strings
  wic: Add general 'plugins' help topic

 scripts/lib/image/engine.py                        |  13 +++
 scripts/lib/image/help.py                          | 121 ++++++++++++++++++++-
 .../lib/mic/kickstart/custom_commands/partition.py |   5 +
 scripts/lib/mic/plugin.py                          |  16 +++
 scripts/wic                                        |  28 ++++-
 5 files changed, 173 insertions(+), 10 deletions(-)

-- 
1.8.3.1

[PATCH 4/5] wic: Add dummy subcommand and usage strings

[Tom Zanussi]

In order to reuse the existing subcommand infrastructure to display
various general-purpose help topics, add a dummy 'help_topic'
subcommand and usage string.  This allows users to invoke general help
topics by the natural form 'wic help <topic>' even though topic
doesn't correspond to a real subcommand.

Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
---
 scripts/wic | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/scripts/wic b/scripts/wic
index 2d3fd09..00eddfd 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -226,6 +226,19 @@ def wic_list_subcommand(args, usage_str):
         sys.exit(1)
 
 
+def wic_help_topic_subcommand(args, usage_str):
+    """
+    Command-line handling for help-only 'subcommands'.  This is
+    essentially a dummy command that doesn nothing but allow users to
+    use the existing subcommand infrastructure to display help on a
+    particular topic not attached to any particular subcommand.
+    """
+    pass
+
+
+wic_help_topic_usage = """
+"""
+
 subcommands = {
     "create": [wic_create_subcommand,
                wic_create_usage,
-- 
1.8.3.1

[PATCH 5/5] wic: Add general 'plugins' help topic

[Tom Zanussi]

Add a category for help topics with an initial help topic discussing
source plugins.

Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
---
 scripts/lib/image/help.py | 101 +++++++++++++++++++++++++++++++++++++++++++++-
 scripts/wic               |  15 ++++---
 2 files changed, 109 insertions(+), 7 deletions(-)

diff --git a/scripts/lib/image/help.py b/scripts/lib/image/help.py
index 6de7ab0..a4f27ab 100644
--- a/scripts/lib/image/help.py
+++ b/scripts/lib/image/help.py
@@ -87,7 +87,11 @@ wic_usage = """
     create            Create a new OpenEmbedded image
     list              List available values for options and image properties
 
- See 'wic help COMMAND' for more information on a specific command.
+ Help topics:
+    plugins            wic plugins - Overview and API
+
+ See 'wic help <COMMAND or HELP TOPIC>' for more information on a specific
+ command or help topic.
 """
 
 wic_help_usage = """
@@ -323,3 +327,98 @@ DESCRIPTION
         ["offset", "offset of the partition within the image"]
 
 """
+
+wic_plugins_help = """
+
+NAME
+    wic plugins - Overview and API
+
+DESCRIPTION
+    plugins allow wic functionality to be extended and specialized by
+    users.  This section documents the plugin interface, which is
+    currently restricted to 'source' plugins.
+
+    'Source' plugins provide a mechanism to customize various aspects
+    of the image generation process in wic, mainly the contents of
+    partitions.
+
+    Source plugins provide a mechanism for mapping values specified in
+    .wks files using the --source keyword to a particular plugin
+    implementation that populates a corresponding partition.
+
+    A source plugin is created as a subclass of SourcePlugin (see
+    scripts/lib/mic/pluginbase.py) and the plugin file containing it
+    is added to scripts/lib/mic/plugins/source/ to make the plugin
+    implementation available to the wic implementation.
+
+    Source plugins can also be implemented and added by external
+    layers - any plugins found in a scripts/lib/mic/plugins/source/
+    directory in an external layer will also be made available.
+
+    When the wic implementation needs to invoke a partition-specific
+    implementation, it looks for the plugin that has the same name as
+    the --source param given to that partition.  For example, if the
+    partition is set up like this:
+
+      part /boot --source bootimg-pcbios   ...
+
+    then the methods defined as class members of the plugin having the
+    matching bootimg-pcbios .name class member would be used.
+
+    To be more concrete, here's the plugin definition that would match
+    a '--source bootimg-pcbios' usage, along with an example method
+    that would be called by the wic implementation when it needed to
+    invoke an implementation-specific partition-preparation function:
+
+    class BootimgPcbiosPlugin(SourcePlugin):
+        name = 'bootimg-pcbios'
+
+    @classmethod
+        def do_prepare_partition(self, part, ...)
+
+    If the subclass itself doesn't implement a function, a 'default'
+    version in a superclass will be located and used, which is why all
+    plugins must be derived from SourcePlugin.
+
+    The SourcePlugin class defines the following methods, which is the
+    current set of methods that can be implemented/overridden by
+    --source plugins.  Any methods not implemented by a SourcePlugin
+    subclass inherit the implementations present in the SourcePlugin
+    class (see the SourcePlugin source for details):
+
+      do_prepare_partition()
+          Called to do the actual content population for a partition
+          i.e. it 'prepares' the final partition image which will be
+          incorporated into the disk image.
+
+      do_configure_partition()
+          Called before do_prepare_partition(), typically used to
+          create custom configuration files for a partition, for
+          example syslinux or grub config files.
+
+      do_install_disk()
+          Called after all partitions have been prepared and assembled
+          into a disk image.  This provides a hook to allow
+          finalization of a disk image e.g. to write an MBR to it.
+
+      do_stage_partition()
+          Special content staging hook called before
+          do_prepare_partition(), normally empty.
+
+          Typically, a partition will just use the passed-in parame
+          e.g straight bootimg_dir, etc, but in some cases, things
+          need to be more tailored e.g. to use a deploy dir + /boot,
+          etc.  This hook allows those files to be staged in a
+          customized fashion.  Not that get_bitbake_var() allows you
+          to acces non-standard variables that you might want to use
+          for this.
+
+    This scheme is extensible - adding more hooks is a simple matter
+    of adding more plugin methods to SourcePlugin and derived classes.
+    The code that then needs to call the plugin methods the uses
+    plugin.get_source_plugin_methods() to find the method(s) needed by
+    the call; this is done by filling up a dict with keys containing
+    the method names of interest - on success, these will be filled in
+    with the actual methods. Please see the implementation for
+    examples and details.
+"""
diff --git a/scripts/wic b/scripts/wic
index 00eddfd..ac3ed16 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -240,12 +240,15 @@ wic_help_topic_usage = """
 """
 
 subcommands = {
-    "create": [wic_create_subcommand,
-               wic_create_usage,
-               wic_create_help],
-    "list":   [wic_list_subcommand,
-               wic_list_usage,
-               wic_list_help],
+    "create":  [wic_create_subcommand,
+                wic_create_usage,
+                wic_create_help],
+    "list":    [wic_list_subcommand,
+                wic_list_usage,
+                wic_list_help],
+    "plugins": [wic_help_topic_subcommand,
+                wic_help_topic_usage,
+                wic_plugins_help],
 }
 
 
-- 
1.8.3.1