[PATCH 00/17] Build wic images with bitbake

[PATCH 00/17] Build wic images with bitbake

[Ed Bartosh]

Hi Reviewers,

This patchset adds new image type 'wic' to the list of image types
supported by bitbake. This should allow to build partitioned image
the same way as any other images.

New image type can be used in image recipes almost the same way as other
image types. There is just one difference - <image>[.<machine>].wks file
should be put to the same location as image recipe.

To enable this functionality I've implemented generating .env files with
bitbake variables and reading them by wic. This seems to be the only way
to get bitbake variables in wic when it's run by bitbake as running
'bitbake -e' is not possible as bitbake is locked.

Test cases for new functionality and small improvements are also included
into this patchset.

The following changes since commit 6b8bb442880ef18097e47761f4f3b6f555f2877c:

  genericx86*: Update BSPs to use 4.1 Kernel (2015-08-19 18:05:56 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/wic/wic-image-type
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/wic-image-type

Ed Bartosh (17):
  image.py: write bitbake variables to .env file
  oe-selftest: test generation of <image>.env
  wic: remove undescore from function name
  wic: add BitbakeVars class
  wic: create new method _parse_line
  wic: add default_image attribute to BitbakeVars
  wic: set default image
  wic: implement getting variables from .env files
  wic: implement --vars option
  wic: rename variable
  wic: deferred call of hlp.get_wic_plugins_help()
  image_types.bbclass: add wic image type
  wic-image-minimal: add wic image recipe and .wks
  oe-selftest: test building wic image by bitbake
  image.py: set bitbake variable ROOTFS_SIZE
  wic: use bitbake variable ROOTFS_SIZE
  image.py: add script output to the rootfs log

 meta/classes/image_types.bbclass                   |  12 ++
 meta/lib/oe/image.py                               |  32 +++++-
 meta/lib/oeqa/selftest/wic.py                      |  34 ++++++
 meta/recipes-extended/images/wic-image-minimal.bb  |  14 +++
 meta/recipes-extended/images/wic-image-minimal.wks |  10 ++
 scripts/lib/image/help.py                          |   6 +-
 scripts/lib/wic/imager/direct.py                   |  14 +++
 scripts/lib/wic/utils/oe/misc.py                   | 125 ++++++++++++++-------
 scripts/wic                                        |  14 ++-
 9 files changed, 213 insertions(+), 48 deletions(-)
 create mode 100644 meta/recipes-extended/images/wic-image-minimal.bb
 create mode 100644 meta/recipes-extended/images/wic-image-minimal.wks

--
Ed

[PATCH 01/17] image.py: write bitbake variables to .env file

[Ed Bartosh]

Write set of bitbake variables associated with the image into
build/tmp/sysroots/<machine>/imagedata/<image>.env

This is needed for wic to be able to get bitbake variables without
running 'bitbake -e'.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oe/image.py | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/meta/lib/oe/image.py b/meta/lib/oe/image.py
index 699c30f..a7fdefa 100644
--- a/meta/lib/oe/image.py
+++ b/meta/lib/oe/image.py
@@ -321,6 +321,27 @@ class Image(ImageDepGraph):
 
         return image_cmd_groups
 
+    def _write_env(self):
+        """
+        Write environment variables
+        to tmp/sysroots/<machine>/imgdata/<image>.env
+        """
+        stdir = self.d.getVar('STAGING_DIR_TARGET', True)
+        outdir = os.path.join(stdir, 'imgdata')
+        if not os.path.exists(outdir):
+            os.makedirs(outdir)
+        basename = self.d.getVar('IMAGE_BASENAME', True)
+        with open(os.path.join(outdir, basename) + '.env', 'w') as envf:
+            for var in sorted(self.d.keys()):
+                # filter out as much as we can to reduce file size
+                if var.startswith('_') or var.startswith('BB_') \
+                   or not var.isupper() or self.d.getVarFlag(var, "func") \
+                   or var in ('BBINCLUDED', 'SRCPV', 'MIRRORS'):
+                    continue
+                value = self.d.getVar(var, True)
+                if value:
+                    envf.write('%s="%s"\n' % (var, value.strip()))
+
     def create(self):
         bb.note("###### Generate images #######")
         pre_process_cmds = self.d.getVar("IMAGE_PREPROCESS_COMMAND", True)
@@ -332,6 +353,8 @@ class Image(ImageDepGraph):
 
         image_cmd_groups = self._get_imagecmds()
 
+        self._write_env()
+
         for image_cmds in image_cmd_groups:
             # create the images in parallel
             nproc = multiprocessing.cpu_count()
-- 
2.1.4

[PATCH 02/17] oe-selftest: test generation of <image>.env

[Ed Bartosh]

Added test case to check if <image>.env file is generated
and contains bitbake variables used in wic code.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 0c503ef..15de621 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -23,6 +23,7 @@
 
 """Test cases for wic."""
 
+import os
 import sys
 
 from glob import glob
@@ -160,3 +161,22 @@ class Wic(oeSelfTest):
         self.assertEqual(0, status)
         self.assertEqual(1, len(glob(self.resultdir + \
                                      "%(wks)s-*.direct" % vars)))
+
+    def test18_image_env(self):
+        """Test generation of <image>.env files."""
+        image = 'core-image-minimal'
+        stdir = get_bb_var('STAGING_DIR_TARGET', image)
+        imgdatadir = os.path.join(stdir, 'imgdata')
+        basename = get_bb_var('IMAGE_BASENAME', image)
+        self.assertEqual(basename, image)
+        path = os.path.join(imgdatadir, basename) + '.env'
+        self.assertTrue(os.path.isfile(path))
+        with open(path) as envfile:
+            content = dict(line.split("=", 1) for line in envfile)
+            # test if variables used by wic present in the .env file
+            for var in ('IMAGE_ROOTFS', 'DEPLOY_DIR_IMAGE', 'BBLAYERS',
+                        'STAGING_DIR_NATIVE', 'STAGING_DATADIR', 'TARGET_SYS',
+                        'IMAGE_BASENAME', 'INITRAMFS_FSTYPES', 'MACHINE_ARCH',
+                        'IMAGE_LINK_NAME', 'STAGING_LIBDIR', 'ROOTFS_SIZE'):
+                self.assertTrue(var in content)
+                self.assertTrue(content[var])
-- 
2.1.4

[PATCH 03/17] wic: remove undescore from function name

[Ed Bartosh]

Renamed __exec_cmd -> _exec_cmd as double underscores cause
strange behaviour when function is called in class method.
Python complains that __exec_cmd method(!!!) of the same class
doesn't exist.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index d399f2a..9c8f52d 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -46,13 +46,13 @@ NATIVE_RECIPES = {"mcopy": "mtools",
                   "syslinux": "syslinux"
                  }
 
-def __exec_cmd(cmd_and_args, as_shell=False, catch=3):
+def _exec_cmd(cmd_and_args, as_shell=False, catch=3):
     """
     Execute command, catching stderr, stdout
 
     Need to execute as_shell if the command uses wildcards
     """
-    msger.debug("__exec_cmd: %s" % cmd_and_args)
+    msger.debug("_exec_cmd: %s" % cmd_and_args)
     args = cmd_and_args.split()
     msger.debug(args)
 
@@ -61,7 +61,7 @@ def __exec_cmd(cmd_and_args, as_shell=False, catch=3):
     else:
         ret, out = runner.runtool(args, catch)
     out = out.strip()
-    msger.debug("__exec_cmd: output for %s (rc = %d): %s" % \
+    msger.debug("_exec_cmd: output for %s (rc = %d): %s" % \
                 (cmd_and_args, ret, out))
 
     return (ret, out)
@@ -73,7 +73,7 @@ def exec_cmd(cmd_and_args, as_shell=False, catch=3):
 
     Exits if rc non-zero
     """
-    ret, out = __exec_cmd(cmd_and_args, as_shell, catch)
+    ret, out = _exec_cmd(cmd_and_args, as_shell, catch)
 
     if ret != 0:
         msger.error("exec_cmd: %s returned '%s' instead of 0" % \
@@ -99,7 +99,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3):
     args = cmd_and_args.split()
     msger.debug(args)
 
-    ret, out = __exec_cmd(native_cmd_and_args, True, catch)
+    ret, out = _exec_cmd(native_cmd_and_args, True, catch)
 
     if ret == 127: # shell command-not-found
         prog = args[0]
@@ -139,7 +139,7 @@ def get_bitbake_var(var, image=None):
 
         log_level = msger.get_loglevel()
         msger.set_loglevel('normal')
-        ret, lines = __exec_cmd(cmd)
+        ret, lines = _exec_cmd(cmd)
         msger.set_loglevel(log_level)
 
         if ret:
-- 
2.1.4

[PATCH 04/17] wic: add BitbakeVars class

[Ed Bartosh]

Moved code of getting bitbake variables into separate class.

Created singleton object of this class in the module namespace.

Preserved existing API get_bitbake_var.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 91 +++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 38 deletions(-)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index 9c8f52d..3537a2e 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -124,48 +124,63 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3):
 
 BOOTDD_EXTRA_SPACE = 16384
 
-_BITBAKE_VARS = defaultdict(dict)
+class BitbakeVars(defaultdict):
+    """
+    Container for Bitbake variables.
+    """
+    def __init__(self):
+        defaultdict.__init__(self, dict)
+
+    def get_var(self, var, image=None):
+        """
+        Get bitbake variable value lazy way, i.e. run
+        'bitbake -e' only when variable is requested.
+        """
+        if image not in self:
+            # Get bitbake -e output
+            cmd = "bitbake -e"
+            if image:
+                cmd += " %s" % image
+
+            log_level = msger.get_loglevel()
+            msger.set_loglevel('normal')
+            ret, lines = _exec_cmd(cmd)
+            msger.set_loglevel(log_level)
+
+            if ret:
+                print "Couldn't get '%s' output." % cmd
+                print "Bitbake failed with error:\n%s\n" % lines
+                return
+
+            # Parse bitbake -e output
+            for line in lines.split('\n'):
+                if "=" not in line:
+                    continue
+                try:
+                    key, val = line.split("=")
+                except ValueError:
+                    continue
+                key = key.strip()
+                val = val.strip()
+                if key.replace('_', '').isalnum():
+                    self[image][key] = val.strip('"')
+
+            # Make first image a default set of variables
+            images = [key for key in self if key]
+            if len(images) == 1:
+                self[None] = self[image]
+
+        return self[image].get(var)
+
+# Create BB_VARS singleton
+BB_VARS = BitbakeVars()
 
 def get_bitbake_var(var, image=None):
     """
-    Get bitbake variable value lazy way, i.e. run
-    'bitbake -e' only when variable is requested.
+    Provide old get_bitbake_var API by wrapping
+    get_var method of BB_VARS singleton.
     """
-    if image not in _BITBAKE_VARS:
-        # Get bitbake -e output
-        cmd = "bitbake -e"
-        if image:
-            cmd += " %s" % image
-
-        log_level = msger.get_loglevel()
-        msger.set_loglevel('normal')
-        ret, lines = _exec_cmd(cmd)
-        msger.set_loglevel(log_level)
-
-        if ret:
-            print "Couldn't get '%s' output." % cmd
-            print "Bitbake failed with error:\n%s\n" % lines
-            return
-
-        # Parse bitbake -e output
-        for line in lines.split('\n'):
-            if "=" not in line:
-                continue
-            try:
-                key, val = line.split("=")
-            except ValueError:
-                continue
-            key = key.strip()
-            val = val.strip()
-            if key.replace('_', '').isalnum():
-                _BITBAKE_VARS[image][key] = val.strip('"')
-
-        # Make first image a default set of variables
-        images = [key for key in _BITBAKE_VARS if key]
-        if len(images) == 1:
-            _BITBAKE_VARS[None] = _BITBAKE_VARS[image]
-
-    return _BITBAKE_VARS[image].get(var)
+    return BB_VARS.get_var(var, image)
 
 def parse_sourceparams(sourceparams):
     """
-- 
2.1.4

[PATCH 05/17] wic: create new method _parse_line

[Ed Bartosh]

Moved code that parses one line of 'bitbake -e' output
to separate method _parse_line.

This method will be also used later to parse lines of .env files.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index 3537a2e..41e435f 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -131,6 +131,22 @@ class BitbakeVars(defaultdict):
     def __init__(self):
         defaultdict.__init__(self, dict)
 
+    def _parse_line(self, line, image):
+        """
+        Parse one line from bitbake -e output.
+        Put result key-value pair into the storage.
+        """
+        if "=" not in line:
+            return
+        try:
+            key, val = line.split("=")
+        except ValueError:
+            return
+        key = key.strip()
+        val = val.strip()
+        if key.replace('_', '').isalnum():
+            self[image][key] = val.strip('"')
+
     def get_var(self, var, image=None):
         """
         Get bitbake variable value lazy way, i.e. run
@@ -154,16 +170,7 @@ class BitbakeVars(defaultdict):
 
             # Parse bitbake -e output
             for line in lines.split('\n'):
-                if "=" not in line:
-                    continue
-                try:
-                    key, val = line.split("=")
-                except ValueError:
-                    continue
-                key = key.strip()
-                val = val.strip()
-                if key.replace('_', '').isalnum():
-                    self[image][key] = val.strip('"')
+                self._parse_line(line, image)
 
             # Make first image a default set of variables
             images = [key for key in self if key]
-- 
2.1.4

[PATCH 06/17] wic: add default_image attribute to BitbakeVars

[Ed Bartosh]

New attribute is used when bitbake variable is requested without
specifying image name. The attribute should be set from outside,
for example when wic is called with '-e <image>' option.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index 41e435f..040176d 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -131,6 +131,9 @@ class BitbakeVars(defaultdict):
     def __init__(self):
         defaultdict.__init__(self, dict)
 
+        # default_image attribute should be set from outside
+        self.default_image = None
+
     def _parse_line(self, line, image):
         """
         Parse one line from bitbake -e output.
@@ -152,6 +155,9 @@ class BitbakeVars(defaultdict):
         Get bitbake variable value lazy way, i.e. run
         'bitbake -e' only when variable is requested.
         """
+        if not image:
+            image = self.default_image
+
         if image not in self:
             # Get bitbake -e output
             cmd = "bitbake -e"
-- 
2.1.4

[PATCH 07/17] wic: set default image

[Ed Bartosh]

Set BitbakeVars.default_image when wic is called with -e option.
This makes get_bitbake_var API to use provided image as a default
source of variables.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/wic | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/scripts/wic b/scripts/wic
index c1d3003..87ada36 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -52,7 +52,7 @@ if bitbake_exe:
 else:
     bitbake_main = None
 
-from wic.utils.oe.misc import get_bitbake_var
+from wic.utils.oe.misc import get_bitbake_var, BB_VARS
 from wic.utils.errors import WicError
 from image import engine
 from image import help as hlp
@@ -141,7 +141,9 @@ def wic_create_subcommand(args, usage_str):
             print "  " + ", ".join(missed)
             sys.exit(1)
 
-    if not options.image_name:
+    if options.image_name:
+        BB_VARS.default_image = options.image_name
+    else:
         options.build_check = False
 
     if options.build_check:
-- 
2.1.4

[PATCH 08/17] wic: implement getting variables from .env files

[Ed Bartosh]

Added functionality of getting variables from <image>.env files to
BitbakeVars class. env files will be parsed if the directory with
env files is known, i.e. when vars_dir attribute is set.
Otherwise 'bitbake -e' output will be parsed.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 59 +++++++++++++++++++++++++---------------
 1 file changed, 37 insertions(+), 22 deletions(-)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index 040176d..0a8f3f1 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -26,6 +26,7 @@
 #
 """Miscellaneous functions."""
 
+import os
 from collections import defaultdict
 
 from wic import msger
@@ -131,12 +132,13 @@ class BitbakeVars(defaultdict):
     def __init__(self):
         defaultdict.__init__(self, dict)
 
-        # default_image attribute should be set from outside
+        # default_image and vars_dir attributes should be set from outside
         self.default_image = None
+        self.vars_dir = None
 
     def _parse_line(self, line, image):
         """
-        Parse one line from bitbake -e output.
+        Parse one line from bitbake -e output or from .env file.
         Put result key-value pair into the storage.
         """
         if "=" not in line:
@@ -152,31 +154,44 @@ class BitbakeVars(defaultdict):
 
     def get_var(self, var, image=None):
         """
-        Get bitbake variable value lazy way, i.e. run
-        'bitbake -e' only when variable is requested.
+        Get bitbake variable from 'bitbake -e' output or from .env file.
+        This is a lazy method, i.e. it runs bitbake or parses file only when
+        only when variable is requested. It also caches results.
         """
         if not image:
             image = self.default_image
 
         if image not in self:
-            # Get bitbake -e output
-            cmd = "bitbake -e"
-            if image:
-                cmd += " %s" % image
-
-            log_level = msger.get_loglevel()
-            msger.set_loglevel('normal')
-            ret, lines = _exec_cmd(cmd)
-            msger.set_loglevel(log_level)
-
-            if ret:
-                print "Couldn't get '%s' output." % cmd
-                print "Bitbake failed with error:\n%s\n" % lines
-                return
-
-            # Parse bitbake -e output
-            for line in lines.split('\n'):
-                self._parse_line(line, image)
+            if image and self.vars_dir:
+                fname = os.path.join(self.vars_dir, image + '.env')
+                if os.path.isfile(fname):
+                    # parse .env file
+                    with open(fname) as varsfile:
+                        for line in varsfile:
+                            self._parse_line(line, image)
+                else:
+                    print "Couldn't get bitbake variable from %s." % fname
+                    print "File %s doesn't exist." % fname
+                    return
+            else:
+                # Get bitbake -e output
+                cmd = "bitbake -e"
+                if image:
+                    cmd += " %s" % image
+
+                log_level = msger.get_loglevel()
+                msger.set_loglevel('normal')
+                ret, lines = _exec_cmd(cmd)
+                msger.set_loglevel(log_level)
+
+                if ret:
+                    print "Couldn't get '%s' output." % cmd
+                    print "Bitbake failed with error:\n%s\n" % lines
+                    return
+
+                # Parse bitbake -e output
+                for line in lines.split('\n'):
+                    self._parse_line(line, image)
 
             # Make first image a default set of variables
             images = [key for key in self if key]
-- 
2.1.4

[PATCH 09/17] wic: implement --vars option

[Ed Bartosh]

This option is used to point wic to the directory with .env
files containing list of bitbake variables and their values.

If this option is used wic will get bitbake variables from
files instead of parsing 'bitbake -e' output.

The main reason for this is to support new mode, when bitbake
runs wic to produce wic images. In this case wic can't run bitbake
again as it's locked, so it will get variables from .env files.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/wic | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/wic b/scripts/wic
index 87ada36..815b84e 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -114,6 +114,9 @@ def wic_create_subcommand(args, usage_str):
     parser.add_option("-c", "--compress-with", choices=("gzip", "bzip2", "xz"),
                       dest='compressor',
                       help="compress image with specified compressor")
+    parser.add_option("-v", "--vars", dest='vars_dir',
+                      help="directory with <image>.env files that store "
+                           "bitbake variables")
     parser.add_option("-D", "--debug", dest="debug", action="store_true",
                       default=False, help="output debug information")
 
@@ -146,6 +149,9 @@ def wic_create_subcommand(args, usage_str):
     else:
         options.build_check = False
 
+    if options.vars_dir:
+        BB_VARS.vars_dir = options.vars_dir
+
     if options.build_check:
         print "Checking basic build environment..."
         if not engine.verify_build_env():
-- 
2.1.4

[PATCH 10/17] wic: rename variable

[Ed Bartosh]

Renamed variable help -> hlp as 'help' is a name of Python
built-in function.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/image/help.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/lib/image/help.py b/scripts/lib/image/help.py
index 5fa5836..dc6ff36 100644
--- a/scripts/lib/image/help.py
+++ b/scripts/lib/image/help.py
@@ -41,9 +41,9 @@ def display_help(subcommand, subcommands):
     if subcommand not in subcommands:
         return False
 
-    help = subcommands.get(subcommand, subcommand_error)[2]
+    hlp = subcommands.get(subcommand, subcommand_error)[2]
     pager = subprocess.Popen('less', stdin=subprocess.PIPE)
-    pager.communicate(help)
+    pager.communicate(hlp)
 
     return True
 
-- 
2.1.4

[PATCH 11/17] wic: deferred call of hlp.get_wic_plugins_help()

[Ed Bartosh]

If get_wic_plugins_help is called from wic main module
it calls git_bitbake_var at some point. This fails when
wic is called from bitbake as 'bitbake -e' can't be
run.

Moved call of this method to help.py in order to call it
later, when BitbakeVariables singleton is properly initialized
to get variables from .env files.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/image/help.py | 2 ++
 scripts/wic               | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/image/help.py b/scripts/lib/image/help.py
index dc6ff36..717d847 100644
--- a/scripts/lib/image/help.py
+++ b/scripts/lib/image/help.py
@@ -42,6 +42,8 @@ def display_help(subcommand, subcommands):
         return False
 
     hlp = subcommands.get(subcommand, subcommand_error)[2]
+    if callable(hlp):
+        hlp = hlp()
     pager = subprocess.Popen('less', stdin=subprocess.PIPE)
     pager.communicate(hlp)
 
diff --git a/scripts/wic b/scripts/wic
index 815b84e..25b0d67 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -288,7 +288,7 @@ subcommands = {
                   hlp.wic_list_help],
     "plugins":   [wic_help_topic_subcommand,
                   wic_help_topic_usage,
-                  hlp.get_wic_plugins_help()],
+                  hlp.get_wic_plugins_help],
     "overview":  [wic_help_topic_subcommand,
                   wic_help_topic_usage,
                   hlp.wic_overview_help],
-- 
2.1.4

[PATCH 12/17] image_types.bbclass: add wic image type

[Ed Bartosh]

wic image type is used to produce partitioned images.

Image configuration should be stored in either <recipe>.<machine>.wks
or <recipe>.wks file.
.wks file should be put to the same location as image recipe
and have the same name.

[YOCTO #7672]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image_types.bbclass | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 8547574..cdd66ff 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -154,6 +154,16 @@ IMAGE_CMD_ubi = "multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" "${UBI_VOLNAM
 
 IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubifs ${MKUBIFS_ARGS}"
 
+IMAGE_CMD_wic () {
+	out=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}
+	wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.${MACHINE}.wks
+	[ -e $wks ] || wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.wks
+	[ -e $wks ] || bbfatal "Kiskstart file $wks doesn't exist"
+	BUILDDIR=${TOPDIR} wic create $wks --vars ${STAGING_DIR_TARGET}/imgdata/ -e ${IMAGE_BASENAME} -o $out/
+	mv $out/build/${IMAGE_BASENAME}*.direct $out.rootfs.wic
+	rm -rf $out/
+}
+
 EXTRA_IMAGECMD = ""
 
 inherit siteinfo
@@ -182,6 +192,7 @@ IMAGE_DEPENDS_elf = "virtual/kernel mkelfimage-native"
 IMAGE_DEPENDS_ubi = "mtd-utils-native"
 IMAGE_DEPENDS_ubifs = "mtd-utils-native"
 IMAGE_DEPENDS_multiubi = "mtd-utils-native"
+IMAGE_DEPENDS_wic = "parted-native"
 
 # This variable is available to request which values are suitable for IMAGE_FSTYPES
 IMAGE_TYPES = " \
@@ -201,6 +212,7 @@ IMAGE_TYPES = " \
     vdi \
     qcow2 \
     elf \
+    wic wic.gz wic.bz2 wic.lzma \
 "
 
 COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum"
-- 
2.1.4

[PATCH 13/17] wic-image-minimal: add wic image recipe and .wks

[Ed Bartosh]

Added example of recipe and .wks file to create partitioned image.

This image is using quite complex partitioning scheme.
It uses its own rootfs to populate two partitions in two different ways.
It also uses core-image-minimal rootfs to populate another partition.

This is how wic reports about artifacts used to create this image:
  ROOTFS_DIR: tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/rootfs
  ROOTFS_DIR["/core"]: tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs
  ROOTFS_DIR["/backup"]: tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/rootfs
  BOOTIMG_DIR: tmp/sysroots/qemux86-64/usr/share
  KERNEL_DIR: tmp/deploy/images/qemux86-64
  NATIVE_SYSROOT: tmp/sysroots/x86_64-linux

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/recipes-extended/images/wic-image-minimal.bb  | 14 ++++++++++++++
 meta/recipes-extended/images/wic-image-minimal.wks | 10 ++++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 meta/recipes-extended/images/wic-image-minimal.bb
 create mode 100644 meta/recipes-extended/images/wic-image-minimal.wks

diff --git a/meta/recipes-extended/images/wic-image-minimal.bb b/meta/recipes-extended/images/wic-image-minimal.bb
new file mode 100644
index 0000000..073c569
--- /dev/null
+++ b/meta/recipes-extended/images/wic-image-minimal.bb
@@ -0,0 +1,14 @@
+SUMMARY = "An example of partitioned image."
+
+IMAGE_INSTALL = "packagegroup-core-boot ${ROOTFS_PKGMANAGE_BOOTSTRAP}"
+
+IMAGE_FSTYPES = "wic.bz2"
+RM_OLD_IMAGE = "1"
+
+# core-image-minimal is referenced in .wks, so we need its rootfs
+# to be ready before our rootfs
+do_rootfs[depends] += "core-image-minimal:do_rootfs"
+
+IMAGE_ROOTFS_EXTRA_SPACE = "2000"
+
+inherit image
diff --git a/meta/recipes-extended/images/wic-image-minimal.wks b/meta/recipes-extended/images/wic-image-minimal.wks
new file mode 100644
index 0000000..29cd8f2
--- /dev/null
+++ b/meta/recipes-extended/images/wic-image-minimal.wks
@@ -0,0 +1,10 @@
+# short-description: Example of partitioned image with complex layout
+# long-description: This image contains boot partition and 3 rootfs partitions
+# created from core-image-minimal and wic-image-minimal image recipes.
+
+part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
+part / --source rootfs --ondisk sda --fstype=ext2 --label platform --align 1024
+part /core --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext2 --label core --align 1024
+part /backup --source rootfs --rootfs-dir=wic-image-minimal --ondisk sda --fstype=ext2 --label backup --align 1024
+
+bootloader  --timeout=0  --append="rootwait console=tty0"
-- 
2.1.4

[PATCH 14/17] oe-selftest: test building wic image by bitbake

[Ed Bartosh]

Added test case to verify building of wic-image-minimal recipe
and produced artifacts: manifest and bzipped partitioned image.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 15de621..dc0745a 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -180,3 +180,17 @@ class Wic(oeSelfTest):
                         'IMAGE_LINK_NAME', 'STAGING_LIBDIR', 'ROOTFS_SIZE'):
                 self.assertTrue(var in content)
                 self.assertTrue(content[var])
+
+    def test19_wic_image_type(self):
+        """Test building wic images by bitbake"""
+        self.assertEqual(0, bitbake('wic-image-minimal').status)
+
+        deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE')
+        machine = get_bb_var('MACHINE')
+        prefix = os.path.join(deploy_dir, 'wic-image-minimal-%s.' % machine)
+        # check if we have result image and manifests synmlinks
+        # pointing to existing files
+        for suffix in ('wic.bz2', 'manifest'):
+            path = prefix + suffix
+            self.assertTrue(os.path.islink(path))
+            self.assertTrue(os.path.isfile(os.path.realpath(path)))
-- 
2.1.4

[PATCH 15/17] image.py: set bitbake variable ROOTFS_SIZE

[Ed Bartosh]

This variable is going to be used by wic to set partition
size. Setting it in image.py makes it possible for wic to
use it without calculating it again.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oe/image.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/image.py b/meta/lib/oe/image.py
index a7fdefa..7f82748 100644
--- a/meta/lib/oe/image.py
+++ b/meta/lib/oe/image.py
@@ -262,14 +262,16 @@ class Image(ImageDepGraph):
     def _write_script(self, type, cmds):
         tempdir = self.d.getVar('T', True)
         script_name = os.path.join(tempdir, "create_image." + type)
+        rootfs_size = self._get_rootfs_size()
 
         self.d.setVar('img_creation_func', '\n'.join(cmds))
         self.d.setVarFlag('img_creation_func', 'func', 1)
         self.d.setVarFlag('img_creation_func', 'fakeroot', 1)
+        self.d.setVar('ROOTFS_SIZE', str(rootfs_size))
 
         with open(script_name, "w+") as script:
             script.write("%s" % bb.build.shell_trap_code())
-            script.write("export ROOTFS_SIZE=%d\n" % self._get_rootfs_size())
+            script.write("export ROOTFS_SIZE=%d\n" % rootfs_size)
             bb.data.emit_func('img_creation_func', script, self.d)
             script.write("img_creation_func\n")
 
-- 
2.1.4

[PATCH 16/17] wic: use bitbake variable ROOTFS_SIZE

[Ed Bartosh]

If bitbake image is referenced in .ks file and --size is not used
there wic uses ROOTFS_SIZE variable to set minimum partition size.

ROOTFS_SIZE is calculated in meta/lib/oe/image.py when rootfs is
created. The calculation is done using other image parameters:
IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT, IMAGE_OVERHEAD_FACTOR
and IMAGE_ROOTFS_EXTRA_SPACE.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/imager/direct.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/scripts/lib/wic/imager/direct.py b/scripts/lib/wic/imager/direct.py
index 57b1335..d68fd2a 100644
--- a/scripts/lib/wic/imager/direct.py
+++ b/scripts/lib/wic/imager/direct.py
@@ -29,6 +29,7 @@ import shutil
 
 from wic import kickstart, msger
 from wic.utils import fs_related
+from wic.utils.oe.misc import get_bitbake_var
 from wic.utils.partitionedfs import Image
 from wic.utils.errors import CreatorError, ImageError
 from wic.imager.baseimager import BaseImageCreator
@@ -229,6 +230,19 @@ class DirectImageCreator(BaseImageCreator):
         fstab_path = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
 
         for p in parts:
+            # get rootfs size from bitbake variable if it's not set in .ks file
+            if not p.size:
+                # and if rootfs name is specified for the partition
+                image_name = p.get_rootfs()
+                if image_name:
+                    # Bitbake variable ROOTFS_SIZE is calculated in
+                    # Image._get_rootfs_size method from meta/lib/oe/image.py
+                    # using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT,
+                    # IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE
+                    rsize_bb = get_bitbake_var('ROOTFS_SIZE', image_name)
+                    if rsize_bb:
+                        # convert from Kb to Mb
+                        p.size = int(rsize_bb) / 1024
             # need to create the filesystems in order to get their
             # sizes before we can add them and do the layout.
             # Image.create() actually calls __format_disks() to create
-- 
2.1.4

[PATCH 17/17] image.py: add script output to the rootfs log

[Ed Bartosh]

Let's add output of image creation script to the bitbake log
as it can contain useful information.

One good example of such an information is wic report about
artifacts and .wks file used for image creation.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oe/image.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/image.py b/meta/lib/oe/image.py
index 7f82748..6f57df2 100644
--- a/meta/lib/oe/image.py
+++ b/meta/lib/oe/image.py
@@ -11,11 +11,14 @@ def generate_image(arg):
             (type, create_img_cmd))
 
     try:
-        subprocess.check_output(create_img_cmd, stderr=subprocess.STDOUT)
+        output = subprocess.check_output(create_img_cmd,
+                                         stderr=subprocess.STDOUT)
     except subprocess.CalledProcessError as e:
         return("Error: The image creation script '%s' returned %d:\n%s" %
                (e.cmd, e.returncode, e.output))
 
+    bb.note("Script output:\n%s" % output)
+
     return None
 
 
-- 
2.1.4

Re: [PATCH 14/17] oe-selftest: test building wic image by bitbake

[Mario Domenech Goulart]

Hi Ed,

On Thu, 20 Aug 2015 14:56:27 +0300 Ed Bartosh <ed.bartosh@linux.intel.com> wrote:

> Added test case to verify building of wic-image-minimal recipe
> and produced artifacts: manifest and bzipped partitioned image.
>
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  meta/lib/oeqa/selftest/wic.py | 14 ++++++++++++++
>  1 file changed, 14 insertions(+)
>
> diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
> index 15de621..dc0745a 100644
> --- a/meta/lib/oeqa/selftest/wic.py
> +++ b/meta/lib/oeqa/selftest/wic.py
> @@ -180,3 +180,17 @@ class Wic(oeSelfTest):
>                          'IMAGE_LINK_NAME', 'STAGING_LIBDIR', 'ROOTFS_SIZE'):
>                  self.assertTrue(var in content)
>                  self.assertTrue(content[var])
> +
> +    def test19_wic_image_type(self):
> +        """Test building wic images by bitbake"""
> +        self.assertEqual(0, bitbake('wic-image-minimal').status)
> +
> +        deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE')
> +        machine = get_bb_var('MACHINE')
> +        prefix = os.path.join(deploy_dir, 'wic-image-minimal-%s.' % machine)
> +        # check if we have result image and manifests synmlinks

Small typo: s/synmlinks/symlinks/.

> +        # pointing to existing files
> +        for suffix in ('wic.bz2', 'manifest'):
> +            path = prefix + suffix
> +            self.assertTrue(os.path.islink(path))
> +            self.assertTrue(os.path.isfile(os.path.realpath(path)))
> -- 
> 2.1.4

Best wishes.
Mario
-- 
http://www.ossystems.com.br

[PATCH 14/17] oe-selftest: test building wic image by bitbake

[Ed Bartosh]

Added test case to verify building of wic-image-minimal recipe
and produced artifacts: manifest and bzipped partitioned image.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 639234b..b7bbc7b 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -187,3 +187,17 @@ class Wic(oeSelfTest):
                         'IMAGE_LINK_NAME', 'STAGING_LIBDIR', 'ROOTFS_SIZE'):
                 self.assertTrue(var in content)
                 self.assertTrue(content[var])
+
+    def test20_wic_image_type(self):
+        """Test building wic images by bitbake"""
+        self.assertEqual(0, bitbake('wic-image-minimal').status)
+
+        deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE')
+        machine = get_bb_var('MACHINE')
+        prefix = os.path.join(deploy_dir, 'wic-image-minimal-%s.' % machine)
+        # check if we have result image and manifests symlinks
+        # pointing to existing files
+        for suffix in ('wic.bz2', 'manifest'):
+            path = prefix + suffix
+            self.assertTrue(os.path.islink(path))
+            self.assertTrue(os.path.isfile(os.path.realpath(path)))
-- 
2.1.4

Re: [PATCH 01/17] image.py: write bitbake variables to .env file

[Richard Purdie]

On Thu, 2015-08-20 at 14:56 +0300, Ed Bartosh wrote:
> Write set of bitbake variables associated with the image into
> build/tmp/sysroots/<machine>/imagedata/<image>.env
> 
> This is needed for wic to be able to get bitbake variables without
> running 'bitbake -e'.

I've just realised what this code is doing, its writing out nearly the
whole data store :(

Is there no way we can know which variables wic may later need? I'm not
keen at all to see an iteration of d.keys() and the list of things not
to write out looks arbitrary at best :(

We need to find a better way of doing this...

I'd also prefer we only do this if we know we're going to be writing a
wic image.

Cheers,

Richard


> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  meta/lib/oe/image.py | 23 +++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
> 
> diff --git a/meta/lib/oe/image.py b/meta/lib/oe/image.py
> index 699c30f..a7fdefa 100644
> --- a/meta/lib/oe/image.py
> +++ b/meta/lib/oe/image.py
> @@ -321,6 +321,27 @@ class Image(ImageDepGraph):
>  
>          return image_cmd_groups
>  
> +    def _write_env(self):
> +        """
> +        Write environment variables
> +        to tmp/sysroots/<machine>/imgdata/<image>.env
> +        """
> +        stdir = self.d.getVar('STAGING_DIR_TARGET', True)
> +        outdir = os.path.join(stdir, 'imgdata')
> +        if not os.path.exists(outdir):
> +            os.makedirs(outdir)
> +        basename = self.d.getVar('IMAGE_BASENAME', True)
> +        with open(os.path.join(outdir, basename) + '.env', 'w') as envf:
> +            for var in sorted(self.d.keys()):
> +                # filter out as much as we can to reduce file size
> +                if var.startswith('_') or var.startswith('BB_') \
> +                   or not var.isupper() or self.d.getVarFlag(var, "func") \
> +                   or var in ('BBINCLUDED', 'SRCPV', 'MIRRORS'):
> +                    continue
> +                value = self.d.getVar(var, True)
> +                if value:
> +                    envf.write('%s="%s"\n' % (var, value.strip()))
> +
>      def create(self):
>          bb.note("###### Generate images #######")
>          pre_process_cmds = self.d.getVar("IMAGE_PREPROCESS_COMMAND", True)
> @@ -332,6 +353,8 @@ class Image(ImageDepGraph):
>  
>          image_cmd_groups = self._get_imagecmds()
>  
> +        self._write_env()
> +
>          for image_cmds in image_cmd_groups:
>              # create the images in parallel
>              nproc = multiprocessing.cpu_count()
> -- 
> 2.1.4
> 

Re: [PATCH 01/17] image.py: write bitbake variables to .env file

[Ed Bartosh]

On Sun, Aug 30, 2015 at 12:24:48PM +0100, Richard Purdie wrote:
> On Thu, 2015-08-20 at 14:56 +0300, Ed Bartosh wrote:
> > Write set of bitbake variables associated with the image into
> > build/tmp/sysroots/<machine>/imagedata/<image>.env
> > 
> > This is needed for wic to be able to get bitbake variables without
> > running 'bitbake -e'.
> 
> I've just realised what this code is doing, its writing out nearly the
> whole data store :(
I'd not say so. The code filters out a lot of content and that filtering can
be improved if needed.

Here is a real example of the file sizes:
$ ls -lh tmp/sysroots/nuc/imgdata/
total 124K
-rw-r--r-- 1 ed users 41K Aug 27 11:34 core-image-minimal.env
-rw-r--r-- 1 ed users 40K Aug 27 11:29 core-image-minimal-initramfs.env
-rw-r--r-- 1 ed users 39K Aug 27 11:35 wic-image-minimal.env

Are they still too big from your point of view?

> Is there no way we can know which variables wic may later need?
Theoretically there is no way to know it. It depends on what wic
plugin writers would want to use.

However, I can make .env files contain only variables that wic
already uses. This would require explicit addition of every new variable
to the code when wic starts to use it, but it's not a big deal as
most probably it will not happen to often.

Frankly, I thought that 40K in size .env file per image is not a big
deal either, but looks like it is.

> I'm not
> keen at all to see an iteration of d.keys() and the list of things not
> to write out looks arbitrary at best :(
It's not arbitrary. It's a result of filtering out variables which wic
will most probably not require and variables with the biggest
content size.

> We need to find a better way of doing this...
See my proposal above.

> I'd also prefer we only do this if we know we're going to be writing a
> wic image.
Any suggestion how to do this?
The problem is that any previously built image can be potentially
used by wic if it's mentioned in .ks file. We can find this out when
we build wic image, but we can't generate .env files on this step.


Regards,
Ed
> > Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> > ---
> >  meta/lib/oe/image.py | 23 +++++++++++++++++++++++
> >  1 file changed, 23 insertions(+)
> > 
> > diff --git a/meta/lib/oe/image.py b/meta/lib/oe/image.py
> > index 699c30f..a7fdefa 100644
> > --- a/meta/lib/oe/image.py
> > +++ b/meta/lib/oe/image.py
> > @@ -321,6 +321,27 @@ class Image(ImageDepGraph):
> >  
> >          return image_cmd_groups
> >  
> > +    def _write_env(self):
> > +        """
> > +        Write environment variables
> > +        to tmp/sysroots/<machine>/imgdata/<image>.env
> > +        """
> > +        stdir = self.d.getVar('STAGING_DIR_TARGET', True)
> > +        outdir = os.path.join(stdir, 'imgdata')
> > +        if not os.path.exists(outdir):
> > +            os.makedirs(outdir)
> > +        basename = self.d.getVar('IMAGE_BASENAME', True)
> > +        with open(os.path.join(outdir, basename) + '.env', 'w') as envf:
> > +            for var in sorted(self.d.keys()):
> > +                # filter out as much as we can to reduce file size
> > +                if var.startswith('_') or var.startswith('BB_') \
> > +                   or not var.isupper() or self.d.getVarFlag(var, "func") \
> > +                   or var in ('BBINCLUDED', 'SRCPV', 'MIRRORS'):
> > +                    continue
> > +                value = self.d.getVar(var, True)
> > +                if value:
> > +                    envf.write('%s="%s"\n' % (var, value.strip()))
> > +
> >      def create(self):
> >          bb.note("###### Generate images #######")
> >          pre_process_cmds = self.d.getVar("IMAGE_PREPROCESS_COMMAND", True)
> > @@ -332,6 +353,8 @@ class Image(ImageDepGraph):
> >  
> >          image_cmd_groups = self._get_imagecmds()
> >  
> > +        self._write_env()
> > +
> >          for image_cmds in image_cmd_groups:
> >              # create the images in parallel
> >              nproc = multiprocessing.cpu_count()
> > -- 
> > 2.1.4
> > 
> 
> 

-- 
--
Regards,
Ed