[PATCH 00/12] V3: Build wic images with bitbake

[PATCH 00/12] V3: Build wic images with bitbake

[Ed Bartosh]

Hi Reviewers,

This patchset adds new image type 'wic' to the list of image types
supported by bitbake. This should allow to build partitioned image
the same way as any other images.

New image type can be used in image recipes almost the same way as other
image types. There is just one difference - <image>[.<machine>].wks file
should be put to the same location as image recipe.

To enable this functionality I've implemented generating .env files with
bitbake variables and reading them by wic. This seems to be the only way
to get bitbake variables in wic when it's run by bitbake as running
'bitbake -e' is not possible as bitbake is locked.

Test cases for new functionality and small improvements are also included
into this patchset.

Changes in V2:
- Fixed typo in the comment. Thanks Mario!
- Rebased on fresh master

Changes in V3:
- introduced WICVARS variable to define a list of variables used by wic.
  only variables mentioned in WICVARS will be written to <image>.env
- rebased on top of fresh master

The following changes since commit 6b17c3831897ee1d46a763d807c5bd863d426bc1:

  linux-yocto-3.14: Update to latest revisions (2015-08-30 12:47:51 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/wic/wic-image-type
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/wic-image-type

Ed Bartosh (12):
  image.py: write bitbake variables to .env file
  oe-selftest: test generation of <image>.env
  wic: add BitbakeVars class
  wic: create new method _parse_line
  wic: add default_image attribute to BitbakeVars
  wic: set default image
  wic: implement getting variables from .env files
  wic: implement --vars option
  image_types.bbclass: add wic image type
  wic-image-minimal: add wic image recipe and .wks
  oe-selftest: test building wic image by bitbake
  wic: use bitbake variable ROOTFS_SIZE

 meta/classes/image_types.bbclass                   |  16 +++
 meta/lib/oe/image.py                               |  18 ++++
 meta/lib/oeqa/selftest/wic.py                      |  34 ++++++
 meta/recipes-extended/images/wic-image-minimal.bb  |  14 +++
 meta/recipes-extended/images/wic-image-minimal.wks |  10 ++
 scripts/lib/wic/imager/direct.py                   |  14 +++
 scripts/lib/wic/utils/oe/misc.py                   | 115 ++++++++++++++-------
 scripts/wic                                        |  12 ++-
 8 files changed, 195 insertions(+), 38 deletions(-)
 create mode 100644 meta/recipes-extended/images/wic-image-minimal.bb
 create mode 100644 meta/recipes-extended/images/wic-image-minimal.wks

--
Ed

[PATCH 01/12] image.py: write bitbake variables to .env file

[Ed Bartosh]

Write set of bitbake variables used by wic into
build/tmp/sysroots/<machine>/imagedata/<image>.env

List of variables is defined in WICVARS variable in
meta/classes/image_types.bbclass.

This is needed for wic to be able to get bitbake variables without
running 'bitbake -e'.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image_types.bbclass |  4 ++++
 meta/lib/oe/image.py             | 18 ++++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 2fd4c37..acee227 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -236,3 +236,7 @@ IMAGE_EXTENSION_live = "hddimg iso"
 # The IMAGE_TYPES_MASKED variable will be used to mask out from the IMAGE_FSTYPES,
 # images that will not be built at do_rootfs time: vmdk, vdi, qcow2, hddimg, iso, etc.
 IMAGE_TYPES_MASKED ?= ""
+
+# The WICVARS variable is used to define list of bitbake variables used in wic code
+# variables from this list is written to <image>.env file
+WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"
diff --git a/meta/lib/oe/image.py b/meta/lib/oe/image.py
index a2f94a1..95c62dc 100644
--- a/meta/lib/oe/image.py
+++ b/meta/lib/oe/image.py
@@ -326,6 +326,22 @@ class Image(ImageDepGraph):
 
         return image_cmd_groups
 
+    def _write_env(self):
+        """
+        Write environment variables used by wic
+        to tmp/sysroots/<machine>/imgdata/<image>.env
+        """
+        stdir = self.d.getVar('STAGING_DIR_TARGET', True)
+        outdir = os.path.join(stdir, 'imgdata')
+        if not os.path.exists(outdir):
+            os.makedirs(outdir)
+        basename = self.d.getVar('IMAGE_BASENAME', True)
+        with open(os.path.join(outdir, basename) + '.env', 'w') as envf:
+            for var in self.d.getVar('WICVARS', True).split():
+                value = self.d.getVar(var, True)
+                if value:
+                    envf.write('%s="%s"\n' % (var, value.strip()))
+
     def create(self):
         bb.note("###### Generate images #######")
         pre_process_cmds = self.d.getVar("IMAGE_PREPROCESS_COMMAND", True)
@@ -337,6 +353,8 @@ class Image(ImageDepGraph):
 
         image_cmd_groups = self._get_imagecmds()
 
+        self._write_env()
+
         for image_cmds in image_cmd_groups:
             # create the images in parallel
             nproc = multiprocessing.cpu_count()
-- 
2.1.4

[PATCH 02/12] oe-selftest: test generation of <image>.env

[Ed Bartosh]

Added test case to check if <image>.env file is generated
and contains bitbake variables used in wic code.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 117bd9d..fe8a2d0 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -23,6 +23,7 @@
 
 """Test cases for wic."""
 
+import os
 import sys
 
 from glob import glob
@@ -168,3 +169,22 @@ class Wic(oeSelfTest):
         self.assertEqual(1, len(glob(self.resultdir + "HYBRID_ISO_IMG-*.direct")))
         self.assertEqual(1, len(glob(self.resultdir + "HYBRID_ISO_IMG-*.iso")))
 
+    def test19_image_env(self):
+        """Test generation of <image>.env files."""
+        image = 'core-image-minimal'
+        stdir = get_bb_var('STAGING_DIR_TARGET', image)
+        imgdatadir = os.path.join(stdir, 'imgdata')
+
+        basename = get_bb_var('IMAGE_BASENAME', image)
+        self.assertEqual(basename, image)
+        path = os.path.join(imgdatadir, basename) + '.env'
+        self.assertTrue(os.path.isfile(path))
+
+        wicvars = get_bb_var('WICVARS', image).split()
+        wicvars.remove('IMAGE_BOOT_FILES') # this variable is optional
+        with open(path) as envfile:
+            content = dict(line.split("=", 1) for line in envfile)
+            # test if variables used by wic present in the .env file
+            for var in wicvars:
+                self.assertTrue(var in content, "%s is not in .env file" % var)
+                self.assertTrue(content[var])
-- 
2.1.4

[PATCH 03/12] wic: add BitbakeVars class

[Ed Bartosh]

Moved code of getting bitbake variables into separate class.

Created singleton object of this class in the module namespace.

Preserved existing API get_bitbake_var.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 91 +++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 38 deletions(-)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index 9c8f52d..3537a2e 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -124,48 +124,63 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3):
 
 BOOTDD_EXTRA_SPACE = 16384
 
-_BITBAKE_VARS = defaultdict(dict)
+class BitbakeVars(defaultdict):
+    """
+    Container for Bitbake variables.
+    """
+    def __init__(self):
+        defaultdict.__init__(self, dict)
+
+    def get_var(self, var, image=None):
+        """
+        Get bitbake variable value lazy way, i.e. run
+        'bitbake -e' only when variable is requested.
+        """
+        if image not in self:
+            # Get bitbake -e output
+            cmd = "bitbake -e"
+            if image:
+                cmd += " %s" % image
+
+            log_level = msger.get_loglevel()
+            msger.set_loglevel('normal')
+            ret, lines = _exec_cmd(cmd)
+            msger.set_loglevel(log_level)
+
+            if ret:
+                print "Couldn't get '%s' output." % cmd
+                print "Bitbake failed with error:\n%s\n" % lines
+                return
+
+            # Parse bitbake -e output
+            for line in lines.split('\n'):
+                if "=" not in line:
+                    continue
+                try:
+                    key, val = line.split("=")
+                except ValueError:
+                    continue
+                key = key.strip()
+                val = val.strip()
+                if key.replace('_', '').isalnum():
+                    self[image][key] = val.strip('"')
+
+            # Make first image a default set of variables
+            images = [key for key in self if key]
+            if len(images) == 1:
+                self[None] = self[image]
+
+        return self[image].get(var)
+
+# Create BB_VARS singleton
+BB_VARS = BitbakeVars()
 
 def get_bitbake_var(var, image=None):
     """
-    Get bitbake variable value lazy way, i.e. run
-    'bitbake -e' only when variable is requested.
+    Provide old get_bitbake_var API by wrapping
+    get_var method of BB_VARS singleton.
     """
-    if image not in _BITBAKE_VARS:
-        # Get bitbake -e output
-        cmd = "bitbake -e"
-        if image:
-            cmd += " %s" % image
-
-        log_level = msger.get_loglevel()
-        msger.set_loglevel('normal')
-        ret, lines = _exec_cmd(cmd)
-        msger.set_loglevel(log_level)
-
-        if ret:
-            print "Couldn't get '%s' output." % cmd
-            print "Bitbake failed with error:\n%s\n" % lines
-            return
-
-        # Parse bitbake -e output
-        for line in lines.split('\n'):
-            if "=" not in line:
-                continue
-            try:
-                key, val = line.split("=")
-            except ValueError:
-                continue
-            key = key.strip()
-            val = val.strip()
-            if key.replace('_', '').isalnum():
-                _BITBAKE_VARS[image][key] = val.strip('"')
-
-        # Make first image a default set of variables
-        images = [key for key in _BITBAKE_VARS if key]
-        if len(images) == 1:
-            _BITBAKE_VARS[None] = _BITBAKE_VARS[image]
-
-    return _BITBAKE_VARS[image].get(var)
+    return BB_VARS.get_var(var, image)
 
 def parse_sourceparams(sourceparams):
     """
-- 
2.1.4

[PATCH 04/12] wic: create new method _parse_line

[Ed Bartosh]

Moved code that parses one line of 'bitbake -e' output
to separate method _parse_line.

This method will be also used later to parse lines of .env files.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index 3537a2e..41e435f 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -131,6 +131,22 @@ class BitbakeVars(defaultdict):
     def __init__(self):
         defaultdict.__init__(self, dict)
 
+    def _parse_line(self, line, image):
+        """
+        Parse one line from bitbake -e output.
+        Put result key-value pair into the storage.
+        """
+        if "=" not in line:
+            return
+        try:
+            key, val = line.split("=")
+        except ValueError:
+            return
+        key = key.strip()
+        val = val.strip()
+        if key.replace('_', '').isalnum():
+            self[image][key] = val.strip('"')
+
     def get_var(self, var, image=None):
         """
         Get bitbake variable value lazy way, i.e. run
@@ -154,16 +170,7 @@ class BitbakeVars(defaultdict):
 
             # Parse bitbake -e output
             for line in lines.split('\n'):
-                if "=" not in line:
-                    continue
-                try:
-                    key, val = line.split("=")
-                except ValueError:
-                    continue
-                key = key.strip()
-                val = val.strip()
-                if key.replace('_', '').isalnum():
-                    self[image][key] = val.strip('"')
+                self._parse_line(line, image)
 
             # Make first image a default set of variables
             images = [key for key in self if key]
-- 
2.1.4

[PATCH 05/12] wic: add default_image attribute to BitbakeVars

[Ed Bartosh]

New attribute is used when bitbake variable is requested without
specifying image name. The attribute should be set from outside,
for example when wic is called with '-e <image>' option.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index 41e435f..040176d 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -131,6 +131,9 @@ class BitbakeVars(defaultdict):
     def __init__(self):
         defaultdict.__init__(self, dict)
 
+        # default_image attribute should be set from outside
+        self.default_image = None
+
     def _parse_line(self, line, image):
         """
         Parse one line from bitbake -e output.
@@ -152,6 +155,9 @@ class BitbakeVars(defaultdict):
         Get bitbake variable value lazy way, i.e. run
         'bitbake -e' only when variable is requested.
         """
+        if not image:
+            image = self.default_image
+
         if image not in self:
             # Get bitbake -e output
             cmd = "bitbake -e"
-- 
2.1.4

[PATCH 06/12] wic: set default image

[Ed Bartosh]

Set BitbakeVars.default_image when wic is called with -e option.
This makes get_bitbake_var API to use provided image as a default
source of variables.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/wic | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/scripts/wic b/scripts/wic
index 5aa2393..5818e2d 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -52,7 +52,7 @@ if bitbake_exe:
 else:
     bitbake_main = None
 
-from wic.utils.oe.misc import get_bitbake_var
+from wic.utils.oe.misc import get_bitbake_var, BB_VARS
 from wic.utils.errors import WicError
 from image import engine
 from image import help as hlp
@@ -141,7 +141,9 @@ def wic_create_subcommand(args, usage_str):
             print "  " + ", ".join(missed)
             sys.exit(1)
 
-    if not options.image_name:
+    if options.image_name:
+        BB_VARS.default_image = options.image_name
+    else:
         options.build_check = False
 
     if options.build_check:
-- 
2.1.4

[PATCH 07/12] wic: implement getting variables from .env files

[Ed Bartosh]

Added functionality of getting variables from <image>.env files to
BitbakeVars class. env files will be parsed if the directory with
env files is known, i.e. when vars_dir attribute is set.
Otherwise 'bitbake -e' output will be parsed.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 59 +++++++++++++++++++++++++---------------
 1 file changed, 37 insertions(+), 22 deletions(-)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index 040176d..0a8f3f1 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -26,6 +26,7 @@
 #
 """Miscellaneous functions."""
 
+import os
 from collections import defaultdict
 
 from wic import msger
@@ -131,12 +132,13 @@ class BitbakeVars(defaultdict):
     def __init__(self):
         defaultdict.__init__(self, dict)
 
-        # default_image attribute should be set from outside
+        # default_image and vars_dir attributes should be set from outside
         self.default_image = None
+        self.vars_dir = None
 
     def _parse_line(self, line, image):
         """
-        Parse one line from bitbake -e output.
+        Parse one line from bitbake -e output or from .env file.
         Put result key-value pair into the storage.
         """
         if "=" not in line:
@@ -152,31 +154,44 @@ class BitbakeVars(defaultdict):
 
     def get_var(self, var, image=None):
         """
-        Get bitbake variable value lazy way, i.e. run
-        'bitbake -e' only when variable is requested.
+        Get bitbake variable from 'bitbake -e' output or from .env file.
+        This is a lazy method, i.e. it runs bitbake or parses file only when
+        only when variable is requested. It also caches results.
         """
         if not image:
             image = self.default_image
 
         if image not in self:
-            # Get bitbake -e output
-            cmd = "bitbake -e"
-            if image:
-                cmd += " %s" % image
-
-            log_level = msger.get_loglevel()
-            msger.set_loglevel('normal')
-            ret, lines = _exec_cmd(cmd)
-            msger.set_loglevel(log_level)
-
-            if ret:
-                print "Couldn't get '%s' output." % cmd
-                print "Bitbake failed with error:\n%s\n" % lines
-                return
-
-            # Parse bitbake -e output
-            for line in lines.split('\n'):
-                self._parse_line(line, image)
+            if image and self.vars_dir:
+                fname = os.path.join(self.vars_dir, image + '.env')
+                if os.path.isfile(fname):
+                    # parse .env file
+                    with open(fname) as varsfile:
+                        for line in varsfile:
+                            self._parse_line(line, image)
+                else:
+                    print "Couldn't get bitbake variable from %s." % fname
+                    print "File %s doesn't exist." % fname
+                    return
+            else:
+                # Get bitbake -e output
+                cmd = "bitbake -e"
+                if image:
+                    cmd += " %s" % image
+
+                log_level = msger.get_loglevel()
+                msger.set_loglevel('normal')
+                ret, lines = _exec_cmd(cmd)
+                msger.set_loglevel(log_level)
+
+                if ret:
+                    print "Couldn't get '%s' output." % cmd
+                    print "Bitbake failed with error:\n%s\n" % lines
+                    return
+
+                # Parse bitbake -e output
+                for line in lines.split('\n'):
+                    self._parse_line(line, image)
 
             # Make first image a default set of variables
             images = [key for key in self if key]
-- 
2.1.4

[PATCH 08/12] wic: implement --vars option

[Ed Bartosh]

This option is used to point wic to the directory with .env
files containing list of bitbake variables and their values.

If this option is used wic will get bitbake variables from
files instead of parsing 'bitbake -e' output.

The main reason for this is to support new mode, when bitbake
runs wic to produce wic images. In this case wic can't run bitbake
again as it's locked, so it will get variables from .env files.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/wic | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/scripts/wic b/scripts/wic
index 5818e2d..25b0d67 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -114,6 +114,9 @@ def wic_create_subcommand(args, usage_str):
     parser.add_option("-c", "--compress-with", choices=("gzip", "bzip2", "xz"),
                       dest='compressor',
                       help="compress image with specified compressor")
+    parser.add_option("-v", "--vars", dest='vars_dir',
+                      help="directory with <image>.env files that store "
+                           "bitbake variables")
     parser.add_option("-D", "--debug", dest="debug", action="store_true",
                       default=False, help="output debug information")
 
@@ -146,6 +149,9 @@ def wic_create_subcommand(args, usage_str):
     else:
         options.build_check = False
 
+    if options.vars_dir:
+        BB_VARS.vars_dir = options.vars_dir
+
     if options.build_check:
         print "Checking basic build environment..."
         if not engine.verify_build_env():
-- 
2.1.4

[PATCH 09/12] image_types.bbclass: add wic image type

[Ed Bartosh]

wic image type is used to produce partitioned images.

Image configuration should be stored in either <recipe>.<machine>.wks
or <recipe>.wks file.
.wks file should be put to the same location as image recipe
and have the same name.

[YOCTO #7672]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image_types.bbclass | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index acee227..ecb066b 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -162,6 +162,16 @@ IMAGE_CMD_ubi = "multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" "${UBI_VOLNAM
 
 IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubifs ${MKUBIFS_ARGS}"
 
+IMAGE_CMD_wic () {
+	out=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}
+	wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.${MACHINE}.wks
+	[ -e $wks ] || wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.wks
+	[ -e $wks ] || bbfatal "Kiskstart file $wks doesn't exist"
+	BUILDDIR=${TOPDIR} wic create $wks --vars ${STAGING_DIR_TARGET}/imgdata/ -e ${IMAGE_BASENAME} -o $out/
+	mv $out/build/${IMAGE_BASENAME}*.direct $out.rootfs.wic
+	rm -rf $out/
+}
+
 EXTRA_IMAGECMD = ""
 
 inherit siteinfo
@@ -190,6 +200,7 @@ IMAGE_DEPENDS_elf = "virtual/kernel mkelfimage-native"
 IMAGE_DEPENDS_ubi = "mtd-utils-native"
 IMAGE_DEPENDS_ubifs = "mtd-utils-native"
 IMAGE_DEPENDS_multiubi = "mtd-utils-native"
+IMAGE_DEPENDS_wic = "parted-native"
 
 # This variable is available to request which values are suitable for IMAGE_FSTYPES
 IMAGE_TYPES = " \
@@ -209,6 +220,7 @@ IMAGE_TYPES = " \
     vdi \
     qcow2 \
     elf \
+    wic wic.gz wic.bz2 wic.lzma \
 "
 
 COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum"
-- 
2.1.4

[PATCH 10/12] wic-image-minimal: add wic image recipe and .wks

[Ed Bartosh]

Added example of recipe and .wks file to create partitioned image.

This image is using quite complex partitioning scheme.
It uses its own rootfs to populate two partitions in two different ways.
It also uses core-image-minimal rootfs to populate another partition.

This is how wic reports about artifacts used to create this image:
  ROOTFS_DIR: tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/rootfs
  ROOTFS_DIR["/core"]: tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs
  ROOTFS_DIR["/backup"]: tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/rootfs
  BOOTIMG_DIR: tmp/sysroots/qemux86-64/usr/share
  KERNEL_DIR: tmp/deploy/images/qemux86-64
  NATIVE_SYSROOT: tmp/sysroots/x86_64-linux

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/recipes-extended/images/wic-image-minimal.bb  | 14 ++++++++++++++
 meta/recipes-extended/images/wic-image-minimal.wks | 10 ++++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 meta/recipes-extended/images/wic-image-minimal.bb
 create mode 100644 meta/recipes-extended/images/wic-image-minimal.wks

diff --git a/meta/recipes-extended/images/wic-image-minimal.bb b/meta/recipes-extended/images/wic-image-minimal.bb
new file mode 100644
index 0000000..073c569
--- /dev/null
+++ b/meta/recipes-extended/images/wic-image-minimal.bb
@@ -0,0 +1,14 @@
+SUMMARY = "An example of partitioned image."
+
+IMAGE_INSTALL = "packagegroup-core-boot ${ROOTFS_PKGMANAGE_BOOTSTRAP}"
+
+IMAGE_FSTYPES = "wic.bz2"
+RM_OLD_IMAGE = "1"
+
+# core-image-minimal is referenced in .wks, so we need its rootfs
+# to be ready before our rootfs
+do_rootfs[depends] += "core-image-minimal:do_rootfs"
+
+IMAGE_ROOTFS_EXTRA_SPACE = "2000"
+
+inherit image
diff --git a/meta/recipes-extended/images/wic-image-minimal.wks b/meta/recipes-extended/images/wic-image-minimal.wks
new file mode 100644
index 0000000..29cd8f2
--- /dev/null
+++ b/meta/recipes-extended/images/wic-image-minimal.wks
@@ -0,0 +1,10 @@
+# short-description: Example of partitioned image with complex layout
+# long-description: This image contains boot partition and 3 rootfs partitions
+# created from core-image-minimal and wic-image-minimal image recipes.
+
+part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
+part / --source rootfs --ondisk sda --fstype=ext2 --label platform --align 1024
+part /core --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext2 --label core --align 1024
+part /backup --source rootfs --rootfs-dir=wic-image-minimal --ondisk sda --fstype=ext2 --label backup --align 1024
+
+bootloader  --timeout=0  --append="rootwait console=tty0"
-- 
2.1.4

[PATCH 11/12] oe-selftest: test building wic image by bitbake

[Ed Bartosh]

Added test case to verify building of wic-image-minimal recipe
and produced artifacts: manifest and bzipped partitioned image.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index fe8a2d0..deb2333 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -188,3 +188,17 @@ class Wic(oeSelfTest):
             for var in wicvars:
                 self.assertTrue(var in content, "%s is not in .env file" % var)
                 self.assertTrue(content[var])
+
+    def test20_wic_image_type(self):
+        """Test building wic images by bitbake"""
+        self.assertEqual(0, bitbake('wic-image-minimal').status)
+
+        deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE')
+        machine = get_bb_var('MACHINE')
+        prefix = os.path.join(deploy_dir, 'wic-image-minimal-%s.' % machine)
+        # check if we have result image and manifests symlinks
+        # pointing to existing files
+        for suffix in ('wic.bz2', 'manifest'):
+            path = prefix + suffix
+            self.assertTrue(os.path.islink(path))
+            self.assertTrue(os.path.isfile(os.path.realpath(path)))
-- 
2.1.4

[PATCH 12/12] wic: use bitbake variable ROOTFS_SIZE

[Ed Bartosh]

If bitbake image is referenced in .ks file and --size is not used
there wic uses ROOTFS_SIZE variable to set minimum partition size.

ROOTFS_SIZE is calculated in meta/lib/oe/image.py when rootfs is
created. The calculation is done using other image parameters:
IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT, IMAGE_OVERHEAD_FACTOR
and IMAGE_ROOTFS_EXTRA_SPACE.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/imager/direct.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/scripts/lib/wic/imager/direct.py b/scripts/lib/wic/imager/direct.py
index 57b1335..d68fd2a 100644
--- a/scripts/lib/wic/imager/direct.py
+++ b/scripts/lib/wic/imager/direct.py
@@ -29,6 +29,7 @@ import shutil
 
 from wic import kickstart, msger
 from wic.utils import fs_related
+from wic.utils.oe.misc import get_bitbake_var
 from wic.utils.partitionedfs import Image
 from wic.utils.errors import CreatorError, ImageError
 from wic.imager.baseimager import BaseImageCreator
@@ -229,6 +230,19 @@ class DirectImageCreator(BaseImageCreator):
         fstab_path = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
 
         for p in parts:
+            # get rootfs size from bitbake variable if it's not set in .ks file
+            if not p.size:
+                # and if rootfs name is specified for the partition
+                image_name = p.get_rootfs()
+                if image_name:
+                    # Bitbake variable ROOTFS_SIZE is calculated in
+                    # Image._get_rootfs_size method from meta/lib/oe/image.py
+                    # using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT,
+                    # IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE
+                    rsize_bb = get_bitbake_var('ROOTFS_SIZE', image_name)
+                    if rsize_bb:
+                        # convert from Kb to Mb
+                        p.size = int(rsize_bb) / 1024
             # need to create the filesystems in order to get their
             # sizes before we can add them and do the layout.
             # Image.create() actually calls __format_disks() to create
-- 
2.1.4

Re: [PATCH 10/12] wic-image-minimal: add wic image recipe and .wks

[Richard Purdie]

On Sun, 2015-08-30 at 20:47 +0300, Ed Bartosh wrote:
> Added example of recipe and .wks file to create partitioned image.
> 
> This image is using quite complex partitioning scheme.
> It uses its own rootfs to populate two partitions in two different ways.
> It also uses core-image-minimal rootfs to populate another partition.
> 
> This is how wic reports about artifacts used to create this image:
>   ROOTFS_DIR: tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/rootfs
>   ROOTFS_DIR["/core"]: tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs
>   ROOTFS_DIR["/backup"]: tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/rootfs
>   BOOTIMG_DIR: tmp/sysroots/qemux86-64/usr/share
>   KERNEL_DIR: tmp/deploy/images/qemux86-64
>   NATIVE_SYSROOT: tmp/sysroots/x86_64-linux
> 
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>

Since we use this as the selftest, I think this may be better positioned
in meta-selftest?

Otherwise the patchset looks good thanks. I'll merge this up to this
patch.

Cheers,

Richard