[PATCH 0/5] V4: Build wic images with bitbake

[PATCH 0/5] V4: Build wic images with bitbake

[Ed Bartosh]

Hi,

This is hopefully last version of the patchset.
I've changed location of .wks and .bb files as Richard suggested.

The following changes since commit 0f84702f0811cba4b6e6e890d478f6e98937b831:

  image_types.bbclass: add wic image type (2015-08-30 21:38:23 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/wic/wic-image-type
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/wic-image-type

Ed Bartosh (5):
  image.py: write bitbake variables to .env file
  oe-selftest: test generation of <image>.env
  wic-image-minimal: add wic image recipe and .wks
  oe-selftest: test building wic image by bitbake
  wic: use bitbake variable ROOTFS_SIZE

 .../recipes-test/images/wic-image-minimal.bb       | 14 +++++++++
 .../recipes-test/images/wic-image-minimal.wks      | 10 +++++++
 meta/classes/image_types.bbclass                   |  4 +++
 meta/lib/oe/image.py                               | 18 ++++++++++++
 meta/lib/oeqa/selftest/wic.py                      | 34 ++++++++++++++++++++++
 scripts/lib/wic/imager/direct.py                   | 14 +++++++++
 6 files changed, 94 insertions(+)
 create mode 100644 meta-selftest/recipes-test/images/wic-image-minimal.bb
 create mode 100644 meta-selftest/recipes-test/images/wic-image-minimal.wks

--
Ed

[PATCH 1/5] image.py: write bitbake variables to .env file

[Ed Bartosh]

Write set of bitbake variables used by wic into
build/tmp/sysroots/<machine>/imagedata/<image>.env

List of variables is defined in WICVARS variable in
meta/classes/image_types.bbclass.

This is needed for wic to be able to get bitbake variables without
running 'bitbake -e'.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image_types.bbclass |  4 ++++
 meta/lib/oe/image.py             | 18 ++++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 05c45f8..ecb066b 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -248,3 +248,7 @@ IMAGE_EXTENSION_live = "hddimg iso"
 # The IMAGE_TYPES_MASKED variable will be used to mask out from the IMAGE_FSTYPES,
 # images that will not be built at do_rootfs time: vmdk, vdi, qcow2, hddimg, iso, etc.
 IMAGE_TYPES_MASKED ?= ""
+
+# The WICVARS variable is used to define list of bitbake variables used in wic code
+# variables from this list is written to <image>.env file
+WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"
diff --git a/meta/lib/oe/image.py b/meta/lib/oe/image.py
index a2f94a1..95c62dc 100644
--- a/meta/lib/oe/image.py
+++ b/meta/lib/oe/image.py
@@ -326,6 +326,22 @@ class Image(ImageDepGraph):
 
         return image_cmd_groups
 
+    def _write_env(self):
+        """
+        Write environment variables used by wic
+        to tmp/sysroots/<machine>/imgdata/<image>.env
+        """
+        stdir = self.d.getVar('STAGING_DIR_TARGET', True)
+        outdir = os.path.join(stdir, 'imgdata')
+        if not os.path.exists(outdir):
+            os.makedirs(outdir)
+        basename = self.d.getVar('IMAGE_BASENAME', True)
+        with open(os.path.join(outdir, basename) + '.env', 'w') as envf:
+            for var in self.d.getVar('WICVARS', True).split():
+                value = self.d.getVar(var, True)
+                if value:
+                    envf.write('%s="%s"\n' % (var, value.strip()))
+
     def create(self):
         bb.note("###### Generate images #######")
         pre_process_cmds = self.d.getVar("IMAGE_PREPROCESS_COMMAND", True)
@@ -337,6 +353,8 @@ class Image(ImageDepGraph):
 
         image_cmd_groups = self._get_imagecmds()
 
+        self._write_env()
+
         for image_cmds in image_cmd_groups:
             # create the images in parallel
             nproc = multiprocessing.cpu_count()
-- 
2.1.4

[PATCH 2/5] oe-selftest: test generation of <image>.env

[Ed Bartosh]

Added test case to check if <image>.env file is generated
and contains bitbake variables used in wic code.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 117bd9d..fe8a2d0 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -23,6 +23,7 @@
 
 """Test cases for wic."""
 
+import os
 import sys
 
 from glob import glob
@@ -168,3 +169,22 @@ class Wic(oeSelfTest):
         self.assertEqual(1, len(glob(self.resultdir + "HYBRID_ISO_IMG-*.direct")))
         self.assertEqual(1, len(glob(self.resultdir + "HYBRID_ISO_IMG-*.iso")))
 
+    def test19_image_env(self):
+        """Test generation of <image>.env files."""
+        image = 'core-image-minimal'
+        stdir = get_bb_var('STAGING_DIR_TARGET', image)
+        imgdatadir = os.path.join(stdir, 'imgdata')
+
+        basename = get_bb_var('IMAGE_BASENAME', image)
+        self.assertEqual(basename, image)
+        path = os.path.join(imgdatadir, basename) + '.env'
+        self.assertTrue(os.path.isfile(path))
+
+        wicvars = get_bb_var('WICVARS', image).split()
+        wicvars.remove('IMAGE_BOOT_FILES') # this variable is optional
+        with open(path) as envfile:
+            content = dict(line.split("=", 1) for line in envfile)
+            # test if variables used by wic present in the .env file
+            for var in wicvars:
+                self.assertTrue(var in content, "%s is not in .env file" % var)
+                self.assertTrue(content[var])
-- 
2.1.4

[PATCH 3/5] wic-image-minimal: add wic image recipe and .wks

[Ed Bartosh]

Added example of recipe and .wks file to create partitioned image.

This image is using quite complex partitioning scheme.
It uses its own rootfs to populate two partitions in two different ways.
It also uses core-image-minimal rootfs to populate another partition.

This is how wic reports about artifacts used to create this image:
  ROOTFS_DIR: tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/rootfs
  ROOTFS_DIR["/core"]: tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs
  ROOTFS_DIR["/backup"]: tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/rootfs
  BOOTIMG_DIR: tmp/sysroots/qemux86-64/usr/share
  KERNEL_DIR: tmp/deploy/images/qemux86-64
  NATIVE_SYSROOT: tmp/sysroots/x86_64-linux

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta-selftest/recipes-test/images/wic-image-minimal.bb  | 14 ++++++++++++++
 meta-selftest/recipes-test/images/wic-image-minimal.wks | 10 ++++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 meta-selftest/recipes-test/images/wic-image-minimal.bb
 create mode 100644 meta-selftest/recipes-test/images/wic-image-minimal.wks

diff --git a/meta-selftest/recipes-test/images/wic-image-minimal.bb b/meta-selftest/recipes-test/images/wic-image-minimal.bb
new file mode 100644
index 0000000..073c569
--- /dev/null
+++ b/meta-selftest/recipes-test/images/wic-image-minimal.bb
@@ -0,0 +1,14 @@
+SUMMARY = "An example of partitioned image."
+
+IMAGE_INSTALL = "packagegroup-core-boot ${ROOTFS_PKGMANAGE_BOOTSTRAP}"
+
+IMAGE_FSTYPES = "wic.bz2"
+RM_OLD_IMAGE = "1"
+
+# core-image-minimal is referenced in .wks, so we need its rootfs
+# to be ready before our rootfs
+do_rootfs[depends] += "core-image-minimal:do_rootfs"
+
+IMAGE_ROOTFS_EXTRA_SPACE = "2000"
+
+inherit image
diff --git a/meta-selftest/recipes-test/images/wic-image-minimal.wks b/meta-selftest/recipes-test/images/wic-image-minimal.wks
new file mode 100644
index 0000000..29cd8f2
--- /dev/null
+++ b/meta-selftest/recipes-test/images/wic-image-minimal.wks
@@ -0,0 +1,10 @@
+# short-description: Example of partitioned image with complex layout
+# long-description: This image contains boot partition and 3 rootfs partitions
+# created from core-image-minimal and wic-image-minimal image recipes.
+
+part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
+part / --source rootfs --ondisk sda --fstype=ext2 --label platform --align 1024
+part /core --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext2 --label core --align 1024
+part /backup --source rootfs --rootfs-dir=wic-image-minimal --ondisk sda --fstype=ext2 --label backup --align 1024
+
+bootloader  --timeout=0  --append="rootwait console=tty0"
-- 
2.1.4

[PATCH 4/5] oe-selftest: test building wic image by bitbake

[Ed Bartosh]

Added test case to verify building of wic-image-minimal recipe
and produced artifacts: manifest and bzipped partitioned image.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index fe8a2d0..deb2333 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -188,3 +188,17 @@ class Wic(oeSelfTest):
             for var in wicvars:
                 self.assertTrue(var in content, "%s is not in .env file" % var)
                 self.assertTrue(content[var])
+
+    def test20_wic_image_type(self):
+        """Test building wic images by bitbake"""
+        self.assertEqual(0, bitbake('wic-image-minimal').status)
+
+        deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE')
+        machine = get_bb_var('MACHINE')
+        prefix = os.path.join(deploy_dir, 'wic-image-minimal-%s.' % machine)
+        # check if we have result image and manifests symlinks
+        # pointing to existing files
+        for suffix in ('wic.bz2', 'manifest'):
+            path = prefix + suffix
+            self.assertTrue(os.path.islink(path))
+            self.assertTrue(os.path.isfile(os.path.realpath(path)))
-- 
2.1.4

[PATCH 5/5] wic: use bitbake variable ROOTFS_SIZE

[Ed Bartosh]

If bitbake image is referenced in .ks file and --size is not used
there wic uses ROOTFS_SIZE variable to set minimum partition size.

ROOTFS_SIZE is calculated in meta/lib/oe/image.py when rootfs is
created. The calculation is done using other image parameters:
IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT, IMAGE_OVERHEAD_FACTOR
and IMAGE_ROOTFS_EXTRA_SPACE.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/imager/direct.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/scripts/lib/wic/imager/direct.py b/scripts/lib/wic/imager/direct.py
index 57b1335..d68fd2a 100644
--- a/scripts/lib/wic/imager/direct.py
+++ b/scripts/lib/wic/imager/direct.py
@@ -29,6 +29,7 @@ import shutil
 
 from wic import kickstart, msger
 from wic.utils import fs_related
+from wic.utils.oe.misc import get_bitbake_var
 from wic.utils.partitionedfs import Image
 from wic.utils.errors import CreatorError, ImageError
 from wic.imager.baseimager import BaseImageCreator
@@ -229,6 +230,19 @@ class DirectImageCreator(BaseImageCreator):
         fstab_path = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
 
         for p in parts:
+            # get rootfs size from bitbake variable if it's not set in .ks file
+            if not p.size:
+                # and if rootfs name is specified for the partition
+                image_name = p.get_rootfs()
+                if image_name:
+                    # Bitbake variable ROOTFS_SIZE is calculated in
+                    # Image._get_rootfs_size method from meta/lib/oe/image.py
+                    # using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT,
+                    # IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE
+                    rsize_bb = get_bitbake_var('ROOTFS_SIZE', image_name)
+                    if rsize_bb:
+                        # convert from Kb to Mb
+                        p.size = int(rsize_bb) / 1024
             # need to create the filesystems in order to get their
             # sizes before we can add them and do the layout.
             # Image.create() actually calls __format_disks() to create
-- 
2.1.4

Re: [PATCH 1/5] image.py: write bitbake variables to .env file

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

On Sun, Aug 30, 2015 at 2:14 PM, Ed Bartosh <ed.bartosh@linux.intel.com>
wrote:

> Write set of bitbake variables used by wic into
> build/tmp/sysroots/<machine>/imagedata/<image>.env
>
> List of variables is defined in WICVARS variable in
> meta/classes/image_types.bbclass.
>
> This is needed for wic to be able to get bitbake variables without
> running 'bitbake -e'.
>
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
>

The function name (_write_env) sounds generic, as does the commit message,
and the filename, yet it relies on a variable which is WIC-specific. This
seems rather inconsistent to me, either this thing is for wic alone, or it
isn't, and the function name, file name, and variable name should reflect
that.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1399 bytes --]

Re: [PATCH 1/5] image.py: write bitbake variables to .env file

[Ed Bartosh]

On Sun, Aug 30, 2015 at 09:22:50PM -0700, Christopher Larson wrote:
> On Sun, Aug 30, 2015 at 2:14 PM, Ed Bartosh <ed.bartosh@linux.intel.com>
> wrote:
> 
> > Write set of bitbake variables used by wic into
> > build/tmp/sysroots/<machine>/imagedata/<image>.env
> >
> > List of variables is defined in WICVARS variable in
> > meta/classes/image_types.bbclass.
> >
> > This is needed for wic to be able to get bitbake variables without
> > running 'bitbake -e'.
> >
> > Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> >
> 
> The function name (_write_env) sounds generic, as does the commit message,
> and the filename, yet it relies on a variable which is WIC-specific. This
> seems rather inconsistent to me, either this thing is for wic alone, or it
> isn't, and the function name, file name, and variable name should reflect
> that.

Thanks for pointing this out. I'll rename _write_env to _write_wic_env it in the next patchset.

--
Regards,
Ed