[PATCH v2 0/7] poky-tiny improvements

[PATCH v2 0/7] poky-tiny improvements

[Alejandro Hernandez]

This is the first set of patches meant to improve some of the characteristics
of poky-tiny, these patches are from or based on Tom Zanussi's work; includes
a new image recipe to be able to boot poky-tiny from initrd, along with 
changes to be able to create it using bootimg-efi, make it compatible with 
systemd-boot, and scripts to measure size of the image, kernel and modules.


The following changes since commit 7d5dec1b1f3d1580e10bf4f92ef4aac4edb5ee77:

  gummiboot: Remove old gummiboot recipe, related class and wks  file (2017-01-11 19:45:41 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib hsalejandro/tinymaster3
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=hsalejandro/tinymaster3

Alejandro Hernandez (3):
  image-.bbclass: standarize usage of DEPLOY_DIR_IMAGE
  core-image-tiny-initramfs: Fix error message shown after a successful
    initrd boot
  systemd-boot.bbclass: Fix SYSYTEMD_BOOT_CFG creation

Tom Zanussi (4):
  image-live-artifacts: Add support for creating image artifacts only
  wic: Look for image artifacts in a common location
  core-image-tiny-initramfs: Add and image creating image artifacts only
  tiny scripts improvements: python3 fixes and new tool ksum

 meta/classes/image-live-artifacts.bbclass          |  40 +++++
 meta/classes/image-live.bbclass                    |  18 +--
 meta/classes/image-vm.bbclass                      |  12 +-
 meta/classes/image.bbclass                         |   3 +
 meta/classes/image_types.bbclass                   |   4 +-
 meta/classes/systemd-boot.bbclass                  |   3 +
 meta/lib/oeqa/selftest/wic.py                      |   2 +-
 .../images/core-image-tiny-initramfs.bb            |  42 ++++++
 scripts/lib/wic/plugins/source/bootimg-efi.py      |  28 +++-
 .../lib/wic/plugins/source/isoimage-isohybrid.py   |  21 ++-
 scripts/tiny/ksize.py                              |  17 ++-
 scripts/tiny/ksum.py                               | 168 +++++++++++++++++++++
 12 files changed, 323 insertions(+), 35 deletions(-)
 create mode 100644 meta/classes/image-live-artifacts.bbclass
 create mode 100644 meta/recipes-core/images/core-image-tiny-initramfs.bb
 create mode 100755 scripts/tiny/ksum.py

-- 
2.6.6

[PATCH v2 1/7] image-live-artifacts: Add support for creating image artifacts only

[Alejandro Hernandez]

From: Tom Zanussi <tom.zanussi@linux.intel.com>

Rather than create an actual image, just put the image artifacts in an
'artifacts' directory that can then be picked up by wic.

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
---
 meta/classes/image-live-artifacts.bbclass | 40 +++++++++++++++++++++++++++++++
 meta/classes/image.bbclass                |  3 +++
 meta/classes/image_types.bbclass          |  2 ++
 3 files changed, 45 insertions(+)
 create mode 100644 meta/classes/image-live-artifacts.bbclass

diff --git a/meta/classes/image-live-artifacts.bbclass b/meta/classes/image-live-artifacts.bbclass
new file mode 100644
index 0000000..1501a9b
--- /dev/null
+++ b/meta/classes/image-live-artifacts.bbclass
@@ -0,0 +1,40 @@
+# Create the artifacts only on an ARTIFACTS_DIR,
+# which can be later picked up by wic
+
+inherit live-vm-common
+
+do_bootimg[depends] += "dosfstools-native:do_populate_sysroot \
+                        mtools-native:do_populate_sysroot \
+                        cdrtools-native:do_populate_sysroot \
+                        virtual/kernel:do_deploy \
+                        ${MLPREFIX}syslinux:do_populate_sysroot \
+                        syslinux-native:do_populate_sysroot \
+                        ${@oe.utils.ifelse(d.getVar('COMPRESSISO', False),'zisofs-tools-native:do_populate_sysroot','')} \
+                        "
+
+
+LABELS_LIVE ?= "boot install"
+
+ARTIFACTS_DIR = "${DEPLOY_DIR_IMAGE}/artifacts"
+
+populate_bootloader() {
+	populate_kernel ${ARTIFACTS_DIR}
+
+	if [ "${PCBIOS}" = "1" ]; then
+		syslinux_hddimg_populate ${ARTIFACTS_DIR}
+	fi
+	if [ "${EFI}" = "1" ]; then
+		efi_hddimg_populate ${ARTIFACTS_DIR}
+	fi
+}
+
+python do_bootimg() {
+    set_live_vm_vars(d, 'LIVE')
+    if d.getVar("PCBIOS", True) == "1":
+        bb.build.exec_func('build_syslinux_cfg', d)
+    if d.getVar("EFI", True) == "1":
+        bb.build.exec_func('build_efi_cfg', d)
+    bb.build.exec_func('populate_bootloader', d)
+}
+
+addtask bootimg before do_image_complete
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 04fd5f9..fd249c8 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -130,6 +130,9 @@ do_rootfs[vardeps] += "${@rootfs_variables(d)}"
 do_build[depends] += "virtual/kernel:do_deploy"
 
 def build_live(d):
+    if bb.utils.contains("IMAGE_FSTYPES", "live-artifacts-only", "live-artifacts-only", "0", d) == "live-artifacts-only":
+         return "image-live-artifacts"
+
     if bb.utils.contains("IMAGE_FSTYPES", "live", "live", "0", d) == "0": # live is not set but hob might set iso or hddimg
         d.setVar('NOISO', bb.utils.contains('IMAGE_FSTYPES', "iso", "0", "1", d))
         d.setVar('NOHDD', bb.utils.contains('IMAGE_FSTYPES', "hddimg", "0", "1", d))
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 8f04849..b9359b2 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -270,6 +270,8 @@ python () {
                 bb.build.addtask('do_write_wks_template', 'do_image_wic', None, d)
 }
 
+IMAGE_CMD_live-artifacts-only = " "
+
 EXTRA_IMAGECMD = ""
 
 inherit siteinfo
-- 
2.6.6

[PATCH v2 2/7] wic: Look for image artifacts in a common location

[Alejandro Hernandez]

From: Tom Zanussi <tom.zanussi@linux.intel.com>

Rather than have each image type look for artifacts in image-specific
locations, move towards having them look for artifacts in a common
location, in this case DEPLOY_DIR_IMAGE/artifacts.

This only applies if the oe-core image classes have been modified to
generate artifacts there (e.g. using image-live-artifacts).

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
---
 meta/classes/image_types.bbclass                   |  2 +-
 meta/lib/oeqa/selftest/wic.py                      |  2 +-
 scripts/lib/wic/plugins/source/bootimg-efi.py      | 28 +++++++++++++++++-----
 .../lib/wic/plugins/source/isoimage-isohybrid.py   | 21 ++++++++++------
 4 files changed, 38 insertions(+), 15 deletions(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index b9359b2..a66210f 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -371,4 +371,4 @@ IMAGE_TYPES_MASKED ?= ""
 
 # The WICVARS variable is used to define list of bitbake variables used in wic code
 # variables from this list is written to <image>.env file
-WICVARS ?= "BBLAYERS IMGDEPLOYDIR DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"
+WICVARS ?= "BBLAYERS IMGDEPLOYDIR DEPLOY_DIR_IMAGE IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"
diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index 49bbfe3..cda6261 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -350,7 +350,7 @@ class Wic(oeSelfTest):
 
         wicvars = set(get_bb_var('WICVARS', image).split())
         # filter out optional variables
-        wicvars = wicvars.difference(('HDDDIR', 'IMAGE_BOOT_FILES',
+        wicvars = wicvars.difference(('DEPLOY_DIR_IMAGE', 'IMAGE_BOOT_FILES',
                                       'INITRD', 'ISODIR'))
         with open(path) as envfile:
             content = dict(line.split("=", 1) for line in envfile)
diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index 305e910..2c16a0f 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -42,7 +42,7 @@ class BootimgEFIPlugin(SourcePlugin):
     name = 'bootimg-efi'
 
     @classmethod
-    def do_configure_grubefi(cls, hdddir, creator, cr_workdir):
+    def do_configure_grubefi(cls, hdddir, creator, cr_workdir, source_params):
         """
         Create loader-specific (grub-efi) config
         """
@@ -82,7 +82,7 @@ class BootimgEFIPlugin(SourcePlugin):
         cfg.close()
 
     @classmethod
-    def do_configure_systemdboot(cls, hdddir, creator, cr_workdir):
+    def do_configure_systemdboot(cls, hdddir, creator, cr_workdir, source_params):
         """
         Create loader-specific systemd-boot/gummiboot config
         """
@@ -98,6 +98,18 @@ class BootimgEFIPlugin(SourcePlugin):
         loader_conf += "default boot\n"
         loader_conf += "timeout %d\n" % bootloader.timeout
 
+        initrd = ""
+
+        if source_params['initrd']:
+	    initrd = source_params['initrd']
+            # obviously we need to have a common common deploy var
+            bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
+            if not bootimg_dir:
+                msger.error("Couldn't find DEPLOY_DIR_IMAGE, exiting\n")
+
+            cp_cmd = "cp %s/%s %s" % (bootimg_dir, initrd, hdddir)
+            exec_cmd(cp_cmd, True)
+
         msger.debug("Writing systemd-boot config %s/hdd/boot/loader/loader.conf" \
                         % cr_workdir)
         cfg = open("%s/hdd/boot/loader/loader.conf" % cr_workdir, "w")
@@ -127,6 +139,9 @@ class BootimgEFIPlugin(SourcePlugin):
             boot_conf += "options LABEL=Boot root=%s %s\n" % \
                              (creator.rootdev, bootloader.append)
 
+           if initrd:
+                boot_conf += "initrd /%s\n" % initrd
+
         msger.debug("Writing systemd-boot config %s/hdd/boot/loader/entries/boot.conf" \
                         % cr_workdir)
         cfg = open("%s/hdd/boot/loader/entries/boot.conf" % cr_workdir, "w")
@@ -148,9 +163,9 @@ class BootimgEFIPlugin(SourcePlugin):
 
         try:
             if source_params['loader'] == 'grub-efi':
-                cls.do_configure_grubefi(hdddir, creator, cr_workdir)
+                cls.do_configure_grubefi(hdddir, creator, cr_workdir, source_params)
             elif source_params['loader'] == 'systemd-boot':
-                cls.do_configure_systemdboot(hdddir, creator, cr_workdir)
+                cls.do_configure_systemdboot(hdddir, creator, cr_workdir, source_params)
             else:
                 msger.error("unrecognized bootimg-efi loader: %s" % source_params['loader'])
         except KeyError:
@@ -167,9 +182,10 @@ class BootimgEFIPlugin(SourcePlugin):
         In this case, prepare content for an EFI (grub) boot partition.
         """
         if not bootimg_dir:
-            bootimg_dir = get_bitbake_var("HDDDIR")
+	    bootimg_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
             if not bootimg_dir:
-                msger.error("Couldn't find HDDDIR, exiting\n")
+                 msger.error("Couldn't find DEPLOY_DIR_IMAGE, exiting\n")
+            bootimg_dir += "/artifacts"
             # just so the result notes display it
             creator.set_bootimg_dir(bootimg_dir)
 
diff --git a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
index 3858fd4..6e64ce9 100644
--- a/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
+++ b/scripts/lib/wic/plugins/source/isoimage-isohybrid.py
@@ -298,7 +298,7 @@ class IsoImagePlugin(SourcePlugin):
         part.rootfs_dir = rootfs_dir
 
         # Prepare rootfs.img
-        hdd_dir = get_bitbake_var("HDDDIR")
+        hdd_dir = get_bitbake_var("DEPLOY_DIR_IMAGE")
         img_iso_dir = get_bitbake_var("ISODIR")
 
         rootfs_img = "%s/rootfs.img" % hdd_dir
@@ -331,12 +331,19 @@ class IsoImagePlugin(SourcePlugin):
         if os.path.isfile(part.source_file):
             os.remove(part.source_file)
 
-        # Prepare initial ramdisk
-        initrd = "%s/initrd" % hdd_dir
-        if not os.path.isfile(initrd):
-            initrd = "%s/initrd" % img_iso_dir
-        if not os.path.isfile(initrd):
-            initrd = cls._build_initramfs_path(rootfs_dir, cr_workdir)
+        # Support using a different initrd other than default
+        if source_params['initrd']:
+	    initrd = source_params['initrd']
+	    if not hdd_dir:
+		    msger.error("Couldn't find DEPLOY_DIR_IMAGE, exiting\n")
+	    cp_cmd = "cp %s/%s %s" % (hdd_dir, initrd, cr_workdir)
+        else:
+            # Prepare initial ramdisk
+            initrd = "%s/initrd" % hdd_dir
+            if not os.path.isfile(initrd):
+                initrd = "%s/initrd" % img_iso_dir
+            if not os.path.isfile(initrd):
+                initrd = cls._build_initramfs_path(rootfs_dir, cr_workdir)
 
         install_cmd = "install -m 0644 %s %s/initrd" \
             % (initrd, isodir)
-- 
2.6.6

[PATCH 3/7] image-.bbclass: standarize usage of DEPLOY_DIR_IMAGE

[Alejandro Hernandez]

Since wic now looks for boot artifacts on DEPLOY_DIR_IMAGE,
we need to standarize the usage of this variable and drop
HDDDIR to avoid confusion, which will let us drop support
for hddimg in the future.

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
---
 meta/classes/image-live.bbclass | 18 +++++++++---------
 meta/classes/image-vm.bbclass   | 12 ++++++------
 2 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/meta/classes/image-live.bbclass b/meta/classes/image-live.bbclass
index a3d1b4e..60364e6 100644
--- a/meta/classes/image-live.bbclass
+++ b/meta/classes/image-live.bbclass
@@ -60,7 +60,7 @@ python() {
         d.appendVarFlag('do_bootimg', 'depends', ' %s:do_image_complete' % initrd_i)
 }
 
-HDDDIR = "${S}/hddimg"
+DEPLOY_DIR_IMAGE = "${S}/hddimg"
 ISODIR = "${S}/iso"
 EFIIMGDIR = "${S}/efi_img"
 COMPACT_ISODIR = "${S}/iso.z"
@@ -230,29 +230,29 @@ build_fat_img() {
 build_hddimg() {
 	# Create an HDD image
 	if [ "${NOHDD}" != "1" ] ; then
-		populate_live ${HDDDIR}
+		populate_live ${DEPLOY_DIR_IMAGE}
 
 		if [ "${PCBIOS}" = "1" ]; then
-			syslinux_hddimg_populate ${HDDDIR}
+			syslinux_hddimg_populate ${DEPLOY_DIR_IMAGE}
 		fi
 		if [ "${EFI}" = "1" ]; then
-			efi_hddimg_populate ${HDDDIR}
+			efi_hddimg_populate ${DEPLOY_DIR_IMAGE}
 		fi
 
-		# Check the size of ${HDDDIR}/rootfs.img, error out if it
+		# Check the size of ${DEPLOY_DIR_IMAGE}/rootfs.img, error out if it
 		# exceeds 4GB, it is the single file's max size of FAT fs.
-		if [ -f ${HDDDIR}/rootfs.img ]; then
-			rootfs_img_size=`stat -c '%s' ${HDDDIR}/rootfs.img`
+		if [ -f ${DEPLOY_DIR_IMAGE}/rootfs.img ]; then
+			rootfs_img_size=`stat -c '%s' ${DEPLOY_DIR_IMAGE}/rootfs.img`
 			max_size=`expr 4 \* 1024 \* 1024 \* 1024`
 			if [ $rootfs_img_size -gt $max_size ]; then
-				bberror "${HDDDIR}/rootfs.img execeeds 4GB,"
+				bberror "${DEPLOY_DIR_IMAGE}/rootfs.img execeeds 4GB,"
 				bberror "this doesn't work on FAT filesystem, you can try either of:"
 				bberror "1) Reduce the size of rootfs.img"
 				bbfatal "2) Use iso, vmdk or vdi to instead of hddimg\n"
 			fi
 		fi
 
-		build_fat_img ${HDDDIR} ${IMGDEPLOYDIR}/${IMAGE_NAME}.hddimg
+		build_fat_img ${DEPLOY_DIR_IMAGE} ${IMGDEPLOYDIR}/${IMAGE_NAME}.hddimg
 
 		if [ "${PCBIOS}" = "1" ]; then
 			syslinux_hddimg_install
diff --git a/meta/classes/image-vm.bbclass b/meta/classes/image-vm.bbclass
index 35c9244..5015753 100644
--- a/meta/classes/image-vm.bbclass
+++ b/meta/classes/image-vm.bbclass
@@ -50,26 +50,26 @@ DISK_SIGNATURE ?= "${DISK_SIGNATURE_GENERATED}"
 DISK_SIGNATURE[vardepsexclude] = "DISK_SIGNATURE_GENERATED"
 
 build_boot_dd() {
-	HDDDIR="${S}/hdd/boot"
+	DEPLOY_DIR_IMAGE="${S}/hdd/boot"
 	HDDIMG="${S}/hdd.image"
 	IMAGE=${IMGDEPLOYDIR}/${IMAGE_NAME}.hdddirect
 
-	populate_kernel $HDDDIR
+	populate_kernel $DEPLOY_DIR_IMAGE
 
 	if [ "${PCBIOS}" = "1" ]; then
-		syslinux_hddimg_populate $HDDDIR
+		syslinux_hddimg_populate $DEPLOY_DIR_IMAGE
 	fi
 	if [ "${EFI}" = "1" ]; then
-		efi_hddimg_populate $HDDDIR
+		efi_hddimg_populate $DEPLOY_DIR_IMAGE
 	fi
 
-	BLOCKS=`du -bks $HDDDIR | cut -f 1`
+	BLOCKS=`du -bks $DEPLOY_DIR_IMAGE | cut -f 1`
 	BLOCKS=`expr $BLOCKS + ${BOOTDD_EXTRA_SPACE}`
 
 	# Remove it since mkdosfs would fail when it exists
 	rm -f $HDDIMG
 	mkdosfs -n ${BOOTDD_VOLUME_ID} -S 512 -C $HDDIMG $BLOCKS 
-	mcopy -i $HDDIMG -s $HDDDIR/* ::/
+	mcopy -i $HDDIMG -s $DEPLOY_DIR_IMAGE/* ::/
 
 	if [ "${PCBIOS}" = "1" ]; then
 		syslinux_hdddirect_install $HDDIMG
-- 
2.6.6

[PATCH v2 4/7] core-image-tiny-initramfs: Add and image creating image artifacts only

[Alejandro Hernandez]

From: Tom Zanussi <tom.zanussi@linux.intel.com>

Add an image that simply creates image artifacts using
image-live-artifacts support instead of creating an actual image.

The image artifacts can then be subsequently assembled by an external
tool such as wic to create an actual image.

This eliminates redundant image creation when using such tools.

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
---
 .../images/core-image-tiny-initramfs.bb            | 30 ++++++++++++++++++++++
 1 file changed, 30 insertions(+)
 create mode 100644 meta/recipes-core/images/core-image-tiny-initramfs.bb

diff --git a/meta/recipes-core/images/core-image-tiny-initramfs.bb b/meta/recipes-core/images/core-image-tiny-initramfs.bb
new file mode 100644
index 0000000..71b7488
--- /dev/null
+++ b/meta/recipes-core/images/core-image-tiny-initramfs.bb
@@ -0,0 +1,30 @@
+# Simple initramfs image artifact generation for tiny images.
+DESCRIPTION = "Tiny image capable of booting a device. The kernel includes \
+the Minimal RAM-based Initial Root Filesystem (initramfs), which finds the \
+first 'init' program more efficiently.  core-image-tiny-initramfs doesn't \
+actually generate an image but rather generates boot and rootfs artifacts \
+into a common location that can subsequently be picked up by external image \
+generation tools such as wic."
+
+PACKAGE_INSTALL = "initramfs-live-boot packagegroup-core-boot dropbear ${VIRTUAL-RUNTIME_base-utils} udev base-passwd ${ROOTFS_BOOTSTRAP_INSTALL}"
+
+# Do not pollute the initrd image with rootfs features
+IMAGE_FEATURES = ""
+
+export IMAGE_BASENAME = "core-image-tiny-initramfs"
+IMAGE_LINGUAS = ""
+
+LICENSE = "MIT"
+
+# don't actually generate an image, just the artifacts needed for one
+IMAGE_FSTYPES = "${INITRAMFS_FSTYPES} live-artifacts-only"
+
+inherit core-image
+
+IMAGE_ROOTFS_SIZE = "8192"
+IMAGE_ROOTFS_EXTRA_SPACE = "0"
+
+BAD_RECOMMENDATIONS += "busybox-syslog"
+
+# Use the same restriction as initramfs-live-install
+COMPATIBLE_HOST = "(i.86|x86_64).*-linux"
-- 
2.6.6

[PATCH 5/7] core-image-tiny-initramfs: Fix error message shown after a successful initrd boot

[Alejandro Hernandez]

When booting core-image-tiny-initramfs, since we want to live on initrd,
on purpose, we never find a rootfs image to switch root to,
this causes init to show an error as it would with other images,
this patch replaces the message shown to the user, avoiding confusion
when it was indeed a successful boot.

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
---
 meta/recipes-core/images/core-image-tiny-initramfs.bb | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/meta/recipes-core/images/core-image-tiny-initramfs.bb b/meta/recipes-core/images/core-image-tiny-initramfs.bb
index 71b7488..6fcb6a9 100644
--- a/meta/recipes-core/images/core-image-tiny-initramfs.bb
+++ b/meta/recipes-core/images/core-image-tiny-initramfs.bb
@@ -28,3 +28,15 @@ BAD_RECOMMENDATIONS += "busybox-syslog"
 
 # Use the same restriction as initramfs-live-install
 COMPATIBLE_HOST = "(i.86|x86_64).*-linux"
+
+python tinyinitrd () {
+  # Modify our init file so the user knows we drop to shell prompt on purpose
+  newinit = None
+  with open(d.expand('${IMAGE_ROOTFS}/init'), 'r') as init:
+    newinit = init.read()
+    newinit = newinit.replace('Cannot find $ROOT_IMAGE file in /run/media/* , dropping to a shell ', 'Poky Tiny Reference Distribution:')
+  with open(d.expand('${IMAGE_ROOTFS}/init'), 'w') as init:
+    init.write(newinit)
+}
+
+IMAGE_PREPROCESS_COMMAND += "tinyinitrd;"
-- 
2.6.6

[PATCH v2 6/7] tiny scripts improvements: python3 fixes and new tool ksum

[Alejandro Hernandez]

From: Tom Zanussi <tom.zanussi@linux.intel.com>

'ksum.py' generates a combined summary of vmlinux and module sizes for
a built kernel, as a quick tool for comparing the overall effects of
systemic tinification changes.  Execute from the base directory of the
kernel build you want to summarize.  Setting the 'verbose' flag will
display the sizes for each file included in the summary.

Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
---
 scripts/tiny/ksize.py |  17 +++--
 scripts/tiny/ksum.py  | 168 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 180 insertions(+), 5 deletions(-)
 create mode 100755 scripts/tiny/ksum.py

diff --git a/scripts/tiny/ksize.py b/scripts/tiny/ksize.py
index b9d2b19..ea1ca7f 100755
--- a/scripts/tiny/ksize.py
+++ b/scripts/tiny/ksize.py
@@ -41,7 +41,7 @@ def usage():
 class Sizes:
     def __init__(self, glob):
         self.title = glob
-        p = Popen("size -t " + glob, shell=True, stdout=PIPE, stderr=PIPE)
+        p = Popen("size -t " + str(glob), shell=True, stdout=PIPE, stderr=PIPE)
         output = p.communicate()[0].splitlines()
         if len(output) > 2:
             sizes = output[-1].split()[0:4]
@@ -62,18 +62,18 @@ class Report:
         r = Report(filename, title)
         path = os.path.dirname(filename)
 
-        p = Popen("ls " + path + "/*.o | grep -v built-in.o",
+        p = Popen("ls " + str(path) + "/*.o | grep -v built-in.o",
                   shell=True, stdout=PIPE, stderr=PIPE)
         glob = ' '.join(p.communicate()[0].splitlines())
-        oreport = Report(glob, path + "/*.o")
-        oreport.sizes.title = path + "/*.o"
+        oreport = Report(glob, str(path) + "/*.o")
+        oreport.sizes.title = str(path) + "/*.o"
         r.parts.append(oreport)
 
         if subglob:
             p = Popen("ls " + subglob, shell=True, stdout=PIPE, stderr=PIPE)
             for f in p.communicate()[0].splitlines():
                 path = os.path.dirname(f)
-                r.parts.append(Report.create(f, path, path + "/*/built-in.o"))
+                r.parts.append(Report.create(f, path, str(path) + "/*/built-in.o"))
             r.parts.sort(reverse=True)
 
         for b in r.parts:
@@ -116,6 +116,13 @@ class Report:
                self.deltas["data"], self.deltas["bss"]))
         print("\n")
 
+    def __lt__(this, that):
+        if that is None:
+            return 1
+        if not isinstance(that, Report):
+            raise TypeError
+        return this.sizes.total < that.sizes.total
+
     def __cmp__(this, that):
         if that is None:
             return 1
diff --git a/scripts/tiny/ksum.py b/scripts/tiny/ksum.py
new file mode 100755
index 0000000..d4f3892
--- /dev/null
+++ b/scripts/tiny/ksum.py
@@ -0,0 +1,168 @@
+#!/usr/bin/env python
+# ex:ts=4:sw=4:sts=4:et
+# -*- tab-width: 4; c-basic-offset: 4; indent-tabs-mode: nil -*-
+#
+# Copyright (c) 2016, Intel Corporation.
+# All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License along
+# with this program; if not, write to the Free Software Foundation, Inc.,
+# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.
+#
+# DESCRIPTION 'ksum.py' generates a combined summary of vmlinux and
+# module sizes for a built kernel, as a quick tool for comparing the
+# overall effects of systemic tinification changes.  Execute from the
+# base directory of the kernel build you want to summarize.  Setting
+# the 'verbose' flag will display the sizes for each file included in
+# the summary.
+#
+# AUTHORS
+# Tom Zanussi <tom.zanussi (at] linux.intel.com>
+#
+
+__version__ = "0.1.0"
+
+# Python Standard Library modules
+import os
+import sys
+import getopt
+from subprocess import *
+
+def usage():
+    prog = os.path.basename(sys.argv[0])
+    print('Usage: %s [OPTION]...' % prog)
+    print('  -v,                 display sizes for each file')
+    print('  -h, --help          display this help and exit')
+    print('')
+    print('Run %s from the top-level Linux kernel build directory.' % prog)
+
+verbose = False
+
+n_ko_files = 0
+ko_file_list = []
+
+ko_text = 0
+ko_data = 0
+ko_bss = 0
+ko_total = 0
+
+vmlinux_file = ""
+vmlinux_level = 0
+
+vmlinux_text = 0
+vmlinux_data = 0
+vmlinux_bss = 0
+vmlinux_total = 0
+
+def is_vmlinux_file(filename):
+    global vmlinux_level
+    if filename == ("vmlinux") and vmlinux_level == 0:
+        vmlinux_level += 1
+        return True
+    return False
+
+def is_ko_file(filename):
+    if filename.endswith(".ko"):
+        return True
+    return False
+
+def collect_object_files():
+    print "Collecting object files recursively from %s..." % os.getcwd()
+    for dirpath, dirs, files in os.walk(os.getcwd()):
+        for filename in files:
+            if is_ko_file(filename):
+                ko_file_list.append(os.path.join(dirpath, filename))
+            elif is_vmlinux_file(filename):
+                global vmlinux_file
+                vmlinux_file = os.path.join(dirpath, filename)
+    print "Collecting object files [DONE]"
+
+def add_ko_file(filename):
+        p = Popen("size -t " + filename, shell=True, stdout=PIPE, stderr=PIPE)
+        output = p.communicate()[0].splitlines()
+        if len(output) > 2:
+            sizes = output[-1].split()[0:4]
+            if verbose:
+                print "     %10d %10d %10d %10d\t" % \
+                    (int(sizes[0]), int(sizes[1]), int(sizes[2]), int(sizes[3])),
+                print "%s" % filename[len(os.getcwd()) + 1:]
+            global n_ko_files, ko_text, ko_data, ko_bss, ko_total
+            ko_text += int(sizes[0])
+            ko_data += int(sizes[1])
+            ko_bss += int(sizes[2])
+            ko_total += int(sizes[3])
+            n_ko_files += 1
+
+def get_vmlinux_totals():
+        p = Popen("size -t " + vmlinux_file, shell=True, stdout=PIPE, stderr=PIPE)
+        output = p.communicate()[0].splitlines()
+        if len(output) > 2:
+            sizes = output[-1].split()[0:4]
+            if verbose:
+                print "     %10d %10d %10d %10d\t" % \
+                    (int(sizes[0]), int(sizes[1]), int(sizes[2]), int(sizes[3])),
+                print "%s" % vmlinux_file[len(os.getcwd()) + 1:]
+            global vmlinux_text, vmlinux_data, vmlinux_bss, vmlinux_total
+            vmlinux_text += int(sizes[0])
+            vmlinux_data += int(sizes[1])
+            vmlinux_bss += int(sizes[2])
+            vmlinux_total += int(sizes[3])
+
+def sum_ko_files():
+    for ko_file in ko_file_list:
+        add_ko_file(ko_file)
+
+def main():
+    try:
+        opts, args = getopt.getopt(sys.argv[1:], "vh", ["help"])
+    except getopt.GetoptError as err:
+        print('%s' % str(err))
+        usage()
+        sys.exit(2)
+
+    for o, a in opts:
+        if o == '-v':
+            global verbose
+            verbose = True
+        elif o in ('-h', '--help'):
+            usage()
+            sys.exit(0)
+        else:
+            assert False, "unhandled option"
+
+    collect_object_files()
+    sum_ko_files()
+    get_vmlinux_totals()
+
+    print "\nTotals:"
+    print "\nvmlinux:"
+    print "    text\tdata\t\tbss\t\ttotal"
+    print "    %-10d\t%-10d\t%-10d\t%-10d" % \
+        (vmlinux_text, vmlinux_data, vmlinux_bss, vmlinux_total)
+    print "\nmodules (%d):" % n_ko_files
+    print "    text\tdata\t\tbss\t\ttotal"
+    print "    %-10d\t%-10d\t%-10d\t%-10d" % \
+        (ko_text, ko_data, ko_bss, ko_total)
+    print "\nvmlinux + modules:"
+    print "    text\tdata\t\tbss\t\ttotal"
+    print "    %-10d\t%-10d\t%-10d\t%-10d" % \
+        (vmlinux_text + ko_text, vmlinux_data + ko_data, \
+         vmlinux_bss + ko_bss, vmlinux_total + ko_total)
+
+if __name__ == "__main__":
+    try:
+        ret = main()
+    except Exception:
+        ret = 1
+        import traceback
+        traceback.print_exc(5)
+    sys.exit(ret)
-- 
2.6.6

[PATCH 7/7] systemd-boot.bbclass: Fix SYSYTEMD_BOOT_CFG creation

[Alejandro Hernandez]

This patch makes sure the directory which will contain
the systemd configuration (loader.conf) is created before
the configuration file is written, fixing errors when it
tried to write it to a non-existent directory

Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
---
 meta/classes/systemd-boot.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/systemd-boot.bbclass b/meta/classes/systemd-boot.bbclass
index 6718783..6a2cbc8 100644
--- a/meta/classes/systemd-boot.bbclass
+++ b/meta/classes/systemd-boot.bbclass
@@ -72,6 +72,9 @@ python build_efi_cfg() {
         return
 
     cfile = d.getVar('SYSTEMD_BOOT_CFG')
+    cdir = os.path.dirname(cfile)
+    if not os.path.exists(cdir):
+        os.makedirs(cdir)
     try:
          cfgfile = open(cfile, 'w')
     except OSError:
-- 
2.6.6

Re: [PATCH v2 1/7] image-live-artifacts: Add support for creating image artifacts only

[Ylinen, Mikko]

[-- Attachment #1: Type: text/plain, Size: 3795 bytes --]

On Wed, Jan 11, 2017 at 9:55 PM, Alejandro Hernandez <
alejandro.hernandez@linux.intel.com> wrote:

> From: Tom Zanussi <tom.zanussi@linux.intel.com>
>
> Rather than create an actual image, just put the image artifacts in an
> 'artifacts' directory that can then be picked up by wic.
>
>
This is a desired goal now that wic is more widely used. However, any
particular
reason why the implementation talks about "live" and copies what the live
image
do_bootimg gives?


> Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
> Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com>
> ---
>  meta/classes/image-live-artifacts.bbclass | 40
> +++++++++++++++++++++++++++++++
>  meta/classes/image.bbclass                |  3 +++
>  meta/classes/image_types.bbclass          |  2 ++
>  3 files changed, 45 insertions(+)
>  create mode 100644 meta/classes/image-live-artifacts.bbclass
>
> diff --git a/meta/classes/image-live-artifacts.bbclass
> b/meta/classes/image-live-artifacts.bbclass
> new file mode 100644
> index 0000000..1501a9b
> --- /dev/null
> +++ b/meta/classes/image-live-artifacts.bbclass
> @@ -0,0 +1,40 @@
> +# Create the artifacts only on an ARTIFACTS_DIR,
> +# which can be later picked up by wic
> +
> +inherit live-vm-common
> +
> +do_bootimg[depends] += "dosfstools-native:do_populate_sysroot \
>

perhaps have 'do_image_artifacts' and let users/other classes set
additional dependencies.


> +                        mtools-native:do_populate_sysroot \
> +                        cdrtools-native:do_populate_sysroot \
> +                        virtual/kernel:do_deploy \
> +                        ${MLPREFIX}syslinux:do_populate_sysroot \
> +                        syslinux-native:do_populate_sysroot \
>

For example, if I use systemd-boot I don't need syslinux built at all. The
EFI_CLASS/PCBIOS_CLASS
could define the additional depends here (like they are currently doing for
bootimg/bootdirectdisk).

The defaults should be just kernel and initrd(s) (if defined).


> +                        ${@oe.utils.ifelse(d.getVar('COMPRESSISO',
> False),'zisofs-tools-native:do_populate_sysroot','')} \
> +                        "
> +
> +
> +LABELS_LIVE ?= "boot install"
> +
> +ARTIFACTS_DIR = "${DEPLOY_DIR_IMAGE}/artifacts"
> +
> +populate_bootloader() {
> +       populate_kernel ${ARTIFACTS_DIR}
> +
> +       if [ "${PCBIOS}" = "1" ]; then
> +               syslinux_hddimg_populate ${ARTIFACTS_DIR}
> +       fi
> +       if [ "${EFI}" = "1" ]; then
> +               efi_hddimg_populate ${ARTIFACTS_DIR}
> +       fi
> +}
> +
> +python do_bootimg() {
> +    set_live_vm_vars(d, 'LIVE')
> +    if d.getVar("PCBIOS", True) == "1":
> +        bb.build.exec_func('build_syslinux_cfg', d)
> +    if d.getVar("EFI", True) == "1":
> +        bb.build.exec_func('build_efi_cfg', d)
> +    bb.build.exec_func('populate_bootloader', d)
> +}
> +
> +addtask bootimg before do_image_complete
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 04fd5f9..fd249c8 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -130,6 +130,9 @@ do_rootfs[vardeps] += "${@rootfs_variables(d)}"
>  do_build[depends] += "virtual/kernel:do_deploy"
>
>  def build_live(d):
> +    if bb.utils.contains("IMAGE_FSTYPES", "live-artifacts-only",
> "live-artifacts-only", "0", d) == "live-artifacts-only":
> +         return "image-live-artifacts"
>

Is there a need for the image artifacts for other than wic? It could be
made so that if wic is in IMAGE_FSTYPES,
image-artifacts are built by default. This means the logic of inheriting
image-artifacts has a better place, e.g.,
in it's own build_wic() check.

-- Mikko

[-- Attachment #2: Type: text/html, Size: 5489 bytes --]

Re: [PATCH 3/7] image-.bbclass: standarize usage of DEPLOY_DIR_IMAGE

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 970 bytes --]

On 11 January 2017 at 19:55, Alejandro Hernandez <
alejandro.hernandez@linux.intel.com> wrote:

> Since wic now looks for boot artifacts on DEPLOY_DIR_IMAGE,
> we need to standarize the usage of this variable and drop
> HDDDIR to avoid confusion, which will let us drop support
> for hddimg in the future.
>
> Signed-off-by: Alejandro Hernandez <alejandro.hernandez@linux.intel.com>
> ---
>  meta/classes/image-live.bbclass | 18 +++++++++---------
>  meta/classes/image-vm.bbclass   | 12 ++++++------
>

There's other places that use this variable:

meta/lib/oeqa/selftest/wic.py:        wicvars =
wicvars.difference(('HDDDIR', 'IMAGE_BOOT_FILES',
scripts/lib/wic/plugins/source/bootimg-efi.py:            bootimg_dir =
get_bitbake_var("HDDDIR")
scripts/lib/wic/plugins/source/bootimg-efi.py:
 msger.error("Couldn't find HDDDIR, exiting\n")
scripts/lib/wic/plugins/source/isoimage-isohybrid.py:        hdd_dir =
get_bitbake_var("HDDDIR")

Ross

[-- Attachment #2: Type: text/html, Size: 1861 bytes --]

Re: [PATCH 3/7] image-.bbclass: standarize usage of DEPLOY_DIR_IMAGE

[Alejandro Hernandez]

[-- Attachment #1: Type: text/plain, Size: 1256 bytes --]


On 01/12/2017 11:10 AM, Burton, Ross wrote:
>
> On 11 January 2017 at 19:55, Alejandro Hernandez 
> <alejandro.hernandez@linux.intel.com 
> <mailto:alejandro.hernandez@linux.intel.com>> wrote:
>
>     Since wic now looks for boot artifacts on DEPLOY_DIR_IMAGE,
>     we need to standarize the usage of this variable and drop
>     HDDDIR to avoid confusion, which will let us drop support
>     for hddimg in the future.
>
>     Signed-off-by: Alejandro Hernandez
>     <alejandro.hernandez@linux.intel.com
>     <mailto:alejandro.hernandez@linux.intel.com>>
>     ---
>      meta/classes/image-live.bbclass | 18 +++++++++---------
>      meta/classes/image-vm.bbclass   | 12 ++++++------
>

Yes, these are changed on the previous patch (2/7)
>
> There's other places that use this variable:
>
> meta/lib/oeqa/selftest/wic.py:  wicvars = 
> wicvars.difference(('HDDDIR', 'IMAGE_BOOT_FILES',
> scripts/lib/wic/plugins/source/bootimg-efi.py:            bootimg_dir 
> = get_bitbake_var("HDDDIR")
> scripts/lib/wic/plugins/source/bootimg-efi.py:               
>  msger.error("Couldn't find HDDDIR, exiting\n")
> scripts/lib/wic/plugins/source/isoimage-isohybrid.py:        hdd_dir = 
> get_bitbake_var("HDDDIR")
>
> Ross


[-- Attachment #2: Type: text/html, Size: 3193 bytes --]

Re: [PATCH v2 1/7] image-live-artifacts: Add support for creating image artifacts only

[Alejandro Hernandez]

[-- Attachment #1: Type: text/plain, Size: 5204 bytes --]

Hey Mikko


On 01/12/2017 06:52 AM, Ylinen, Mikko wrote:
>
>
> On Wed, Jan 11, 2017 at 9:55 PM, Alejandro Hernandez 
> <alejandro.hernandez@linux.intel.com 
> <mailto:alejandro.hernandez@linux.intel.com>> wrote:
>
>     From: Tom Zanussi <tom.zanussi@linux.intel.com
>     <mailto:tom.zanussi@linux.intel.com>>
>
>     Rather than create an actual image, just put the image artifacts in an
>     'artifacts' directory that can then be picked up by wic.
>
I thought about changing it to boot-artifacts-only, but once I did, 
since it is based on image-live the code looked more confusing in my 
opinion.
Also, we wanted what image-live does  in some parts, in this case 
do_bootimg, but we also avoid other things from image-live.
>
> This is a desired goal now that wic is more widely used. However, any 
> particular
> reason why the implementation talks about "live" and copies what the 
> live image
> do_bootimg gives?
>
>     Signed-off-by: Alejandro Hernandez
>     <alejandro.hernandez@linux.intel.com
>     <mailto:alejandro.hernandez@linux.intel.com>>
>     Signed-off-by: Tom Zanussi <tom.zanussi@linux.intel.com
>     <mailto:tom.zanussi@linux.intel.com>>
>     ---
>      meta/classes/image-live-artifacts.bbclass | 40
>     +++++++++++++++++++++++++++++++
>      meta/classes/image.bbclass                |  3 +++
>      meta/classes/image_types.bbclass          |  2 ++
>      3 files changed, 45 insertions(+)
>      create mode 100644 meta/classes/image-live-artifacts.bbclass
>
>     diff --git a/meta/classes/image-live-artifacts.bbclass
>     b/meta/classes/image-live-artifacts.bbclass
>     new file mode 100644
>     index 0000000..1501a9b
>     --- /dev/null
>     +++ b/meta/classes/image-live-artifacts.bbclass
>     @@ -0,0 +1,40 @@
>     +# Create the artifacts only on an ARTIFACTS_DIR,
>     +# which can be later picked up by wic
>     +
>     +inherit live-vm-common
>     +
>     +do_bootimg[depends] += "dosfstools-native:do_populate_sysroot \
>
>
Again, this was thought based on what image-live does, I'm not saying I 
disagree with you regarding handling the dependencies, but in this case 
you're saying you have the same issue with image-live?

> perhaps have 'do_image_artifacts' and let users/other classes set 
> additional dependencies.
>
>     +                   mtools-native:do_populate_sysroot \
>     +                        cdrtools-native:do_populate_sysroot \
>     +                        virtual/kernel:do_deploy \
>     +                        ${MLPREFIX}syslinux:do_populate_sysroot \
>     +                        syslinux-native:do_populate_sysroot \
>
>
> For example, if I use systemd-boot I don't need syslinux built at all. 
> The EFI_CLASS/PCBIOS_CLASS
> could define the additional depends here (like they are currently 
> doing for bootimg/bootdirectdisk).
>
> The defaults should be just kernel and initrd(s) (if defined).
>
>     +                   ${@oe.utils.ifelse(d.getVar('COMPRESSISO',
>     False),'zisofs-tools-native:do_populate_sysroot','')} \
>     +                        "
>     +
>     +
>     +LABELS_LIVE ?= "boot install"
>     +
>     +ARTIFACTS_DIR = "${DEPLOY_DIR_IMAGE}/artifacts"
>     +
>     +populate_bootloader() {
>     +       populate_kernel ${ARTIFACTS_DIR}
>     +
>     +       if [ "${PCBIOS}" = "1" ]; then
>     +               syslinux_hddimg_populate ${ARTIFACTS_DIR}
>     +       fi
>     +       if [ "${EFI}" = "1" ]; then
>     +               efi_hddimg_populate ${ARTIFACTS_DIR}
>     +       fi
>     +}
>     +
>     +python do_bootimg() {
>     +    set_live_vm_vars(d, 'LIVE')
>     +    if d.getVar("PCBIOS", True) == "1":
>     +        bb.build.exec_func('build_syslinux_cfg', d)
>     +    if d.getVar("EFI", True) == "1":
>     +        bb.build.exec_func('build_efi_cfg', d)
>     +    bb.build.exec_func('populate_bootloader', d)
>     +}
>     +
>     +addtask bootimg before do_image_complete
>     diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>     index 04fd5f9..fd249c8 100644
>     --- a/meta/classes/image.bbclass
>     +++ b/meta/classes/image.bbclass
>     @@ -130,6 +130,9 @@ do_rootfs[vardeps] += "${@rootfs_variables(d)}"
>      do_build[depends] += "virtual/kernel:do_deploy"
>
>      def build_live(d):
>     +    if bb.utils.contains("IMAGE_FSTYPES", "live-artifacts-only",
>     "live-artifacts-only", "0", d) == "live-artifacts-only":
>     +         return "image-live-artifacts"
>
>
Not as far as I know, but so that would work when building something AND 
the artifacts, but what if you want to build the artifacts only?, you 
would have to build everything else as well wouldn't you?

I'm trying to think if this change is not more related to actually 
changing image-live, but I could be misunderstanding things
> Is there a need for the image artifacts for other than wic? It could 
> be made so that if wic is in IMAGE_FSTYPES,
> image-artifacts are built by default. This means the logic of 
> inheriting image-artifacts has a better place, e.g.,
> in it's own build_wic() check.
>
> -- Mikko


[-- Attachment #2: Type: text/html, Size: 9558 bytes --]