[PATCH 0/3] bmaptool support

[PATCH 0/3] bmaptool support

[Ed Bartosh]

Hi,

This patchset adds ability to use bmaptool to flash images faster than
using traditional methods.

Bmaptool is a generic tool for creating the block map (bmap) for a file
and and copying files using the block map. The idea is that large file
containing unused blocks, like raw system image files, can be copied or
flashed a lot faster with bmaptool than with traditional tools like
"dd" or "cp".

Here is an example of flashing image using dd and bmaptool showing
around 25% performance gain in flashing time:

> ls -slh
 19M -rw-r--r-- 1 ed ed  26M Apr 26 11:36 core-image-minimal-qemux86-64-20160426083554.rootfs.ext4
4.0K -rw-r--r-- 1 ed ed 2.7K Apr 26 11:36 core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap

> time dd if=core-image-minimal-qemux86-64-20160426083554.rootfs.ext4 of=/dev/sdb
52302+0 records in
52302+0 records out
26778624 bytes (27 MB) copied, 2.51183 s, 10.7 MB/s

real    0m2.515s
user    0m0.014s
sys     0m0.329s

> time bmaptool copy core-image-minimal-qemux86-64-20160426083554.rootfs.ext4 /dev/sdb
bmaptool: info: discovered bmap file 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap'
bmaptool: info: block map format version 2.0
bmaptool: info: 6538 blocks of size 4096 (25.5 MiB), mapped 4799 blocks (18.7 MiB or 73.4%)
bmaptool: info: copying image 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4' to block device '/dev/sdb' using bmap file 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap'
bmaptool: info: 100% copied
bmaptool: info: synchronizing '/dev/sdb'
bmaptool: info: copying time: 1.8s, copying speed 10.3 MiB/sec

real    0m1.909s
user    0m0.226s
sys     0m0.052s

The test was repeated 5 times with the same image and the same usb stick device.

The following changes since commit a9b503b268e94d311f892fa00c5d6bd9ffdb228e:

  license.bbclass: make sure that image manifest dir exists (2016-04-22 16:28:57 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/oe-core/bmap-tools-9414
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/bmap-tools-9414

Alexander D. Kanevskiy (2):
  bmap-tools: utility to create block map files
  image types: add bmap generation option

Ed Bartosh (1):
  selftest: add bmap test

 meta/classes/image_types.bbclass                  |  4 ++-
 meta/lib/oeqa/selftest/imagefeatures.py           | 27 +++++++++++++++
 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 41 +++++++++++++++++++++++
 3 files changed, 71 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb

--
Regards,
Ed

[PATCH 1/3] bmap-tools: utility to create block map files

[Ed Bartosh]

From: "Alexander D. Kanevskiy" <kad@kad.name>

Bmap-tools - tools to generate block map (AKA bmap) and
flash images using bmap. Bmaptool is a generic tool for
creating the block map (bmap) for a file and copying files
using the block map.

The idea is that large file containing unused blocks, like
raw system image files, can be copied or flashed a lot
faster with bmaptool than with traditional tools like
"dd" or "cp".

Standalone version of the tool provided in deploy/tools
directory and can be easily used on any Linux system with
Python 2.7+ installed.

[YOCTO #9414]

Signed-off-by: Alexander D. Kanevskiy <kad@kad.name>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 41 +++++++++++++++++++++++
 1 file changed, 41 insertions(+)
 create mode 100644 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb

diff --git a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
new file mode 100644
index 0000000..4640820
--- /dev/null
+++ b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
@@ -0,0 +1,41 @@
+SUMMARY = "Tools to generate block map (AKA bmap) and flash images using bmap"
+DESCRIPTION = "Bmap-tools - tools to generate block map (AKA bmap) and flash images using \
+bmap. Bmaptool is a generic tool for creating the block map (bmap) for a file, \
+and copying files using the block map. The idea is that large file containing \
+unused blocks, like raw system image files, can be copied or flashed a lot \
+faster with bmaptool than with traditional tools like "dd" or "cp"."
+HOMEPAGE = "http://git.infradead.org/users/dedekind/bmap-tools.git"
+SECTION = "devel/python"
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
+
+SRC_URI = "ftp://ftp.infradead.org/pub/bmap-tools/bmap-tools-${PV}.tgz"
+SRC_URI[md5sum] = "92cdad1cb4dfa0cca7176c8e22752616"
+SRC_URI[sha256sum] = "cc6c7f7dc0a37e2a32deb127308e24e6c4b80bfb54f3803c308efab02bf2d434"
+
+inherit setuptools deploy
+
+RDEPENDS_${PN} += "python-compression python-core python-unittest"
+
+BBCLASSEXTEND = "nativesdk native"
+
+do_install_append_class-native() {
+    sed -i -e 's|^#!.*/usr/bin/env python|#! /usr/bin/env nativepython|' ${D}${bindir}/bmaptool
+}
+
+do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_TOOLS}"
+do_deploy_class-native() {
+    cp bmaptool __main__.py
+    python -m zipfile -c bmaptool.zip bmaptools __main__.py
+    echo '#!/usr/bin/env python' | cat - bmaptool.zip > bmaptool-standalone
+    install -d ${DEPLOYDIR}
+    install -m 0755 bmaptool-standalone ${DEPLOYDIR}/bmaptool-${PV}
+    rm -f ${DEPLOYDIR}/bmaptool
+    ln -sf ./bmaptool-${PV} ${DEPLOYDIR}/bmaptool
+}
+
+do_deploy() {
+        :
+}
+
+addtask deploy before do_package after do_install
-- 
2.1.4

[PATCH 2/3] image types: add bmap generation option

[Ed Bartosh]

From: "Alexander D. Kanevskiy" <kad@kad.name>

bmap image conversion type allows to create block map files
for sparse images. Bmap file can be used together with bmap-tools
for efficiently flash images to raw devices (hdd or usb drive)

[YOCTO #9414]

Signed-off-by: Alexander D. Kanevskiy <kad@kad.name>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image_types.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 53af7ca..19f89f3 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -262,7 +262,7 @@ IMAGE_TYPES = " \
     wic wic.gz wic.bz2 wic.lzma \
 "
 
-COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum"
+COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap"
 COMPRESS_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 COMPRESS_CMD_gz = "gzip -f -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
 COMPRESS_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
@@ -275,12 +275,14 @@ COMPRESS_CMD_sha224sum = "sha224sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
 COMPRESS_CMD_sha256sum = "sha256sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha256sum"
 COMPRESS_CMD_sha384sum = "sha384sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha384sum"
 COMPRESS_CMD_sha512sum = "sha512sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha512sum"
+COMPRESS_CMD_bmap = "bmaptool create ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.bmap"
 COMPRESS_DEPENDS_lzma = "xz-native"
 COMPRESS_DEPENDS_gz = ""
 COMPRESS_DEPENDS_bz2 = "pbzip2-native"
 COMPRESS_DEPENDS_xz = "xz-native"
 COMPRESS_DEPENDS_lz4 = "lz4-native"
 COMPRESS_DEPENDS_sum = "mtd-utils-native"
+COMPRESS_DEPENDS_bmap = "bmap-tools-native"
 
 RUNNABLE_IMAGE_TYPES ?= "ext2 ext3 ext4"
 RUNNABLE_MACHINE_PATTERNS ?= "qemu"
-- 
2.1.4

[PATCH 3/3] selftest: add bmap test

[Ed Bartosh]

Added test_bmap to imagefeatures tests.
It tests if bmap file is generated for the images and
if the image is sparse.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/imagefeatures.py | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/meta/lib/oeqa/selftest/imagefeatures.py b/meta/lib/oeqa/selftest/imagefeatures.py
index 8a53899..08e382f 100644
--- a/meta/lib/oeqa/selftest/imagefeatures.py
+++ b/meta/lib/oeqa/selftest/imagefeatures.py
@@ -98,3 +98,30 @@ class ImageFeatures(oeSelfTest):
         # Build a core-image-weston
         bitbake('core-image-weston')
 
+    def test_bmap(self):
+        """
+        Summary:     Check bmap support
+        Expected:    1. core-image-minimal can be build with bmap support
+                     2. core-image-minimal is sparse
+        Product:     oe-core
+        Author:      Ed Bartosh <ed.bartosh@linux.intel.com>
+        """
+
+        features = 'IMAGE_FSTYPES += " ext4 ext4.bmap"'
+        self.write_config(features)
+
+        image_name = 'core-image-minimal'
+        bitbake(image_name)
+
+        deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE')
+        link_name = get_bb_var('IMAGE_LINK_NAME', image_name)
+        image_path = os.path.join(deploy_dir, "%s.ext4" % link_name)
+        bmap_path = "%s.bmap" % image_path
+
+        # check if result image and bmap file are in deploy directory
+        self.assertTrue(os.path.exists(image_path))
+        self.assertTrue(os.path.exists(bmap_path))
+
+        # check if result image is sparse
+        image_stat = os.stat(image_path)
+        self.assertTrue(image_stat.st_size > image_stat.st_blocks * 512)
-- 
2.1.4