[PATCH v2 0/3] bmaptool support

[PATCH v2 0/3] bmaptool support

[Ed Bartosh]

Hi,

This patchset adds ability to use bmaptool to flash images faster than
using traditional methods.

Bmaptool is a generic tool for creating the block map (bmap) for a file
and and copying files using the block map. The idea is that large file
containing unused blocks, like raw system image files, can be copied or
flashed a lot faster with bmaptool than with traditional tools like
"dd" or "cp".

Here is an example of flashing image using dd and bmaptool showing
around 25% performance gain in flashing time:

> ls -slh
 19M -rw-r--r-- 1 ed ed  26M Apr 26 11:36 core-image-minimal-qemux86-64-20160426083554.rootfs.ext4
4.0K -rw-r--r-- 1 ed ed 2.7K Apr 26 11:36 core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap

> time dd if=core-image-minimal-qemux86-64-20160426083554.rootfs.ext4 of=/dev/sdb
52302+0 records in
52302+0 records out
26778624 bytes (27 MB) copied, 2.51183 s, 10.7 MB/s

real    0m2.515s
user    0m0.014s
sys     0m0.329s

> time bmaptool copy core-image-minimal-qemux86-64-20160426083554.rootfs.ext4 /dev/sdb
bmaptool: info: discovered bmap file 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap'
bmaptool: info: block map format version 2.0
bmaptool: info: 6538 blocks of size 4096 (25.5 MiB), mapped 4799 blocks (18.7 MiB or 73.4%)
bmaptool: info: copying image 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4' to block device '/dev/sdb' using bmap file 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap'
bmaptool: info: 100% copied
bmaptool: info: synchronizing '/dev/sdb'
bmaptool: info: copying time: 1.8s, copying speed 10.3 MiB/sec

real    0m1.909s
user    0m0.226s
sys     0m0.052s

The test was repeated 5 times with the same image and the same usb stick device.

Changes in v2: Combined Alexander's and my bmap-tools recipes.
               Got rid of generating standalone script as it breaks build of bmap-tools-native.

The following changes since commit a9b503b268e94d311f892fa00c5d6bd9ffdb228e:

  license.bbclass: make sure that image manifest dir exists (2016-04-22 16:28:57 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/oe-core/bmap-tools-9414
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/bmap-tools-9414

Alexander D. Kanevskiy (1):
  image types: add bmap generation option

Ed Bartosh (2):
  bmap-tools: initial commit, version 3.2
  selftest: add bmap test

 meta/classes/image_types.bbclass                  |  4 +++-
 meta/lib/oeqa/selftest/imagefeatures.py           | 27 +++++++++++++++++++++++
 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 24 ++++++++++++++++++++
 3 files changed, 54 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb

-- 
2.1.4

[PATCH v2 1/3] bmap-tools: initial commit, version 3.2

[Ed Bartosh]

Bmap-tools - tools to generate block map (AKA bmap) and flash images
using bmap. Bmaptool is a generic tool for creating the block map
(bmap) for a file and copying files using the block map.

The idea is that large file containing unused blocks, like raw system
image files, can be copied or flashed a lot faster with bmaptool than
with traditional tools like "dd" or "cp".

[YOCTO #9414]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 24 +++++++++++++++++++++++
 1 file changed, 24 insertions(+)
 create mode 100644 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb

diff --git a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
new file mode 100644
index 0000000..e10f5fd
--- /dev/null
+++ b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
@@ -0,0 +1,24 @@
+SUMMARY = "Tools to generate block map (AKA bmap) and flash images using bmap"
+DESCRIPTION = "Bmap-tools - tools to generate block map (AKA bmap) and flash images using \
+bmap. Bmaptool is a generic tool for creating the block map (bmap) for a file, \
+and copying files using the block map. The idea is that large file containing \
+unused blocks, like raw system image files, can be copied or flashed a lot \
+faster with bmaptool than with traditional tools like "dd" or "cp"."
+HOMEPAGE = "http://git.infradead.org/users/dedekind/bmap-tools.git"
+SECTION = "console/utils"
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
+
+SRC_URI = "ftp://ftp.infradead.org/pub/${BPN}/${BPN}-${PV}.tgz"
+SRC_URI[md5sum] = "92cdad1cb4dfa0cca7176c8e22752616"
+SRC_URI[sha256sum] = "cc6c7f7dc0a37e2a32deb127308e24e6c4b80bfb54f3803c308efab02bf2d434"
+
+RDEPENDS_${PN} = "python-core python-compression"
+
+inherit setuptools
+
+BBCLASSEXTEND = "native"
+
+do_install_append_class-native() {
+    sed -i -e 's|^#!.*/usr/bin/env python|#! /usr/bin/env nativepython|' ${D}${bindir}/bmaptool
+}
-- 
2.1.4

Re: [PATCH v2 1/3] bmap-tools: initial commit, version 3.2

[Alexander Kanevskiy]

[-- Attachment #1: Type: text/plain, Size: 2680 bytes --]

On Wed, Apr 27, 2016 at 12:36 PM, Ed Bartosh <ed.bartosh@linux.intel.com>
wrote:

> Bmap-tools - tools to generate block map (AKA bmap) and flash images
> using bmap. Bmaptool is a generic tool for creating the block map
> (bmap) for a file and copying files using the block map.
>
> The idea is that large file containing unused blocks, like raw system
> image files, can be copied or flashed a lot faster with bmaptool than
> with traditional tools like "dd" or "cp".
>
> [YOCTO #9414]
>
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 24
> +++++++++++++++++++++++
>  1 file changed, 24 insertions(+)
>  create mode 100644 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
>
> diff --git a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
> b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
> new file mode 100644
> index 0000000..e10f5fd
> --- /dev/null
> +++ b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
> @@ -0,0 +1,24 @@
> +SUMMARY = "Tools to generate block map (AKA bmap) and flash images using
> bmap"
> +DESCRIPTION = "Bmap-tools - tools to generate block map (AKA bmap) and
> flash images using \
> +bmap. Bmaptool is a generic tool for creating the block map (bmap) for a
> file, \
> +and copying files using the block map. The idea is that large file
> containing \
> +unused blocks, like raw system image files, can be copied or flashed a
> lot \
> +faster with bmaptool than with traditional tools like "dd" or "cp"."
> +HOMEPAGE = "http://git.infradead.org/users/dedekind/bmap-tools.git"
> +SECTION = "console/utils"
> +LICENSE = "GPLv2"
> +LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
> +
> +SRC_URI = "ftp://ftp.infradead.org/pub/${BPN}/${BPN}-${PV}.tgz"
> +SRC_URI[md5sum] = "92cdad1cb4dfa0cca7176c8e22752616"
> +SRC_URI[sha256sum] =
> "cc6c7f7dc0a37e2a32deb127308e24e6c4b80bfb54f3803c308efab02bf2d434"
> +
> +RDEPENDS_${PN} = "python-core python-compression"
> +
> +inherit setuptools
> +
> +BBCLASSEXTEND = "native"
> +
> +do_install_append_class-native() {
> +    sed -i -e 's|^#!.*/usr/bin/env python|#! /usr/bin/env nativepython|'
> ${D}${bindir}/bmaptool
> +}
>

better to use recipe that would create in deploy/tools standalone version
of the tool.
Otherwise, there is no easy way for user to utilize generated bmap files.

--
> 2.1.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>



-- 
br, Alexander Kanevskiy

[-- Attachment #2: Type: text/html, Size: 4297 bytes --]

Re: [PATCH v2 1/3] bmap-tools: initial commit, version 3.2

[Paul Eggleton]

On Fri, 29 Apr 2016 01:07:56 Alexander Kanevskiy wrote:
> On Wed, Apr 27, 2016 at 12:36 PM, Ed Bartosh <ed.bartosh@linux.intel.com>
> 
> wrote:
> > Bmap-tools - tools to generate block map (AKA bmap) and flash images
> > using bmap. Bmaptool is a generic tool for creating the block map
> > (bmap) for a file and copying files using the block map.
> > 
> > The idea is that large file containing unused blocks, like raw system
> > image files, can be copied or flashed a lot faster with bmaptool than
> > with traditional tools like "dd" or "cp".
> > 
> > [YOCTO #9414]
> > 
> > Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> > ---
> > 
> >  meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 24
> > 
> > +++++++++++++++++++++++
> > 
> >  1 file changed, 24 insertions(+)
> >  create mode 100644 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
> > 
> > diff --git a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
> > b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
> > new file mode 100644
> > index 0000000..e10f5fd
> > --- /dev/null
> > +++ b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
> > @@ -0,0 +1,24 @@
> > +SUMMARY = "Tools to generate block map (AKA bmap) and flash images using
> > bmap"
> > +DESCRIPTION = "Bmap-tools - tools to generate block map (AKA bmap) and
> > flash images using \
> > +bmap. Bmaptool is a generic tool for creating the block map (bmap) for a
> > file, \
> > +and copying files using the block map. The idea is that large file
> > containing \
> > +unused blocks, like raw system image files, can be copied or flashed a
> > lot \
> > +faster with bmaptool than with traditional tools like "dd" or "cp"."
> > +HOMEPAGE = "http://git.infradead.org/users/dedekind/bmap-tools.git"
> > +SECTION = "console/utils"
> > +LICENSE = "GPLv2"
> > +LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
> > +
> > +SRC_URI = "ftp://ftp.infradead.org/pub/${BPN}/${BPN}-${PV}.tgz"
> > +SRC_URI[md5sum] = "92cdad1cb4dfa0cca7176c8e22752616"
> > +SRC_URI[sha256sum] =
> > "cc6c7f7dc0a37e2a32deb127308e24e6c4b80bfb54f3803c308efab02bf2d434"
> > +
> > +RDEPENDS_${PN} = "python-core python-compression"
> > +
> > +inherit setuptools
> > +
> > +BBCLASSEXTEND = "native"
> > +
> > +do_install_append_class-native() {
> > +    sed -i -e 's|^#!.*/usr/bin/env python|#! /usr/bin/env nativepython|'
> > ${D}${bindir}/bmaptool
> > +}
> 
> better to use recipe that would create in deploy/tools standalone version
> of the tool.
> Otherwise, there is no easy way for user to utilize generated bmap files.

You both probably have had notification already, but just to close the loop I 
found the issue with the do_deploy version, see the comments on bug 9414:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=9414

Cheers,
Paul

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH v2 1/3] bmap-tools: initial commit, version 3.2

[Alexander Kanevskiy]

[-- Attachment #1: Type: text/plain, Size: 1321 bytes --]

On Fri, Apr 29, 2016 at 7:55 AM, Paul Eggleton <
paul.eggleton@linux.intel.com> wrote:

> On Fri, 29 Apr 2016 01:07:56 Alexander Kanevskiy wrote:
> > On Wed, Apr 27, 2016 at 12:36 PM, Ed Bartosh <ed.bartosh@linux.intel.com
> >
> >
> > wrote:
> > > Bmap-tools - tools to generate block map (AKA bmap) and flash images
> > > using bmap. Bmaptool is a generic tool for creating the block map
> > > (bmap) for a file and copying files using the block map.
> > >
> > > The idea is that large file containing unused blocks, like raw system
> > > image files, can be copied or flashed a lot faster with bmaptool than
> > > with traditional tools like "dd" or "cp".
> > >
> > > [YOCTO #9414]
> > >
> > > Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> >
> > better to use recipe that would create in deploy/tools standalone version
> > of the tool.
> > Otherwise, there is no easy way for user to utilize generated bmap files.
>
> You both probably have had notification already, but just to close the
> loop I
> found the issue with the do_deploy version, see the comments on bug 9414:
>
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=9414
>
>
Thank you Paul for your help! Your fix mentioned in the bug for do_depoly
is indeed the best solution.


-- 
br, Alexander Kanevskiy

[-- Attachment #2: Type: text/html, Size: 2111 bytes --]

Re: [PATCH v2 1/3] bmap-tools: initial commit, version 3.2

[Ed Bartosh]

On Fri, Apr 29, 2016 at 11:38:38AM +0300, Alexander Kanevskiy wrote:
> On Fri, Apr 29, 2016 at 7:55 AM, Paul Eggleton <
> paul.eggleton@linux.intel.com> wrote:
> 
> > On Fri, 29 Apr 2016 01:07:56 Alexander Kanevskiy wrote:
> > > On Wed, Apr 27, 2016 at 12:36 PM, Ed Bartosh <ed.bartosh@linux.intel.com
> > >
> > >
> > > wrote:
> > > > Bmap-tools - tools to generate block map (AKA bmap) and flash images
> > > > using bmap. Bmaptool is a generic tool for creating the block map
> > > > (bmap) for a file and copying files using the block map.
> > > >
> > > > The idea is that large file containing unused blocks, like raw system
> > > > image files, can be copied or flashed a lot faster with bmaptool than
> > > > with traditional tools like "dd" or "cp".
> > > >
> > > > [YOCTO #9414]
> > > >
> > > > Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> > >
> > > better to use recipe that would create in deploy/tools standalone version
> > > of the tool.
> > > Otherwise, there is no easy way for user to utilize generated bmap files.
> >
> > You both probably have had notification already, but just to close the
> > loop I
> > found the issue with the do_deploy version, see the comments on bug 9414:
> >
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=9414
> >
> >
> Thank you Paul for your help! Your fix mentioned in the bug for do_depoly
> is indeed the best solution.
>

OK. I'll add this functionality back in a separate patch.

--
Regards,
Ed

[PATCH v2 2/3] image types: add bmap generation option

[Ed Bartosh]

From: "Alexander D. Kanevskiy" <kad@kad.name>

bmap image conversion type allows to create block map files
for sparse images. Bmap file can be used together with bmap-tools
for efficiently flash images to raw devices (hdd or usb drive)

[YOCTO #9414]

Signed-off-by: Alexander D. Kanevskiy <kad@kad.name>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image_types.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 53af7ca..19f89f3 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -262,7 +262,7 @@ IMAGE_TYPES = " \
     wic wic.gz wic.bz2 wic.lzma \
 "
 
-COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum"
+COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap"
 COMPRESS_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 COMPRESS_CMD_gz = "gzip -f -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
 COMPRESS_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
@@ -275,12 +275,14 @@ COMPRESS_CMD_sha224sum = "sha224sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
 COMPRESS_CMD_sha256sum = "sha256sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha256sum"
 COMPRESS_CMD_sha384sum = "sha384sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha384sum"
 COMPRESS_CMD_sha512sum = "sha512sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha512sum"
+COMPRESS_CMD_bmap = "bmaptool create ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.bmap"
 COMPRESS_DEPENDS_lzma = "xz-native"
 COMPRESS_DEPENDS_gz = ""
 COMPRESS_DEPENDS_bz2 = "pbzip2-native"
 COMPRESS_DEPENDS_xz = "xz-native"
 COMPRESS_DEPENDS_lz4 = "lz4-native"
 COMPRESS_DEPENDS_sum = "mtd-utils-native"
+COMPRESS_DEPENDS_bmap = "bmap-tools-native"
 
 RUNNABLE_IMAGE_TYPES ?= "ext2 ext3 ext4"
 RUNNABLE_MACHINE_PATTERNS ?= "qemu"
-- 
2.1.4

[PATCH v2 3/3] selftest: add bmap test

[Ed Bartosh]

Added test_bmap to imagefeatures tests.
It tests if bmap file is generated for the images and
if the image is sparse.

[YOCTO #9414]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/imagefeatures.py | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/meta/lib/oeqa/selftest/imagefeatures.py b/meta/lib/oeqa/selftest/imagefeatures.py
index 8a53899..08e382f 100644
--- a/meta/lib/oeqa/selftest/imagefeatures.py
+++ b/meta/lib/oeqa/selftest/imagefeatures.py
@@ -98,3 +98,30 @@ class ImageFeatures(oeSelfTest):
         # Build a core-image-weston
         bitbake('core-image-weston')
 
+    def test_bmap(self):
+        """
+        Summary:     Check bmap support
+        Expected:    1. core-image-minimal can be build with bmap support
+                     2. core-image-minimal is sparse
+        Product:     oe-core
+        Author:      Ed Bartosh <ed.bartosh@linux.intel.com>
+        """
+
+        features = 'IMAGE_FSTYPES += " ext4 ext4.bmap"'
+        self.write_config(features)
+
+        image_name = 'core-image-minimal'
+        bitbake(image_name)
+
+        deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE')
+        link_name = get_bb_var('IMAGE_LINK_NAME', image_name)
+        image_path = os.path.join(deploy_dir, "%s.ext4" % link_name)
+        bmap_path = "%s.bmap" % image_path
+
+        # check if result image and bmap file are in deploy directory
+        self.assertTrue(os.path.exists(image_path))
+        self.assertTrue(os.path.exists(bmap_path))
+
+        # check if result image is sparse
+        image_stat = os.stat(image_path)
+        self.assertTrue(image_stat.st_size > image_stat.st_blocks * 512)
-- 
2.1.4