[PATCH v3 0/6] bmaptool support

[PATCH v3 0/6] bmaptool support

[Ed Bartosh]

Hi,

This patchset adds ability to use bmaptool to flash images faster than
using traditional methods. It contains bmap-tools recipe and generation
of .bmap files for the images generated by bitbake and wic.

Bmaptool is a generic tool for creating the block map (bmap) for a file
and and copying files using the block map. The idea is that large file
containing unused blocks, like raw system image files, can be copied or
flashed a lot faster with bmaptool than with traditional tools like
"dd" or "cp".

Here is an example of flashing image using dd and bmaptool showing
around 25% performance gain in flashing time:

> ls -slh
 19M -rw-r--r-- 1 ed ed  26M Apr 26 11:36 core-image-minimal-qemux86-64-20160426083554.rootfs.ext4
4.0K -rw-r--r-- 1 ed ed 2.7K Apr 26 11:36 core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap

> time dd if=core-image-minimal-qemux86-64-20160426083554.rootfs.ext4 of=/dev/sdb
52302+0 records in
52302+0 records out
26778624 bytes (27 MB) copied, 2.51183 s, 10.7 MB/s

real    0m2.515s
user    0m0.014s
sys     0m0.329s

> time bmaptool copy core-image-minimal-qemux86-64-20160426083554.rootfs.ext4 /dev/sdb
bmaptool: info: discovered bmap file 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap'
bmaptool: info: block map format version 2.0
bmaptool: info: 6538 blocks of size 4096 (25.5 MiB), mapped 4799 blocks (18.7 MiB or 73.4%)
bmaptool: info: copying image 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4' to block device '/dev/sdb' using bmap file 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap'
bmaptool: info: 100% copied
bmaptool: info: synchronizing '/dev/sdb'
bmaptool: info: copying time: 1.8s, copying speed 10.3 MiB/sec

real    0m1.909s
user    0m0.226s
sys     0m0.052s

The test was repeated 5 times with the same image and the same usb stick device.

Changes in v2: Combined Alexander's and my bmap-tools recipes.
               Got rid of generating standalone script as it breaks build of bmap-tools-native.

Changes in v3: Returned back generation of standalone bmaptool script
               Implemented --bmap option for wic
               Added test case to test bmap generation by wic --bmap

The following changes since commit f7b520878babbaa7527151f22c031ae160512753:

  poky: Switch to post release name/version (2016-04-29 07:58:46 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ed/oe-core/bmap-tools-9414
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/bmap-tools-9414

Alexander D. Kanevskiy (1):
  image types: add bmap generation option

Ed Bartosh (5):
  bmap-tools: initial commit, version 3.2
  selftest: add bmap test
  bmap-tools: generate standalone script
  wic: implement --bmap option
  oe-selftest: wic: add test_bmap test case

 meta/classes/image_types.bbclass                  |  4 ++-
 meta/lib/oeqa/selftest/imagefeatures.py           | 27 +++++++++++++++
 meta/lib/oeqa/selftest/wic.py                     |  8 +++++
 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 42 +++++++++++++++++++++++
 scripts/lib/wic/creator.py                        |  1 +
 scripts/lib/wic/engine.py                         | 11 ++++--
 scripts/lib/wic/imager/direct.py                  | 23 ++++++++-----
 scripts/lib/wic/plugins/imager/direct_plugin.py   |  3 +-
 scripts/wic                                       |  3 +-
 9 files changed, 108 insertions(+), 14 deletions(-)
 create mode 100644 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb

-- 
2.1.4

[PATCH v3 1/6] bmap-tools: initial commit, version 3.2

[Ed Bartosh]

Bmap-tools - tools to generate block map (AKA bmap) and flash images
using bmap. Bmaptool is a generic tool for creating the block map
(bmap) for a file and copying files using the block map.

The idea is that large file containing unused blocks, like raw system
image files, can be copied or flashed a lot faster with bmaptool than
with traditional tools like "dd" or "cp".

[YOCTO #9414]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 24 +++++++++++++++++++++++
 1 file changed, 24 insertions(+)
 create mode 100644 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb

diff --git a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
new file mode 100644
index 0000000..e10f5fd
--- /dev/null
+++ b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
@@ -0,0 +1,24 @@
+SUMMARY = "Tools to generate block map (AKA bmap) and flash images using bmap"
+DESCRIPTION = "Bmap-tools - tools to generate block map (AKA bmap) and flash images using \
+bmap. Bmaptool is a generic tool for creating the block map (bmap) for a file, \
+and copying files using the block map. The idea is that large file containing \
+unused blocks, like raw system image files, can be copied or flashed a lot \
+faster with bmaptool than with traditional tools like "dd" or "cp"."
+HOMEPAGE = "http://git.infradead.org/users/dedekind/bmap-tools.git"
+SECTION = "console/utils"
+LICENSE = "GPLv2"
+LIC_FILES_CHKSUM = "file://COPYING;md5=b234ee4d69f5fce4486a80fdaf4a4263"
+
+SRC_URI = "ftp://ftp.infradead.org/pub/${BPN}/${BPN}-${PV}.tgz"
+SRC_URI[md5sum] = "92cdad1cb4dfa0cca7176c8e22752616"
+SRC_URI[sha256sum] = "cc6c7f7dc0a37e2a32deb127308e24e6c4b80bfb54f3803c308efab02bf2d434"
+
+RDEPENDS_${PN} = "python-core python-compression"
+
+inherit setuptools
+
+BBCLASSEXTEND = "native"
+
+do_install_append_class-native() {
+    sed -i -e 's|^#!.*/usr/bin/env python|#! /usr/bin/env nativepython|' ${D}${bindir}/bmaptool
+}
-- 
2.1.4

[PATCH v3 2/6] image types: add bmap generation option

[Ed Bartosh]

From: "Alexander D. Kanevskiy" <kad@kad.name>

bmap image conversion type allows to create block map files
for sparse images. Bmap file can be used together with bmap-tools
for efficiently flash images to raw devices (hdd or usb drive)

[YOCTO #9414]

Signed-off-by: Alexander D. Kanevskiy <kad@kad.name>
Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/classes/image_types.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index 53af7ca..19f89f3 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -262,7 +262,7 @@ IMAGE_TYPES = " \
     wic wic.gz wic.bz2 wic.lzma \
 "
 
-COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum"
+COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap"
 COMPRESS_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 COMPRESS_CMD_gz = "gzip -f -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
 COMPRESS_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
@@ -275,12 +275,14 @@ COMPRESS_CMD_sha224sum = "sha224sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} >
 COMPRESS_CMD_sha256sum = "sha256sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha256sum"
 COMPRESS_CMD_sha384sum = "sha384sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha384sum"
 COMPRESS_CMD_sha512sum = "sha512sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha512sum"
+COMPRESS_CMD_bmap = "bmaptool create ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.bmap"
 COMPRESS_DEPENDS_lzma = "xz-native"
 COMPRESS_DEPENDS_gz = ""
 COMPRESS_DEPENDS_bz2 = "pbzip2-native"
 COMPRESS_DEPENDS_xz = "xz-native"
 COMPRESS_DEPENDS_lz4 = "lz4-native"
 COMPRESS_DEPENDS_sum = "mtd-utils-native"
+COMPRESS_DEPENDS_bmap = "bmap-tools-native"
 
 RUNNABLE_IMAGE_TYPES ?= "ext2 ext3 ext4"
 RUNNABLE_MACHINE_PATTERNS ?= "qemu"
-- 
2.1.4

[PATCH v3 3/6] selftest: add bmap test

[Ed Bartosh]

Added test_bmap to imagefeatures tests.
It tests if bmap file is generated for the images and
if the image is sparse.

[YOCTO #9414]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/imagefeatures.py | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)

diff --git a/meta/lib/oeqa/selftest/imagefeatures.py b/meta/lib/oeqa/selftest/imagefeatures.py
index 8a53899..08e382f 100644
--- a/meta/lib/oeqa/selftest/imagefeatures.py
+++ b/meta/lib/oeqa/selftest/imagefeatures.py
@@ -98,3 +98,30 @@ class ImageFeatures(oeSelfTest):
         # Build a core-image-weston
         bitbake('core-image-weston')
 
+    def test_bmap(self):
+        """
+        Summary:     Check bmap support
+        Expected:    1. core-image-minimal can be build with bmap support
+                     2. core-image-minimal is sparse
+        Product:     oe-core
+        Author:      Ed Bartosh <ed.bartosh@linux.intel.com>
+        """
+
+        features = 'IMAGE_FSTYPES += " ext4 ext4.bmap"'
+        self.write_config(features)
+
+        image_name = 'core-image-minimal'
+        bitbake(image_name)
+
+        deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE')
+        link_name = get_bb_var('IMAGE_LINK_NAME', image_name)
+        image_path = os.path.join(deploy_dir, "%s.ext4" % link_name)
+        bmap_path = "%s.bmap" % image_path
+
+        # check if result image and bmap file are in deploy directory
+        self.assertTrue(os.path.exists(image_path))
+        self.assertTrue(os.path.exists(bmap_path))
+
+        # check if result image is sparse
+        image_stat = os.stat(image_path)
+        self.assertTrue(image_stat.st_size > image_stat.st_blocks * 512)
-- 
2.1.4

[PATCH v3 4/6] bmap-tools: generate standalone script

[Ed Bartosh]

Generated standalone bmaptool script that is more convenient
to use than native script. It can be run straight from
its location ./tmp/deploy/tools/bmaptool. The script doesn't
depend on anything except Python.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

diff --git a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
index e10f5fd..0f95788 100644
--- a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
+++ b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
@@ -15,10 +15,28 @@ SRC_URI[sha256sum] = "cc6c7f7dc0a37e2a32deb127308e24e6c4b80bfb54f3803c308efab02b
 
 RDEPENDS_${PN} = "python-core python-compression"
 
-inherit setuptools
+inherit setuptools deploy
 
 BBCLASSEXTEND = "native"
 
 do_install_append_class-native() {
     sed -i -e 's|^#!.*/usr/bin/env python|#! /usr/bin/env nativepython|' ${D}${bindir}/bmaptool
 }
+
+do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_TOOLS}"
+do_deploy[stamp-extra-info] = ""
+do_deploy_class-native() {
+    cp bmaptool __main__.py
+    python -m zipfile -c bmaptool.zip bmaptools __main__.py
+    echo '#!/usr/bin/env python' | cat - bmaptool.zip > bmaptool-standalone
+    install -d ${DEPLOYDIR}
+    install -m 0755 bmaptool-standalone ${DEPLOYDIR}/bmaptool-${PV}
+    rm -f ${DEPLOYDIR}/bmaptool
+    ln -sf ./bmaptool-${PV} ${DEPLOYDIR}/bmaptool
+}
+
+do_deploy() {
+    :
+}
+
+addtask deploy before do_package after do_install
-- 
2.1.4

Re: [PATCH v3 4/6] bmap-tools: generate standalone script

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 907 bytes --]

On Mon, May 2, 2016 at 12:22 PM, Ed Bartosh <ed.bartosh@linux.intel.com>
wrote:

> Generated standalone bmaptool script that is more convenient
> to use than native script. It can be run straight from
> its location ./tmp/deploy/tools/bmaptool. The script doesn't
> depend on anything except Python.
>
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
>

You're right that this is very handy, but we hit a problem when building
from shared state. This uses do_deploy to deploy bmaptool into
DEPLOY_DIR_IMAGE, but IMAGE_DEPENDS only depends on do_populate_sysroot,
not do_deploy, so an image build from sstate won't write this script out.

I don't know the right way to fix this, but it's a problem. I just ran into
it :)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1374 bytes --]

Re: [PATCH v3 4/6] bmap-tools: generate standalone script

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1268 bytes --]

On Tue, May 3, 2016 at 7:31 PM, Christopher Larson <clarson@kergoth.com>
wrote:

>
>
> On Mon, May 2, 2016 at 12:22 PM, Ed Bartosh <ed.bartosh@linux.intel.com>
> wrote:
>
>> Generated standalone bmaptool script that is more convenient
>> to use than native script. It can be run straight from
>> its location ./tmp/deploy/tools/bmaptool. The script doesn't
>> depend on anything except Python.
>>
>> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
>>
>
> You're right that this is very handy, but we hit a problem when building
> from shared state. This uses do_deploy to deploy bmaptool into
> DEPLOY_DIR_IMAGE, but IMAGE_DEPENDS only depends on do_populate_sysroot,
> not do_deploy, so an image build from sstate won't write this script out.
>
> I don't know the right way to fix this, but it's a problem. I just ran
> into it :)
>

Also, the convention is to add do_deploy before do_build, not before
do_package, otherwise do_deploy's checksum is included int he checksums of
do_package and subsequent tasks, even though its output isn't used by any
subsequent tasks.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2058 bytes --]

Re: [PATCH v3 4/6] bmap-tools: generate standalone script

[Ed Bartosh]

On Tue, May 03, 2016 at 07:33:58PM +0000, Christopher Larson wrote:
> On Tue, May 3, 2016 at 7:31 PM, Christopher Larson <clarson@kergoth.com>
> wrote:
> 
> >
> >
> > On Mon, May 2, 2016 at 12:22 PM, Ed Bartosh <ed.bartosh@linux.intel.com>
> > wrote:
> >
> >> Generated standalone bmaptool script that is more convenient
> >> to use than native script. It can be run straight from
> >> its location ./tmp/deploy/tools/bmaptool. The script doesn't
> >> depend on anything except Python.
> >>
> >> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> >>
> >
> > You're right that this is very handy, but we hit a problem when building
> > from shared state. This uses do_deploy to deploy bmaptool into
> > DEPLOY_DIR_IMAGE, but IMAGE_DEPENDS only depends on do_populate_sysroot,
> > not do_deploy, so an image build from sstate won't write this script out.
> >
> > I don't know the right way to fix this, but it's a problem. I just ran
> > into it :)
> >
> 
> Also, the convention is to add do_deploy before do_build, not before
> do_package, otherwise do_deploy's checksum is included int he checksums of
> do_package and subsequent tasks, even though its output isn't used by any
> subsequent tasks.

I also found this a bit unusual to put the tool into deploy/tools.
That's why I isolated this change. I think it requires some more
testing, but the rest of patchset can be accepted without it.

--
Regards,
Ed

Re: [PATCH v3 4/6] bmap-tools: generate standalone script

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2351 bytes --]

On Wed, May 4, 2016 at 7:01 AM, Ed Bartosh <ed.bartosh@linux.intel.com>
wrote:

> On Tue, May 03, 2016 at 07:33:58PM +0000, Christopher Larson wrote:
> > On Tue, May 3, 2016 at 7:31 PM, Christopher Larson <clarson@kergoth.com>
> > wrote:
> >
> > >
> > >
> > > On Mon, May 2, 2016 at 12:22 PM, Ed Bartosh <
> ed.bartosh@linux.intel.com>
> > > wrote:
> > >
> > >> Generated standalone bmaptool script that is more convenient
> > >> to use than native script. It can be run straight from
> > >> its location ./tmp/deploy/tools/bmaptool. The script doesn't
> > >> depend on anything except Python.
> > >>
> > >> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> > >>
> > >
> > > You're right that this is very handy, but we hit a problem when
> building
> > > from shared state. This uses do_deploy to deploy bmaptool into
> > > DEPLOY_DIR_IMAGE, but IMAGE_DEPENDS only depends on
> do_populate_sysroot,
> > > not do_deploy, so an image build from sstate won't write this script
> out.
> > >
> > > I don't know the right way to fix this, but it's a problem. I just ran
> > > into it :)
> > >
> >
> > Also, the convention is to add do_deploy before do_build, not before
> > do_package, otherwise do_deploy's checksum is included int he checksums
> of
> > do_package and subsequent tasks, even though its output isn't used by any
> > subsequent tasks.
>
> I also found this a bit unusual to put the tool into deploy/tools.
> That's why I isolated this change. I think it requires some more
> testing, but the rest of patchset can be accepted without it.


Yeah, putting it there does require some discussion in general. Personally,
I'm in favor of such a thing, because we don't want to put generated
convenience tools in the sysroot, since we tell people explicitly to not
use sysroot contents directly.

Regarding getting it built for the image, I'd suggest adding something like
IMAGE_TASKDEPENDS to image_type.bbclass, which is basically like
IMAGE_DEPENDS, but with the task included. I.e. IMAGE_TASKDEPENDS_bmap =
"bmap-tools-native:do_deploy". Then you wouldn't need the 'before
do_package', and it'd fix the from-sstate usage.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3175 bytes --]

Re: [PATCH v3 4/6] bmap-tools: generate standalone script

[Alexander Kanevskiy]

[-- Attachment #1: Type: text/plain, Size: 2762 bytes --]

On Wed, May 4, 2016 at 9:55 PM, Christopher Larson <clarson@kergoth.com>
wrote:

>
> >
>> > Also, the convention is to add do_deploy before do_build, not before
>> > do_package, otherwise do_deploy's checksum is included int he checksums
>> of
>> > do_package and subsequent tasks, even though its output isn't used by
>> any
>> > subsequent tasks.
>>
>> I also found this a bit unusual to put the tool into deploy/tools.
>> That's why I isolated this change. I think it requires some more
>> testing, but the rest of patchset can be accepted without it.
>
>
> Yeah, putting it there does require some discussion in general.
> Personally, I'm in favor of such a thing, because we don't want to put
> generated convenience tools in the sysroot, since we tell people explicitly
> to not use sysroot contents directly.
>
> Regarding getting it built for the image, I'd suggest adding something
> like IMAGE_TASKDEPENDS to image_type.bbclass, which is basically like
> IMAGE_DEPENDS, but with the task included. I.e. IMAGE_TASKDEPENDS_bmap =
> "bmap-tools-native:do_deploy". Then you wouldn't need the 'before
> do_package', and it'd fix the from-sstate usage.
>

We had some private conversation with Richard about deploy/tools, I'll try
to summarize results of our discussion here:

Thing this, that content under deploy/tools/* supposed to be exported to
"release" directory, and should not be used by any other tool during normal
build/development process.
Those are for usage on another host, where bitbake and content of sysroots
are not usually available.
bmaptool is a good example as stand-alone binary that require only python2
and nothing else to run.
Another example is dfu-util-static, which is also, not dependant even on
host's libusb even it runs on other host.

So, in general, it's for static/standalone binaries that are supposed to be
published as part of "release" of the build, same like deploy/images/*

For any classes, any internals of build process, they never should depend
on content of deploy/* and use *-native tools and binaries out of sysroot.
E.g. wic require bmap-tool, it should be dependant on presence of
bmap-tools-native installed into sysroot, and should be using both modules
and binary out of this sysroot.

This is the way how I'm using it inside Ostro Project:

description in our image type class:
https://github.com/ostroproject/meta-ostro/blob/master/meta-ostro/classes/image-dsk.bbclass#L32
https://github.com/ostroproject/meta-ostro/blob/master/meta-ostro/classes/image-dsk.bbclass#L131

actual  usage of bmaptools modules from sysroot:
https://github.com/ostroproject/meta-ostro/blob/master/meta-ostro/lib/image-dsk.py#L16


-- 
br, Alexander Kanevskiy

[-- Attachment #2: Type: text/html, Size: 4504 bytes --]

Re: [PATCH v3 4/6] bmap-tools: generate standalone script

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2845 bytes --]

On Thu, May 19, 2016 at 7:19 AM, Alexander Kanevskiy <kad@kad.name> wrote:

> On Wed, May 4, 2016 at 9:55 PM, Christopher Larson <clarson@kergoth.com>
> wrote:
>
>>
>> >
>>> > Also, the convention is to add do_deploy before do_build, not before
>>> > do_package, otherwise do_deploy's checksum is included int he
>>> checksums of
>>> > do_package and subsequent tasks, even though its output isn't used by
>>> any
>>> > subsequent tasks.
>>>
>>> I also found this a bit unusual to put the tool into deploy/tools.
>>> That's why I isolated this change. I think it requires some more
>>> testing, but the rest of patchset can be accepted without it.
>>
>>
>> Yeah, putting it there does require some discussion in general.
>> Personally, I'm in favor of such a thing, because we don't want to put
>> generated convenience tools in the sysroot, since we tell people explicitly
>> to not use sysroot contents directly.
>>
>> Regarding getting it built for the image, I'd suggest adding something
>> like IMAGE_TASKDEPENDS to image_type.bbclass, which is basically like
>> IMAGE_DEPENDS, but with the task included. I.e. IMAGE_TASKDEPENDS_bmap =
>> "bmap-tools-native:do_deploy". Then you wouldn't need the 'before
>> do_package', and it'd fix the from-sstate usage.
>>
>
> We had some private conversation with Richard about deploy/tools, I'll try
> to summarize results of our discussion here:
>
> Thing this, that content under deploy/tools/* supposed to be exported to
> "release" directory, and should not be used by any other tool during normal
> build/development process.
> Those are for usage on another host, where bitbake and content of sysroots
> are not usually available.
> bmaptool is a good example as stand-alone binary that require only python2
> and nothing else to run.
> Another example is dfu-util-static, which is also, not dependant even on
> host's libusb even it runs on other host.
>
> So, in general, it's for static/standalone binaries that are supposed to
> be published as part of "release" of the build, same like deploy/images/*
>
> For any classes, any internals of build process, they never should depend
> on content of deploy/* and use *-native tools and binaries out of sysroot.
> E.g. wic require bmap-tool, it should be dependant on presence of
> bmap-tools-native installed into sysroot, and should be using both modules
> and binary out of this sysroot.
>

I don't really understand the point you're trying to make here. As far as I
can tell, wic put bmaptool into deploy/tools/ as a convenience for the
user, not for wic itself to use, so it already does what you're saying it
should do.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 4153 bytes --]

Re: [PATCH v3 4/6] bmap-tools: generate standalone script

[Alexander Kanevskiy]

[-- Attachment #1: Type: text/plain, Size: 4325 bytes --]

On Thu, May 19, 2016 at 5:54 PM, Christopher Larson <clarson@kergoth.com>
wrote:

> On Thu, May 19, 2016 at 7:19 AM, Alexander Kanevskiy <kad@kad.name> wrote:
>
>> On Wed, May 4, 2016 at 9:55 PM, Christopher Larson <clarson@kergoth.com>
>> wrote:
>>
>>>
>>>> Yeah, putting it there does require some discussion in general.
>>> Personally, I'm in favor of such a thing, because we don't want to put
>>> generated convenience tools in the sysroot, since we tell people explicitly
>>> to not use sysroot contents directly.
>>>
>>> Regarding getting it built for the image, I'd suggest adding something
>>> like IMAGE_TASKDEPENDS to image_type.bbclass, which is basically like
>>> IMAGE_DEPENDS, but with the task included. I.e. IMAGE_TASKDEPENDS_bmap =
>>> "bmap-tools-native:do_deploy". Then you wouldn't need the 'before
>>> do_package', and it'd fix the from-sstate usage.
>>>
>>
>> We had some private conversation with Richard about deploy/tools, I'll
>> try to summarize results of our discussion here:
>>
>> Thing this, that content under deploy/tools/* supposed to be exported to
>> "release" directory, and should not be used by any other tool during normal
>> build/development process.
>> Those are for usage on another host, where bitbake and content of
>> sysroots are not usually available.
>> bmaptool is a good example as stand-alone binary that require only
>> python2 and nothing else to run.
>> Another example is dfu-util-static, which is also, not dependant even on
>> host's libusb even it runs on other host.
>>
>> So, in general, it's for static/standalone binaries that are supposed to
>> be published as part of "release" of the build, same like deploy/images/*
>>
>> For any classes, any internals of build process, they never should depend
>> on content of deploy/* and use *-native tools and binaries out of sysroot.
>> E.g. wic require bmap-tool, it should be dependant on presence of
>> bmap-tools-native installed into sysroot, and should be using both modules
>> and binary out of this sysroot.
>>
>
> I don't really understand the point you're trying to make here. As far as
> I can tell, wic put bmaptool into deploy/tools/ as a convenience for the
> user, not for wic itself to use, so it already does what you're saying it
> should do.
>

Yeah, to some extend I'm mixing two topics. Most of above is about putting
tools into depoloy/* (as Richard raised concern about that).


Second topic is about subject "which part of image or wic should depend on
which task". Both me and Ed are trying to understand exact scenario (or
even better test case) would trigger issue which lead to idea above of
putting dependencies like IMAGE_TASKDEPENDS_bmap =
"bmap-tools-native:do_deploy".

My initial idea,  while creating support for bmap-tools in Ostro, was that
image type would depend on bmap-tools-native.
This recipe is also responsible to handle deploy/*. Code or dependencies of
specific image type don't know anything about deploy part, and IMHO should
not know.
In case of support in OE-core, I had impression that Ed implemented it same
way. However, I didn't check myself part that he did in wic, so not sure
about part of your comment "wic put bmaptool into deploy/tools/".

I also tried in my environment to reproduce scenario as you described in
some previous mails in this thread:

> You're right that this is very handy, but we hit a problem when building
from shared state. This uses do_deploy to deploy bmaptool into
DEPLOY_DIR_IMAGE, but IMAGE_DEPENDS only depends > on do_populate_sysroot,
not do_deploy, so an image build from sstate won't write this script out.

but don't see issue with it: executing image build where all objects are
available in sstate, but bmap-tools-native not present in sysroot leads to
execution of following tasks:

$ cat tmp-glibc/work/x86_64-linux/bmap-tools-native/3.2/temp/log.task_order
do_populate_sysroot_setscene (29437): log.do_populate_sysroot_setscene.29437
do_deploy_setscene (29438): log.do_deploy_setscene.29438
do_populate_lic_setscene (29439): log.do_populate_lic_setscene.29439
$

if it for some reason not like that in code that Ed did for OE-core, we
need to look at test scenario, investigate and  fix the issue.

-- 
br, Alexander Kanevskiy

[-- Attachment #2: Type: text/html, Size: 6619 bytes --]

Re: [PATCH v3 4/6] bmap-tools: generate standalone script

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 4790 bytes --]

On Thu, May 19, 2016 at 2:50 PM, Alexander Kanevskiy <kad@kad.name> wrote:

> On Thu, May 19, 2016 at 5:54 PM, Christopher Larson <clarson@kergoth.com>
> wrote:
>
>> On Thu, May 19, 2016 at 7:19 AM, Alexander Kanevskiy <kad@kad.name>
>> wrote:
>>
>>> On Wed, May 4, 2016 at 9:55 PM, Christopher Larson <clarson@kergoth.com>
>>> wrote:
>>>
>>>>
>>>>> Yeah, putting it there does require some discussion in general.
>>>> Personally, I'm in favor of such a thing, because we don't want to put
>>>> generated convenience tools in the sysroot, since we tell people explicitly
>>>> to not use sysroot contents directly.
>>>>
>>>> Regarding getting it built for the image, I'd suggest adding something
>>>> like IMAGE_TASKDEPENDS to image_type.bbclass, which is basically like
>>>> IMAGE_DEPENDS, but with the task included. I.e. IMAGE_TASKDEPENDS_bmap =
>>>> "bmap-tools-native:do_deploy". Then you wouldn't need the 'before
>>>> do_package', and it'd fix the from-sstate usage.
>>>>
>>>
>>> We had some private conversation with Richard about deploy/tools, I'll
>>> try to summarize results of our discussion here:
>>>
>>> Thing this, that content under deploy/tools/* supposed to be exported to
>>> "release" directory, and should not be used by any other tool during normal
>>> build/development process.
>>> Those are for usage on another host, where bitbake and content of
>>> sysroots are not usually available.
>>> bmaptool is a good example as stand-alone binary that require only
>>> python2 and nothing else to run.
>>> Another example is dfu-util-static, which is also, not dependant even on
>>> host's libusb even it runs on other host.
>>>
>>> So, in general, it's for static/standalone binaries that are supposed to
>>> be published as part of "release" of the build, same like deploy/images/*
>>>
>>> For any classes, any internals of build process, they never should
>>> depend on content of deploy/* and use *-native tools and binaries out of
>>> sysroot.
>>> E.g. wic require bmap-tool, it should be dependant on presence of
>>> bmap-tools-native installed into sysroot, and should be using both modules
>>> and binary out of this sysroot.
>>>
>>
>> I don't really understand the point you're trying to make here. As far as
>> I can tell, wic put bmaptool into deploy/tools/ as a convenience for the
>> user, not for wic itself to use, so it already does what you're saying it
>> should do.
>>
>
> Yeah, to some extend I'm mixing two topics. Most of above is about putting
> tools into depoloy/* (as Richard raised concern about that).
>
>
> Second topic is about subject "which part of image or wic should depend on
> which task". Both me and Ed are trying to understand exact scenario (or
> even better test case) would trigger issue which lead to idea above of
> putting dependencies like IMAGE_TASKDEPENDS_bmap =
> "bmap-tools-native:do_deploy".
>
> My initial idea,  while creating support for bmap-tools in Ostro, was that
> image type would depend on bmap-tools-native.
> This recipe is also responsible to handle deploy/*. Code or dependencies
> of specific image type don't know anything about deploy part, and IMHO
> should not know.
> In case of support in OE-core, I had impression that Ed implemented it
> same way. However, I didn't check myself part that he did in wic, so not
> sure about part of your comment "wic put bmaptool into deploy/tools/".
>
> I also tried in my environment to reproduce scenario as you described in
> some previous mails in this thread:
>
> > You're right that this is very handy, but we hit a problem when building
> from shared state. This uses do_deploy to deploy bmaptool into
> DEPLOY_DIR_IMAGE, but IMAGE_DEPENDS only depends > on do_populate_sysroot,
> not do_deploy, so an image build from sstate won't write this script out.
>
> but don't see issue with it: executing image build where all objects are
> available in sstate, but bmap-tools-native not present in sysroot leads to
> execution of following tasks:
>
> $ cat tmp-glibc/work/x86_64-linux/bmap-tools-native/3.2/temp/log.task_order
> do_populate_sysroot_setscene (29437):
> log.do_populate_sysroot_setscene.29437
> do_deploy_setscene (29438): log.do_deploy_setscene.29438
> do_populate_lic_setscene (29439): log.do_populate_lic_setscene.29439
> $
>
> if it for some reason not like that in code that Ed did for OE-core, we
> need to look at test scenario, investigate and  fix the issue.
>

For what it's worth, I'm no longer able to reproduce this. I did hit it at
one point, but it seems fine now. *shrug*
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 7445 bytes --]

Re: [PATCH v3 4/6] bmap-tools: generate standalone script

[Richard Purdie]

On Mon, 2016-05-02 at 15:22 +0300, Ed Bartosh wrote:
> Generated standalone bmaptool script that is more convenient
> to use than native script. It can be run straight from
> its location ./tmp/deploy/tools/bmaptool. The script doesn't
> depend on anything except Python.
> 
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  meta/recipes-support/bmap-tools/bmap-tools_3.2.bb | 20
> +++++++++++++++++++-
>  1 file changed, 19 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
> b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
> index e10f5fd..0f95788 100644
> --- a/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
> +++ b/meta/recipes-support/bmap-tools/bmap-tools_3.2.bb
> @@ -15,10 +15,28 @@ SRC_URI[sha256sum] =
> "cc6c7f7dc0a37e2a32deb127308e24e6c4b80bfb54f3803c308efab02b
>  
>  RDEPENDS_${PN} = "python-core python-compression"
>  
> -inherit setuptools
> +inherit setuptools deploy
>  
>  BBCLASSEXTEND = "native"
>  
>  do_install_append_class-native() {
>      sed -i -e 's|^#!.*/usr/bin/env python|#! /usr/bin/env
> nativepython|' ${D}${bindir}/bmaptool
>  }
> +
> +do_deploy[sstate-outputdirs] = "${DEPLOY_DIR_TOOLS}"
> +do_deploy[stamp-extra-info] = ""
> +do_deploy_class-native() {
> +    cp bmaptool __main__.py
> +    python -m zipfile -c bmaptool.zip bmaptools __main__.py
> +    echo '#!/usr/bin/env python' | cat - bmaptool.zip > bmaptool
> -standalone
> +    install -d ${DEPLOYDIR}
> +    install -m 0755 bmaptool-standalone ${DEPLOYDIR}/bmaptool-${PV}
> +    rm -f ${DEPLOYDIR}/bmaptool
> +    ln -sf ./bmaptool-${PV} ${DEPLOYDIR}/bmaptool
> +}
> +
> +do_deploy() {
> +    :
> +}
> +
> +addtask deploy before do_package after do_install

I'm not sure if we want to do this in the first place. Regardless, the
addtask here is incorrect too.

Cheers,

Richard

[wic][PATCH 1/4] wic: add bmaptool to the list of utilities

[Ed Bartosh]

Added bmaptool -> bmap-tools pair to the dictionary
executable -> recipe as bmaptool is going to be used by wic
to generate .bmap file.

[YOCTO #9413]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/utils/oe/misc.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/utils/oe/misc.py b/scripts/lib/wic/utils/oe/misc.py
index 0854dfb..fe188c9 100644
--- a/scripts/lib/wic/utils/oe/misc.py
+++ b/scripts/lib/wic/utils/oe/misc.py
@@ -34,7 +34,8 @@ from wic import msger
 from wic.utils import runner
 
 # executable -> recipe pairs for exec_native_cmd
-NATIVE_RECIPES = {"mcopy": "mtools",
+NATIVE_RECIPES = {"bmaptool": "bmap-tools",
+                  "mcopy": "mtools",
                   "mkdosfs": "dosfstools",
                   "mkfs.btrfs": "btrfs-tools",
                   "mkfs.ext2": "e2fsprogs",
-- 
2.1.4

[wic][PATCH 2/4] wic: implement --bmap option

[Ed Bartosh]

This option enables generation of <image>.bmap file for the
result image using native bmaptool.

[YOCTO #9413]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/creator.py                      |  1 +
 scripts/lib/wic/engine.py                       | 11 ++++++++---
 scripts/lib/wic/imager/direct.py                | 23 +++++++++++++++--------
 scripts/lib/wic/plugins/imager/direct_plugin.py |  3 ++-
 scripts/wic                                     |  3 ++-
 5 files changed, 28 insertions(+), 13 deletions(-)

diff --git a/scripts/lib/wic/creator.py b/scripts/lib/wic/creator.py
index d4972e8..8f7d150 100644
--- a/scripts/lib/wic/creator.py
+++ b/scripts/lib/wic/creator.py
@@ -69,6 +69,7 @@ class Creator():
         optparser.add_option('', '--tmpfs', action='store_true', dest='enabletmpfs',
                              help='Setup tmpdir as tmpfs to accelerate, experimental'
                                   ' feature, use it if you have more than 4G memory')
+        optparser.add_option('', '--bmap', action='store_true', help='generate .bmap')
         return optparser
 
     def postoptparse(self, options):
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 5d35c46..5b10463 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -145,7 +145,7 @@ def list_source_plugins():
 
 def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
                native_sysroot, scripts_path, image_output_dir,
-               compressor, debug):
+               compressor, bmap, debug):
     """Create image
 
     wks_file - user-defined OE kickstart file
@@ -156,6 +156,7 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
     scripts_path - absolute path to /scripts dir
     image_output_dir - dirname to create for image
     compressor - compressor utility to compress the image
+    bmap - enable generation of .bmap
 
     Normally, the values for the build artifacts values are determined
     by 'wic -e' from the output of the 'bitbake -e' command given an
@@ -186,8 +187,12 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
 
     crobj = creator.Creator()
 
-    crobj.main(["direct", native_sysroot, kernel_dir, bootimg_dir, rootfs_dir,
-                wks_file, image_output_dir, oe_builddir, compressor or ""])
+    cmdline = ["direct", native_sysroot, kernel_dir, bootimg_dir, rootfs_dir,
+                wks_file, image_output_dir, oe_builddir, compressor or ""]
+    if bmap:
+        cmdline.append('--bmap')
+
+    crobj.main(cmdline)
 
     print("\nThe image(s) were created using OE kickstart file:\n  %s" % wks_file)
 
diff --git a/scripts/lib/wic/imager/direct.py b/scripts/lib/wic/imager/direct.py
index 5a3b655..ffde232 100644
--- a/scripts/lib/wic/imager/direct.py
+++ b/scripts/lib/wic/imager/direct.py
@@ -33,7 +33,7 @@ from wic.utils.partitionedfs import Image
 from wic.utils.errors import CreatorError, ImageError
 from wic.imager.baseimager import BaseImageCreator
 from wic.plugin import pluginmgr
-from wic.utils.oe.misc import exec_cmd
+from wic.utils.oe.misc import exec_cmd, exec_native_cmd
 
 disk_methods = {
     "do_install_disk":None,
@@ -71,7 +71,8 @@ class DirectImageCreator(BaseImageCreator):
     """
 
     def __init__(self, oe_builddir, image_output_dir, rootfs_dir, bootimg_dir,
-                 kernel_dir, native_sysroot, compressor, creatoropts=None):
+                 kernel_dir, native_sysroot, compressor, creatoropts=None,
+                 bmap=False):
         """
         Initialize a DirectImageCreator instance.
 
@@ -93,6 +94,7 @@ class DirectImageCreator(BaseImageCreator):
         self.kernel_dir = kernel_dir
         self.native_sysroot = native_sysroot
         self.compressor = compressor
+        self.bmap = bmap
 
     def __get_part_num(self, num, parts):
         """calculate the real partition number, accounting for partitions not
@@ -333,12 +335,17 @@ class DirectImageCreator(BaseImageCreator):
                                                         self.bootimg_dir,
                                                         self.kernel_dir,
                                                         self.native_sysroot)
-        # Compress the image
-        if self.compressor:
-            for disk_name, disk in self.__image.disks.items():
-                full_path = self._full_path(self.__imgdir, disk_name, "direct")
-                msger.debug("Compressing disk %s with %s" % \
-                            (disk_name, self.compressor))
+
+        for disk_name, disk in self.__image.disks.items():
+            full_path = self._full_path(self.__imgdir, disk_name, "direct")
+            # Generate .bmap
+            if self.bmap:
+                msger.debug("Generating bmap file for %s" % disk_name)
+                exec_native_cmd("bmaptool create %s -o %s.bmap" % (full_path, full_path),
+                                self.native_sysroot)
+            # Compress the image
+            if self.compressor:
+                msger.debug("Compressing disk %s with %s" % (disk_name, self.compressor))
                 exec_cmd("%s %s" % (self.compressor, full_path))
 
     def print_outimage_info(self):
diff --git a/scripts/lib/wic/plugins/imager/direct_plugin.py b/scripts/lib/wic/plugins/imager/direct_plugin.py
index 6d3f46c..8fe3930 100644
--- a/scripts/lib/wic/plugins/imager/direct_plugin.py
+++ b/scripts/lib/wic/plugins/imager/direct_plugin.py
@@ -86,7 +86,8 @@ class DirectPlugin(ImagerPlugin):
                                             kernel_dir,
                                             native_sysroot,
                                             compressor,
-                                            creatoropts)
+                                            creatoropts,
+                                            opts.bmap)
 
         try:
             creator.create()
diff --git a/scripts/wic b/scripts/wic
index 11c8316..3d33430 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -115,6 +115,7 @@ def wic_create_subcommand(args, usage_str):
     parser.add_option("-c", "--compress-with", choices=("gzip", "bzip2", "xz"),
                       dest='compressor',
                       help="compress image with specified compressor")
+    parser.add_option("-m", "--bmap", action="store_true", help="generate .bmap")
     parser.add_option("-v", "--vars", dest='vars_dir',
                       help="directory with <image>.env files that store "
                            "bitbake variables")
@@ -245,7 +246,7 @@ def wic_create_subcommand(args, usage_str):
     print("Creating image(s)...\n")
     engine.wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
                       native_sysroot, scripts_path, image_output_dir,
-                      options.compressor, options.debug)
+                      options.compressor, options.bmap, options.debug)
 
 
 def wic_list_subcommand(args, usage_str):
-- 
2.1.4

[wic][PATCH 3/4] wic: add help for --bmap commandline option

[Ed Bartosh]

Included description of -m/--bmap option to the help page
produced by 'wic help create'.

[YOCTO #9413]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/help.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/scripts/lib/wic/help.py b/scripts/lib/wic/help.py
index 7dcc717..6395596 100644
--- a/scripts/lib/wic/help.py
+++ b/scripts/lib/wic/help.py
@@ -152,7 +152,7 @@ SYNOPSIS
         [-e | --image-name] [-s, --skip-build-check] [-D, --debug]
         [-r, --rootfs-dir] [-b, --bootimg-dir]
         [-k, --kernel-dir] [-n, --native-sysroot] [-f, --build-rootfs]
-        [-c, --compress-with]
+        [-c, --compress-with] [-m, --bmap]
 
 DESCRIPTION
     This command creates an OpenEmbedded image based on the 'OE
@@ -221,6 +221,9 @@ DESCRIPTION
 
     The -c option is used to specify compressor utility to compress
     an image. gzip, bzip2 and xz compressors are supported.
+
+    The -m option is used to produce .bmap file for the image. This file
+    can be used to flash image using bmaptool utility.
 """
 
 wic_list_usage = """
-- 
2.1.4

[wic][PATCH 4/4] oe-selftest: wic: add test_bmap test case

[Ed Bartosh]

Tested generation of .bmap file using wic --bmap command
line option.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index a569fbf..895c6d6 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -276,3 +276,11 @@ class Wic(oeSelfTest):
             status, output = qemu.run_serial(command)
             self.assertEqual(1, status, 'Failed to run command "%s": %s' % (command, output))
             self.assertEqual(output, '/dev/root /\r\n/dev/vda3 /mnt')
+
+    def test_bmap(self):
+        """Test generation of .bmap file"""
+        image = "directdisk"
+        status = runCmd("wic create %s -e core-image-minimal --bmap" % image).status
+        self.assertEqual(0, status)
+        self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
+        self.assertEqual(1, len(glob(self.resultdir + "%s-*direct.bmap" % image)))
-- 
2.1.4

[PATCH v3 5/6] wic: implement --bmap option

[Ed Bartosh]

This option enables generation of <image>.bmap file for the
result image using native bmaptool.

[YOCTO #9413]

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 scripts/lib/wic/creator.py                      |  1 +
 scripts/lib/wic/engine.py                       | 11 ++++++++---
 scripts/lib/wic/imager/direct.py                | 23 +++++++++++++++--------
 scripts/lib/wic/plugins/imager/direct_plugin.py |  3 ++-
 scripts/wic                                     |  3 ++-
 5 files changed, 28 insertions(+), 13 deletions(-)

diff --git a/scripts/lib/wic/creator.py b/scripts/lib/wic/creator.py
index 5231297..6f4af8d 100644
--- a/scripts/lib/wic/creator.py
+++ b/scripts/lib/wic/creator.py
@@ -69,6 +69,7 @@ class Creator(object):
         optparser.add_option('', '--tmpfs', action='store_true', dest='enabletmpfs',
                              help='Setup tmpdir as tmpfs to accelerate, experimental'
                                   ' feature, use it if you have more than 4G memory')
+        optparser.add_option('', '--bmap', action='store_true', help='generate .bmap')
         return optparser
 
     def postoptparse(self, options):
diff --git a/scripts/lib/wic/engine.py b/scripts/lib/wic/engine.py
index 76b93e8..9af6eb4 100644
--- a/scripts/lib/wic/engine.py
+++ b/scripts/lib/wic/engine.py
@@ -145,7 +145,7 @@ def list_source_plugins():
 
 def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
                native_sysroot, scripts_path, image_output_dir,
-               compressor, debug):
+               compressor, bmap, debug):
     """Create image
 
     wks_file - user-defined OE kickstart file
@@ -156,6 +156,7 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
     scripts_path - absolute path to /scripts dir
     image_output_dir - dirname to create for image
     compressor - compressor utility to compress the image
+    bmap - enable generation of .bmap
 
     Normally, the values for the build artifacts values are determined
     by 'wic -e' from the output of the 'bitbake -e' command given an
@@ -186,8 +187,12 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
 
     crobj = creator.Creator()
 
-    crobj.main(["direct", native_sysroot, kernel_dir, bootimg_dir, rootfs_dir,
-                wks_file, image_output_dir, oe_builddir, compressor or ""])
+    cmdline = ["direct", native_sysroot, kernel_dir, bootimg_dir, rootfs_dir,
+                wks_file, image_output_dir, oe_builddir, compressor or ""]
+    if bmap:
+        cmdline.append('--bmap')
+
+    crobj.main(cmdline)
 
     print "\nThe image(s) were created using OE kickstart file:\n  %s" % wks_file
 
diff --git a/scripts/lib/wic/imager/direct.py b/scripts/lib/wic/imager/direct.py
index 1937042..1b6272e 100644
--- a/scripts/lib/wic/imager/direct.py
+++ b/scripts/lib/wic/imager/direct.py
@@ -34,7 +34,7 @@ from wic.utils.partitionedfs import Image
 from wic.utils.errors import CreatorError, ImageError
 from wic.imager.baseimager import BaseImageCreator
 from wic.plugin import pluginmgr
-from wic.utils.oe.misc import exec_cmd
+from wic.utils.oe.misc import exec_cmd, exec_native_cmd
 
 disk_methods = {
     "do_install_disk":None,
@@ -52,7 +52,8 @@ class DirectImageCreator(BaseImageCreator):
     """
 
     def __init__(self, oe_builddir, image_output_dir, rootfs_dir, bootimg_dir,
-                 kernel_dir, native_sysroot, compressor, creatoropts=None):
+                 kernel_dir, native_sysroot, compressor, creatoropts=None,
+                 bmap=False):
         """
         Initialize a DirectImageCreator instance.
 
@@ -74,6 +75,7 @@ class DirectImageCreator(BaseImageCreator):
         self.kernel_dir = kernel_dir
         self.native_sysroot = native_sysroot
         self.compressor = compressor
+        self.bmap = bmap
 
     def __get_part_num(self, num, parts):
         """calculate the real partition number, accounting for partitions not
@@ -314,12 +316,17 @@ class DirectImageCreator(BaseImageCreator):
                                                         self.bootimg_dir,
                                                         self.kernel_dir,
                                                         self.native_sysroot)
-        # Compress the image
-        if self.compressor:
-            for disk_name, disk in self.__image.disks.items():
-                full_path = self._full_path(self.__imgdir, disk_name, "direct")
-                msger.debug("Compressing disk %s with %s" % \
-                            (disk_name, self.compressor))
+
+        for disk_name, disk in self.__image.disks.items():
+            full_path = self._full_path(self.__imgdir, disk_name, "direct")
+            # Generate .bmap
+            if self.bmap:
+                msger.debug("Generating bmap file for %s" % disk_name)
+                exec_native_cmd("bmaptool create %s -o %s.bmap" % (full_path, full_path),
+                                self.native_sysroot)
+            # Compress the image
+            if self.compressor:
+                msger.debug("Compressing disk %s with %s" % (disk_name, self.compressor))
                 exec_cmd("%s %s" % (self.compressor, full_path))
 
     def print_outimage_info(self):
diff --git a/scripts/lib/wic/plugins/imager/direct_plugin.py b/scripts/lib/wic/plugins/imager/direct_plugin.py
index 6d3f46c..8fe3930 100644
--- a/scripts/lib/wic/plugins/imager/direct_plugin.py
+++ b/scripts/lib/wic/plugins/imager/direct_plugin.py
@@ -86,7 +86,8 @@ class DirectPlugin(ImagerPlugin):
                                             kernel_dir,
                                             native_sysroot,
                                             compressor,
-                                            creatoropts)
+                                            creatoropts,
+                                            opts.bmap)
 
         try:
             creator.create()
diff --git a/scripts/wic b/scripts/wic
index 2286f20..c4767ce 100755
--- a/scripts/wic
+++ b/scripts/wic
@@ -114,6 +114,7 @@ def wic_create_subcommand(args, usage_str):
     parser.add_option("-c", "--compress-with", choices=("gzip", "bzip2", "xz"),
                       dest='compressor',
                       help="compress image with specified compressor")
+    parser.add_option("-m", "--bmap", action="store_true", help="generate .bmap")
     parser.add_option("-v", "--vars", dest='vars_dir',
                       help="directory with <image>.env files that store "
                            "bitbake variables")
@@ -244,7 +245,7 @@ def wic_create_subcommand(args, usage_str):
     print "Creating image(s)...\n"
     engine.wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
                       native_sysroot, scripts_path, image_output_dir,
-                      options.compressor, options.debug)
+                      options.compressor, options.bmap, options.debug)
 
 
 def wic_list_subcommand(args, usage_str):
-- 
2.1.4

[PATCH v3 6/6] oe-selftest: wic: add test_bmap test case

[Ed Bartosh]

Tested generation of .bmap file using wic --bmap command
line option.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 meta/lib/oeqa/selftest/wic.py | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/meta/lib/oeqa/selftest/wic.py b/meta/lib/oeqa/selftest/wic.py
index a569fbf..895c6d6 100644
--- a/meta/lib/oeqa/selftest/wic.py
+++ b/meta/lib/oeqa/selftest/wic.py
@@ -276,3 +276,11 @@ class Wic(oeSelfTest):
             status, output = qemu.run_serial(command)
             self.assertEqual(1, status, 'Failed to run command "%s": %s' % (command, output))
             self.assertEqual(output, '/dev/root /\r\n/dev/vda3 /mnt')
+
+    def test_bmap(self):
+        """Test generation of .bmap file"""
+        image = "directdisk"
+        status = runCmd("wic create %s -e core-image-minimal --bmap" % image).status
+        self.assertEqual(0, status)
+        self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
+        self.assertEqual(1, len(glob(self.resultdir + "%s-*direct.bmap" % image)))
-- 
2.1.4

Re: [PATCH v3 0/6] bmaptool support

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2790 bytes --]

On Mon, May 2, 2016 at 12:22 PM, Ed Bartosh <ed.bartosh@linux.intel.com>
wrote:

> Hi,
>
> This patchset adds ability to use bmaptool to flash images faster than
> using traditional methods. It contains bmap-tools recipe and generation
> of .bmap files for the images generated by bitbake and wic.
>
> Bmaptool is a generic tool for creating the block map (bmap) for a file
> and and copying files using the block map. The idea is that large file
> containing unused blocks, like raw system image files, can be copied or
> flashed a lot faster with bmaptool than with traditional tools like
> "dd" or "cp".
>
> Here is an example of flashing image using dd and bmaptool showing
> around 25% performance gain in flashing time:
>
> > ls -slh
>  19M -rw-r--r-- 1 ed ed  26M Apr 26 11:36
> core-image-minimal-qemux86-64-20160426083554.rootfs.ext4
> 4.0K -rw-r--r-- 1 ed ed 2.7K Apr 26 11:36
> core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap
>
> > time dd if=core-image-minimal-qemux86-64-20160426083554.rootfs.ext4
> of=/dev/sdb
> 52302+0 records in
> 52302+0 records out
> 26778624 bytes (27 MB) copied, 2.51183 s, 10.7 MB/s
>
> real    0m2.515s
> user    0m0.014s
> sys     0m0.329s
>
> > time bmaptool copy
> core-image-minimal-qemux86-64-20160426083554.rootfs.ext4 /dev/sdb
> bmaptool: info: discovered bmap file
> 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap'
> bmaptool: info: block map format version 2.0
> bmaptool: info: 6538 blocks of size 4096 (25.5 MiB), mapped 4799 blocks
> (18.7 MiB or 73.4%)
> bmaptool: info: copying image
> 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4' to block device
> '/dev/sdb' using bmap file
> 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap'
> bmaptool: info: 100% copied
> bmaptool: info: synchronizing '/dev/sdb'
> bmaptool: info: copying time: 1.8s, copying speed 10.3 MiB/sec
>
> real    0m1.909s
> user    0m0.226s
> sys     0m0.052s
>
> The test was repeated 5 times with the same image and the same usb stick
> device.
>
> Changes in v2: Combined Alexander's and my bmap-tools recipes.
>                Got rid of generating standalone script as it breaks build
> of bmap-tools-native.
>
> Changes in v3: Returned back generation of standalone bmaptool script
>                Implemented --bmap option for wic
>                Added test case to test bmap generation by wic --bmap


I just wanted to say, this + the sparseness series is just awesome. Such a
timesaver in my test cycle, particularly on my old macbook which has a usb
2.0 sd card reader :)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3338 bytes --]

Re: [PATCH v3 0/6] bmaptool support

[Ed Bartosh]

On Tue, May 03, 2016 at 08:22:48PM +0000, Christopher Larson wrote:
> On Mon, May 2, 2016 at 12:22 PM, Ed Bartosh <ed.bartosh@linux.intel.com>
> wrote:
> 
> > Hi,
> >
> > This patchset adds ability to use bmaptool to flash images faster than
> > using traditional methods. It contains bmap-tools recipe and generation
> > of .bmap files for the images generated by bitbake and wic.
> >
> > Bmaptool is a generic tool for creating the block map (bmap) for a file
> > and and copying files using the block map. The idea is that large file
> > containing unused blocks, like raw system image files, can be copied or
> > flashed a lot faster with bmaptool than with traditional tools like
> > "dd" or "cp".
> >
> > Here is an example of flashing image using dd and bmaptool showing
> > around 25% performance gain in flashing time:
> >
> > > ls -slh
> >  19M -rw-r--r-- 1 ed ed  26M Apr 26 11:36
> > core-image-minimal-qemux86-64-20160426083554.rootfs.ext4
> > 4.0K -rw-r--r-- 1 ed ed 2.7K Apr 26 11:36
> > core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap
> >
> > > time dd if=core-image-minimal-qemux86-64-20160426083554.rootfs.ext4
> > of=/dev/sdb
> > 52302+0 records in
> > 52302+0 records out
> > 26778624 bytes (27 MB) copied, 2.51183 s, 10.7 MB/s
> >
> > real    0m2.515s
> > user    0m0.014s
> > sys     0m0.329s
> >
> > > time bmaptool copy
> > core-image-minimal-qemux86-64-20160426083554.rootfs.ext4 /dev/sdb
> > bmaptool: info: discovered bmap file
> > 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap'
> > bmaptool: info: block map format version 2.0
> > bmaptool: info: 6538 blocks of size 4096 (25.5 MiB), mapped 4799 blocks
> > (18.7 MiB or 73.4%)
> > bmaptool: info: copying image
> > 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4' to block device
> > '/dev/sdb' using bmap file
> > 'core-image-minimal-qemux86-64-20160426083554.rootfs.ext4.bmap'
> > bmaptool: info: 100% copied
> > bmaptool: info: synchronizing '/dev/sdb'
> > bmaptool: info: copying time: 1.8s, copying speed 10.3 MiB/sec
> >
> > real    0m1.909s
> > user    0m0.226s
> > sys     0m0.052s
> >
> > The test was repeated 5 times with the same image and the same usb stick
> > device.
> >
> > Changes in v2: Combined Alexander's and my bmap-tools recipes.
> >                Got rid of generating standalone script as it breaks build
> > of bmap-tools-native.
> >
> > Changes in v3: Returned back generation of standalone bmaptool script
> >                Implemented --bmap option for wic
> >                Added test case to test bmap generation by wic --bmap
> 
> 
> I just wanted to say, this + the sparseness series is just awesome. Such a
> timesaver in my test cycle, particularly on my old macbook which has a usb
> 2.0 sd card reader :)

Thank you! I also think bmaptool is awesome utility. It can save lots of
flushing time especially for the 'very sparse' images.

I had this work in my TODO for quite long time. Now, thanks to Alexander who did
almost all hard work, it's eventually coming to oe/yocto.

--
Regards,
Ed

Re: [PATCH v3 0/6] bmaptool support

[Richard Purdie]

On Mon, 2016-05-02 at 15:22 +0300, Ed Bartosh wrote:
> This patchset adds ability to use bmaptool to flash images faster
> than
> using traditional methods. It contains bmap-tools recipe and
> generation
> of .bmap files for the images generated by bitbake and wic.

I'm happy to merge this assuming it passes testing. Could you resend
the two wic changes against master-next please though (which is after
the wic python 3 changes). I resolved some of the earlier conflicts but
those were getting too involved.

Cheers,

Richard