[PATCH 0/2] V2 Incremental image generation(rpm based rootfs)

[PATCH 0/2] V2 Incremental image generation(rpm based rootfs)

[Robert Yang]

Changes of V2:

* Move the config sample from meta-yocto/conf/local.conf.sample to
  meta-yocto/conf/local.conf.sample.extended as Saul suggested.

Testing:
1) Enable INC_IMAGE_GEN = "1" in conf/local.conf
2) $ bitbake core-image-sato

   * Test when remove some pkgs
     a) Save the modify time of file which is in
        tmp/work/qemux86_64-poky-linux/core-image-sato-1.0-r0/rootfs/dev/.
     b) Edit meta/recipes-sato/images/core-image-sato.bb, remove
        ${SATO_IMAGE_FEATURES} from IMAGE_FEATURES.
     c) bitbake core-image-sato

     Result: The project built well, check the modify time of the file
     which is in
     tmp/work/qemux86_64-poky-linux/core-image-sato-1.0-r0/rootfs/dev/,
     They should be the same to step a)'s, which means that these files
     were note newly created.

   * Testh when add some pkgs
     a) Save the modify time of file which is in
        tmp/work/qemux86_64-poky-linux/core-image-sato-1.0-r0/rootfs/dev/.
     b) Edit meta/recipes-sato/images/core-image-sato.bb, add the
        ${SATO_IMAGE_FEATURES} back to IMAGE_FEATURES.
     c) bitbake core-image-sato

     Result: The project built well, check the modify time of the file
     which is in
     tmp/work/qemux86_64-poky-linux/core-image-sato-1.0-r0/rootfs/dev/,
     They should be the same to step a)'s, which means that these files
     were note newly created.

   * Test when edit a pkg
     a) bitbake bzip2 -cclean
     b) rm -f rm -f sstate-cache/sstate-bzip2-*
     c) Edit meta/recipes-extended/bzip2/bzip2_1.0.6.bb
     d) bitbake core-image-sato
     Result: The project built well, and check:
        tmp/work/qemux86-poky-linux/core-image-sato-1.0-r0/temp/log.do_rootfs
     Only the bzip2 has been re-installed

// Robert

The following changes since commit f5aa3bbda623c8fae3a761d72fddc95631ad0706:

  coreutils: ensure --color works so DEPEND on libcap (2011-12-24 10:05:47 +0000)

are available in the git repository at:
  git://git.pokylinux.org/poky-contrib robert/inc_image_gen
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/inc_image_gen

Robert Yang (2):
  Incremental image generation(rpm based rootfs)
  Incremental image generation(Add config sample)

 meta-yocto/conf/local.conf.sample.extended |    7 +++
 meta/classes/image.bbclass                 |   11 ++++-
 meta/classes/package_rpm.bbclass           |   72 ++++++++++++++++++++++++----
 3 files changed, 78 insertions(+), 12 deletions(-)

-- 
1.7.4.1

[PATCH 1/2] V2 Incremental image generation(rpm based rootfs)

[Robert Yang]

Incremental image generation, the rootfs would be totally removed and
re-created in the second generation by default, but with INC_IMAGE_GEN =
"1", the rootfs would be kept, and will do update(remove/add some pkgs)
on it.
NOTE: This is not suggested when you want to create a productive rootfs

For example:
  1) Add the follow config option to a conf file:
     INC_IMAGE_GEN = "1"

  2) bitbake core-image-sato
     modify a package
     bitbake core-image-sato

The rootfs would not be totally removed and re-created in the second
generation, it would be simply updated based on the "package".

Implatation:
1) Figure out the pkg which need to be removed or re-installed, then use
'rpm -e to remove the old one. Use the rpm's BUILDTIME to determine
which pkg has been rebuilt.

2) Figure out the pkg which is newly added, and use 'rpm -U' to install
it.

This only for the rpm based rootfs, I don't know whether we also need
this for ipk or deb  based rootfs.

[YOCTO #1651]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/image.bbclass       |   11 +++++-
 meta/classes/package_rpm.bbclass |   72 ++++++++++++++++++++++++++++++++-----
 2 files changed, 71 insertions(+), 12 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 865d430..71b60f3 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -134,15 +134,22 @@ do_rootfs[umask] = 022
 
 fakeroot do_rootfs () {
 	#set -x
-	rm -rf ${IMAGE_ROOTFS}
+    # When use the incremental image generation, don't remove the rootfs
+    if [ "${INC_IMAGE_GEN}" != "1" ]; then
+        rm -rf ${IMAGE_ROOTFS}
+    fi
 	rm -rf ${MULTILIB_TEMP_ROOTFS}
 	mkdir -p ${IMAGE_ROOTFS}
 	mkdir -p ${DEPLOY_DIR_IMAGE}
 
 	cp ${COREBASE}/meta/files/deploydir_readme.txt ${DEPLOY_DIR_IMAGE}/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt
 
-	if [ "${USE_DEVFS}" != "1" ]; then
+    # If "${IMAGE_ROOTFS}/dev" exists, then the device had been made by
+    # the previous build
+	if [ "${USE_DEVFS}" != "1" -a ! -r "${IMAGE_ROOTFS}/dev" ]; then
 		for devtable in ${@get_devtable_list(d)}; do
+            # Always return ture since there maybe already one when use the
+            # incremental image generation
 			makedevs -r ${IMAGE_ROOTFS} -D $devtable
 		done
 	fi
diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
index d03dc3f..541d351 100644
--- a/meta/classes/package_rpm.bbclass
+++ b/meta/classes/package_rpm.bbclass
@@ -147,6 +147,66 @@ resolve_package_rpm () {
 	echo $pkg_name
 }
 
+# rpm common command and options
+rpm_common_comand () {
+
+    local target_rootfs="${INSTALL_ROOTFS_RPM}"
+    local extra_args="$@"
+
+    ${RPM} --root ${target_rootfs} \
+        --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
+        --predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \
+        -D "_var ${localstatedir}" \
+        -D "_dbpath ${rpmlibdir}" \
+        --noparentdirs --nolinktos \
+        -D "__dbi_txn create nofsync private" \
+        -D "_cross_scriptlet_wrapper ${WORKDIR}/scriptlet_wrapper" $extra_args
+}
+
+# install or remove the pkg
+rpm_update_pkg () {
+
+    local target_rootfs="${INSTALL_ROOTFS_RPM}"
+
+    # Save the rpm's build time for incremental image generation, and the file
+    # would be moved to ${T}
+    rm -f ${target_rootfs}/install/total_solution_bt.manifest
+    for i in `cat ${target_rootfs}/install/total_solution.manifest`; do
+        # Use "rpm" rather than "${RPM}" here, since we don't need the
+        # '--dbpath' option
+        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`" >> \
+            ${target_rootfs}/install/total_solution_bt.manifest
+    done
+
+    # Only install the different pkgs if incremental image generation is set
+    if [ "${INC_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest ]; then
+        cur_list="${target_rootfs}/install/total_solution_bt.manifest"
+        pre_list="${T}/total_solution_bt.manifest"
+        sort -u $cur_list -o $cur_list
+        sort -u $pre_list -o $pre_list
+        comm -1 -3 $cur_list $pre_list | sed 's#.*/\(.*\)\.rpm .*#\1#' > \
+            ${target_rootfs}/install/remove.manifest
+        comm -2 -3 $cur_list $pre_list | awk '{print $1}' > \
+            ${target_rootfs}/install/incremental.manifest
+
+        # Attempt to remove unwanted pkgs, the scripts(pre, post, etc.) has not
+        # been run by now, so don't have to run them(preun, postun, etc.) when
+        # erase the pkg
+        if [ -s ${target_rootfs}/install/remove.manifest ]; then
+            rpm_common_comand --noscripts --nodeps \
+                -e `cat ${target_rootfs}/install/remove.manifest`
+        fi
+
+        # Attempt to install the incremental pkgs
+        rpm_common_comand --nodeps --replacefiles --replacepkgs \
+            -Uvh ${target_rootfs}/install/incremental.manifest
+    else
+        # Attempt to install
+        rpm_common_comand --replacepkgs \
+            -Uhv ${target_rootfs}/install/total_solution.manifest
+    fi
+}
+
 #
 # install a bunch of packages using rpm
 # the following shell variables needs to be set before calling this func:
@@ -406,16 +466,8 @@ EOF
 
 	chmod 0755 ${WORKDIR}/scriptlet_wrapper
 
-	# Attempt install
-	${RPM} --root ${target_rootfs} \
-		--predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
-		--predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \
-		-D "_var ${localstatedir}" \
-		-D "_dbpath ${rpmlibdir}" \
-		--noparentdirs --nolinktos --replacepkgs \
-		-D "__dbi_txn create nofsync private" \
-		-D "_cross_scriptlet_wrapper ${WORKDIR}/scriptlet_wrapper" \
-		-Uhv ${target_rootfs}/install/total_solution.manifest
+    rpm_update_pkg
+
 }
 
 python write_specfile () {
-- 
1.7.4.1

[PATCH 2/2] V2 Incremental image generation(Add config sample)

[Robert Yang]

Add the config sample for incremental image generation to
meta-yocto/conf/local.conf.sample.extended

[YOCTO #1651]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta-yocto/conf/local.conf.sample.extended |    7 +++++++
 1 files changed, 7 insertions(+), 0 deletions(-)

diff --git a/meta-yocto/conf/local.conf.sample.extended b/meta-yocto/conf/local.conf.sample.extended
index 7c26572..64efaa0 100644
--- a/meta-yocto/conf/local.conf.sample.extended
+++ b/meta-yocto/conf/local.conf.sample.extended
@@ -123,3 +123,10 @@
 # The following is a list of classes to import to use in the generation of images
 # currently an example class is image_types_uboot
 # IMAGE_CLASSES = " image_types_uboot"
+
+# Incremental image generation(only for rpm based rootfs currently), the rootfs
+# would be totally removed and re-created in the second generation by default,
+# but with INC_IMAGE_GEN = "1", the rootfs would be kept, and will do
+# update(remove/add some pkgs) on it.
+# NOTE: This is not suggested when you want to create a productive rootfs
+#INC_IMAGE_GEN = "1"
-- 
1.7.4.1

Re: [PATCH 1/2] V2 Incremental image generation(rpm based rootfs)

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 1223 bytes --]


Op 31 dec. 2011, om 09:10 heeft Robert Yang het volgende geschreven:

> Incremental image generation, the rootfs would be totally removed and
> re-created in the second generation by default, but with INC_IMAGE_GEN =
> "1", the rootfs would be kept, and will do update(remove/add some pkgs)
> on it.
> NOTE: This is not suggested when you want to create a productive rootfs
> 
> For example:
>  1) Add the follow config option to a conf file:
>     INC_IMAGE_GEN = "1"
> 
>  2) bitbake core-image-sato
>     modify a package
>     bitbake core-image-sato
> 
> The rootfs would not be totally removed and re-created in the second
> generation, it would be simply updated based on the "package".
> 
> Implatation:
> 1) Figure out the pkg which need to be removed or re-installed, then use
> 'rpm -e to remove the old one. Use the rpm's BUILDTIME to determine
> which pkg has been rebuilt.
> 
> 2) Figure out the pkg which is newly added, and use 'rpm -U' to install
> it.
> 
> This only for the rpm based rootfs, I don't know whether we also need
> this for ipk or deb  based rootfs.

If you insist on calling it "INC_IMAGE_GEN" then yes, you need to implement it for every package format.


[-- Attachment #2: Message signed with OpenPGP using GPGMail --]
[-- Type: application/pgp-signature, Size: 210 bytes --]

Re: [PATCH 1/2] V2 Incremental image generation(rpm based rootfs)

[Richard Purdie]

On Sat, 2011-12-31 at 16:10 +0800, Robert Yang wrote:
> Incremental image generation, the rootfs would be totally removed and
> re-created in the second generation by default, but with INC_IMAGE_GEN =
> "1", the rootfs would be kept, and will do update(remove/add some pkgs)
> on it.
> NOTE: This is not suggested when you want to create a productive rootfs
> 
> For example:
>   1) Add the follow config option to a conf file:
>      INC_IMAGE_GEN = "1"
> 
>   2) bitbake core-image-sato
>      modify a package
>      bitbake core-image-sato
> 
> The rootfs would not be totally removed and re-created in the second
> generation, it would be simply updated based on the "package".
> 
> Implatation:
> 1) Figure out the pkg which need to be removed or re-installed, then use
> 'rpm -e to remove the old one. Use the rpm's BUILDTIME to determine
> which pkg has been rebuilt.
> 
> 2) Figure out the pkg which is newly added, and use 'rpm -U' to install
> it.
> 
> This only for the rpm based rootfs, I don't know whether we also need
> this for ipk or deb  based rootfs.

As Koen points out, this should be called INC_RPM_IMAGE_GEN since it
only works for rpm at present. It would be good to make this work on the
other package backends. If we keep the generic variable name, the other
package backends could error out saying they don't support it.

> [YOCTO #1651]
> 
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>  meta/classes/image.bbclass       |   11 +++++-
>  meta/classes/package_rpm.bbclass |   72 ++++++++++++++++++++++++++++++++-----
>  2 files changed, 71 insertions(+), 12 deletions(-)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 865d430..71b60f3 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -134,15 +134,22 @@ do_rootfs[umask] = 022
>  
>  fakeroot do_rootfs () {
>  	#set -x
> -	rm -rf ${IMAGE_ROOTFS}
> +    # When use the incremental image generation, don't remove the rootfs
> +    if [ "${INC_IMAGE_GEN}" != "1" ]; then
> +        rm -rf ${IMAGE_ROOTFS}
> +    fi
>  	rm -rf ${MULTILIB_TEMP_ROOTFS}
>  	mkdir -p ${IMAGE_ROOTFS}
>  	mkdir -p ${DEPLOY_DIR_IMAGE}
>  
>  	cp ${COREBASE}/meta/files/deploydir_readme.txt ${DEPLOY_DIR_IMAGE}/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt
>  
> -	if [ "${USE_DEVFS}" != "1" ]; then
> +    # If "${IMAGE_ROOTFS}/dev" exists, then the device had been made by
> +    # the previous build
> +	if [ "${USE_DEVFS}" != "1" -a ! -r "${IMAGE_ROOTFS}/dev" ]; then
>  		for devtable in ${@get_devtable_list(d)}; do
> +            # Always return ture since there maybe already one when use the
> +            # incremental image generation
>  			makedevs -r ${IMAGE_ROOTFS} -D $devtable
>  		done
>  	fi
> diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass
> index d03dc3f..541d351 100644
> --- a/meta/classes/package_rpm.bbclass
> +++ b/meta/classes/package_rpm.bbclass
> @@ -147,6 +147,66 @@ resolve_package_rpm () {
>  	echo $pkg_name
>  }
>  
> +# rpm common command and options
> +rpm_common_comand () {
> +
> +    local target_rootfs="${INSTALL_ROOTFS_RPM}"
> +    local extra_args="$@"
> +
> +    ${RPM} --root ${target_rootfs} \
> +        --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \
> +        --predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \
> +        -D "_var ${localstatedir}" \
> +        -D "_dbpath ${rpmlibdir}" \
> +        --noparentdirs --nolinktos \
> +        -D "__dbi_txn create nofsync private" \
> +        -D "_cross_scriptlet_wrapper ${WORKDIR}/scriptlet_wrapper" $extra_args
> +}
> +
> +# install or remove the pkg
> +rpm_update_pkg () {
> +
> +    local target_rootfs="${INSTALL_ROOTFS_RPM}"
> +
> +    # Save the rpm's build time for incremental image generation, and the file
> +    # would be moved to ${T}
> +    rm -f ${target_rootfs}/install/total_solution_bt.manifest
> +    for i in `cat ${target_rootfs}/install/total_solution.manifest`; do
> +        # Use "rpm" rather than "${RPM}" here, since we don't need the
> +        # '--dbpath' option
> +        echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`" >> \
> +            ${target_rootfs}/install/total_solution_bt.manifest
> +    done
> +
> +    # Only install the different pkgs if incremental image generation is set
> +    if [ "${INC_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest ]; then
> +        cur_list="${target_rootfs}/install/total_solution_bt.manifest"
> +        pre_list="${T}/total_solution_bt.manifest"
> +        sort -u $cur_list -o $cur_list
> +        sort -u $pre_list -o $pre_list
> +        comm -1 -3 $cur_list $pre_list | sed 's#.*/\(.*\)\.rpm .*#\1#' > \
> +            ${target_rootfs}/install/remove.manifest
> +        comm -2 -3 $cur_list $pre_list | awk '{print $1}' > \
> +            ${target_rootfs}/install/incremental.manifest
> +
> +        # Attempt to remove unwanted pkgs, the scripts(pre, post, etc.) has not
> +        # been run by now, so don't have to run them(preun, postun, etc.) when
> +        # erase the pkg

Really? I'd have thought it possible the scripts have run during the
previous do_rootfs?

Otherwise the patch looks reasonable to me.

Cheers,

Richard

Re: [PATCH 1/2] V2 Incremental image generation(rpm based rootfs)

[Robert Yang]

Hi Richard,

Please see my comments inline ...

On 01/06/2012 11:22 PM, Richard Purdie wrote:
> On Sat, 2011-12-31 at 16:10 +0800, Robert Yang wrote:
>> Incremental image generation, the rootfs would be totally removed and
>> re-created in the second generation by default, but with INC_IMAGE_GEN =
>> "1", the rootfs would be kept, and will do update(remove/add some pkgs)
>> on it.
>> it.
>>
>> This only for the rpm based rootfs, I don't know whether we also need
>> this for ipk or deb  based rootfs.
>
> As Koen points out, this should be called INC_RPM_IMAGE_GEN since it
> only works for rpm at present. It would be good to make this work on the
> other package backends. If we keep the generic variable name, the other
> package backends could error out saying they don't support it.
>

I did a rough investigation on deb these days, it seems not easy to do in
M2, so I'd like to make it work for both deb and ipk in M3. I will change the
name to INC_RPM_IMAGE_GEN at present, and send a V3 sooner.

>> [YOCTO #1651]
>>
>> Signed-off-by: Robert Yang<liezhi.yang@windriver.com>
>> ---

>> +
>> +        # Attempt to remove unwanted pkgs, the scripts(pre, post, etc.) has not
>> +        # been run by now, so don't have to run them(preun, postun, etc.) when
>> +        # erase the pkg
>
> Really? I'd have thought it possible the scripts have run during the
> previous do_rootfs?
>

Yes, I think so, I think these post scripts are put in /etc/rpm-postinsts/,
and will be run when the system starts at the first time. The best solution
is that remove the script when "rpm -e", but it is hard to know which pkg
owns which script, so I just prevent running the preun and postun when
"rpm -e", I think this is acceptable since the INC_RPM_IMAGE_GEN is not
suggested in the productive rootfs.

// Robert

> Otherwise the patch looks reasonable to me.
>
> Cheers,
>
> Richard
>
>
>