* [PATCH 0/2] V2 Incremental image generation(rpm based rootfs)
@ 2011-12-31 8:10 Robert Yang
2011-12-31 8:10 ` [PATCH 1/2] " Robert Yang
2011-12-31 8:10 ` [PATCH 2/2] V2 Incremental image generation(Add config sample) Robert Yang
0 siblings, 2 replies; 6+ messages in thread
From: Robert Yang @ 2011-12-31 8:10 UTC (permalink / raw)
To: mark.hatle, sgw, openembedded-core
Changes of V2:
* Move the config sample from meta-yocto/conf/local.conf.sample to
meta-yocto/conf/local.conf.sample.extended as Saul suggested.
Testing:
1) Enable INC_IMAGE_GEN = "1" in conf/local.conf
2) $ bitbake core-image-sato
* Test when remove some pkgs
a) Save the modify time of file which is in
tmp/work/qemux86_64-poky-linux/core-image-sato-1.0-r0/rootfs/dev/.
b) Edit meta/recipes-sato/images/core-image-sato.bb, remove
${SATO_IMAGE_FEATURES} from IMAGE_FEATURES.
c) bitbake core-image-sato
Result: The project built well, check the modify time of the file
which is in
tmp/work/qemux86_64-poky-linux/core-image-sato-1.0-r0/rootfs/dev/,
They should be the same to step a)'s, which means that these files
were note newly created.
* Testh when add some pkgs
a) Save the modify time of file which is in
tmp/work/qemux86_64-poky-linux/core-image-sato-1.0-r0/rootfs/dev/.
b) Edit meta/recipes-sato/images/core-image-sato.bb, add the
${SATO_IMAGE_FEATURES} back to IMAGE_FEATURES.
c) bitbake core-image-sato
Result: The project built well, check the modify time of the file
which is in
tmp/work/qemux86_64-poky-linux/core-image-sato-1.0-r0/rootfs/dev/,
They should be the same to step a)'s, which means that these files
were note newly created.
* Test when edit a pkg
a) bitbake bzip2 -cclean
b) rm -f rm -f sstate-cache/sstate-bzip2-*
c) Edit meta/recipes-extended/bzip2/bzip2_1.0.6.bb
d) bitbake core-image-sato
Result: The project built well, and check:
tmp/work/qemux86-poky-linux/core-image-sato-1.0-r0/temp/log.do_rootfs
Only the bzip2 has been re-installed
// Robert
The following changes since commit f5aa3bbda623c8fae3a761d72fddc95631ad0706:
coreutils: ensure --color works so DEPEND on libcap (2011-12-24 10:05:47 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib robert/inc_image_gen
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/inc_image_gen
Robert Yang (2):
Incremental image generation(rpm based rootfs)
Incremental image generation(Add config sample)
meta-yocto/conf/local.conf.sample.extended | 7 +++
meta/classes/image.bbclass | 11 ++++-
meta/classes/package_rpm.bbclass | 72 ++++++++++++++++++++++++----
3 files changed, 78 insertions(+), 12 deletions(-)
--
1.7.4.1
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH 1/2] V2 Incremental image generation(rpm based rootfs) 2011-12-31 8:10 [PATCH 0/2] V2 Incremental image generation(rpm based rootfs) Robert Yang @ 2011-12-31 8:10 ` Robert Yang 2011-12-31 8:48 ` Koen Kooi 2012-01-06 15:22 ` Richard Purdie 2011-12-31 8:10 ` [PATCH 2/2] V2 Incremental image generation(Add config sample) Robert Yang 1 sibling, 2 replies; 6+ messages in thread From: Robert Yang @ 2011-12-31 8:10 UTC (permalink / raw) To: mark.hatle, sgw, openembedded-core Incremental image generation, the rootfs would be totally removed and re-created in the second generation by default, but with INC_IMAGE_GEN = "1", the rootfs would be kept, and will do update(remove/add some pkgs) on it. NOTE: This is not suggested when you want to create a productive rootfs For example: 1) Add the follow config option to a conf file: INC_IMAGE_GEN = "1" 2) bitbake core-image-sato modify a package bitbake core-image-sato The rootfs would not be totally removed and re-created in the second generation, it would be simply updated based on the "package". Implatation: 1) Figure out the pkg which need to be removed or re-installed, then use 'rpm -e to remove the old one. Use the rpm's BUILDTIME to determine which pkg has been rebuilt. 2) Figure out the pkg which is newly added, and use 'rpm -U' to install it. This only for the rpm based rootfs, I don't know whether we also need this for ipk or deb based rootfs. [YOCTO #1651] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- meta/classes/image.bbclass | 11 +++++- meta/classes/package_rpm.bbclass | 72 ++++++++++++++++++++++++++++++++----- 2 files changed, 71 insertions(+), 12 deletions(-) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 865d430..71b60f3 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -134,15 +134,22 @@ do_rootfs[umask] = 022 fakeroot do_rootfs () { #set -x - rm -rf ${IMAGE_ROOTFS} + # When use the incremental image generation, don't remove the rootfs + if [ "${INC_IMAGE_GEN}" != "1" ]; then + rm -rf ${IMAGE_ROOTFS} + fi rm -rf ${MULTILIB_TEMP_ROOTFS} mkdir -p ${IMAGE_ROOTFS} mkdir -p ${DEPLOY_DIR_IMAGE} cp ${COREBASE}/meta/files/deploydir_readme.txt ${DEPLOY_DIR_IMAGE}/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt - if [ "${USE_DEVFS}" != "1" ]; then + # If "${IMAGE_ROOTFS}/dev" exists, then the device had been made by + # the previous build + if [ "${USE_DEVFS}" != "1" -a ! -r "${IMAGE_ROOTFS}/dev" ]; then for devtable in ${@get_devtable_list(d)}; do + # Always return ture since there maybe already one when use the + # incremental image generation makedevs -r ${IMAGE_ROOTFS} -D $devtable done fi diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass index d03dc3f..541d351 100644 --- a/meta/classes/package_rpm.bbclass +++ b/meta/classes/package_rpm.bbclass @@ -147,6 +147,66 @@ resolve_package_rpm () { echo $pkg_name } +# rpm common command and options +rpm_common_comand () { + + local target_rootfs="${INSTALL_ROOTFS_RPM}" + local extra_args="$@" + + ${RPM} --root ${target_rootfs} \ + --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \ + --predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \ + -D "_var ${localstatedir}" \ + -D "_dbpath ${rpmlibdir}" \ + --noparentdirs --nolinktos \ + -D "__dbi_txn create nofsync private" \ + -D "_cross_scriptlet_wrapper ${WORKDIR}/scriptlet_wrapper" $extra_args +} + +# install or remove the pkg +rpm_update_pkg () { + + local target_rootfs="${INSTALL_ROOTFS_RPM}" + + # Save the rpm's build time for incremental image generation, and the file + # would be moved to ${T} + rm -f ${target_rootfs}/install/total_solution_bt.manifest + for i in `cat ${target_rootfs}/install/total_solution.manifest`; do + # Use "rpm" rather than "${RPM}" here, since we don't need the + # '--dbpath' option + echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`" >> \ + ${target_rootfs}/install/total_solution_bt.manifest + done + + # Only install the different pkgs if incremental image generation is set + if [ "${INC_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest ]; then + cur_list="${target_rootfs}/install/total_solution_bt.manifest" + pre_list="${T}/total_solution_bt.manifest" + sort -u $cur_list -o $cur_list + sort -u $pre_list -o $pre_list + comm -1 -3 $cur_list $pre_list | sed 's#.*/\(.*\)\.rpm .*#\1#' > \ + ${target_rootfs}/install/remove.manifest + comm -2 -3 $cur_list $pre_list | awk '{print $1}' > \ + ${target_rootfs}/install/incremental.manifest + + # Attempt to remove unwanted pkgs, the scripts(pre, post, etc.) has not + # been run by now, so don't have to run them(preun, postun, etc.) when + # erase the pkg + if [ -s ${target_rootfs}/install/remove.manifest ]; then + rpm_common_comand --noscripts --nodeps \ + -e `cat ${target_rootfs}/install/remove.manifest` + fi + + # Attempt to install the incremental pkgs + rpm_common_comand --nodeps --replacefiles --replacepkgs \ + -Uvh ${target_rootfs}/install/incremental.manifest + else + # Attempt to install + rpm_common_comand --replacepkgs \ + -Uhv ${target_rootfs}/install/total_solution.manifest + fi +} + # # install a bunch of packages using rpm # the following shell variables needs to be set before calling this func: @@ -406,16 +466,8 @@ EOF chmod 0755 ${WORKDIR}/scriptlet_wrapper - # Attempt install - ${RPM} --root ${target_rootfs} \ - --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \ - --predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \ - -D "_var ${localstatedir}" \ - -D "_dbpath ${rpmlibdir}" \ - --noparentdirs --nolinktos --replacepkgs \ - -D "__dbi_txn create nofsync private" \ - -D "_cross_scriptlet_wrapper ${WORKDIR}/scriptlet_wrapper" \ - -Uhv ${target_rootfs}/install/total_solution.manifest + rpm_update_pkg + } python write_specfile () { -- 1.7.4.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] V2 Incremental image generation(rpm based rootfs) 2011-12-31 8:10 ` [PATCH 1/2] " Robert Yang @ 2011-12-31 8:48 ` Koen Kooi 2012-01-06 15:22 ` Richard Purdie 1 sibling, 0 replies; 6+ messages in thread From: Koen Kooi @ 2011-12-31 8:48 UTC (permalink / raw) To: Patches and discussions about the oe-core layer [-- Attachment #1: Type: text/plain, Size: 1223 bytes --] Op 31 dec. 2011, om 09:10 heeft Robert Yang het volgende geschreven: > Incremental image generation, the rootfs would be totally removed and > re-created in the second generation by default, but with INC_IMAGE_GEN = > "1", the rootfs would be kept, and will do update(remove/add some pkgs) > on it. > NOTE: This is not suggested when you want to create a productive rootfs > > For example: > 1) Add the follow config option to a conf file: > INC_IMAGE_GEN = "1" > > 2) bitbake core-image-sato > modify a package > bitbake core-image-sato > > The rootfs would not be totally removed and re-created in the second > generation, it would be simply updated based on the "package". > > Implatation: > 1) Figure out the pkg which need to be removed or re-installed, then use > 'rpm -e to remove the old one. Use the rpm's BUILDTIME to determine > which pkg has been rebuilt. > > 2) Figure out the pkg which is newly added, and use 'rpm -U' to install > it. > > This only for the rpm based rootfs, I don't know whether we also need > this for ipk or deb based rootfs. If you insist on calling it "INC_IMAGE_GEN" then yes, you need to implement it for every package format. [-- Attachment #2: Message signed with OpenPGP using GPGMail --] [-- Type: application/pgp-signature, Size: 210 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] V2 Incremental image generation(rpm based rootfs) 2011-12-31 8:10 ` [PATCH 1/2] " Robert Yang 2011-12-31 8:48 ` Koen Kooi @ 2012-01-06 15:22 ` Richard Purdie 2012-01-10 10:40 ` Robert Yang 1 sibling, 1 reply; 6+ messages in thread From: Richard Purdie @ 2012-01-06 15:22 UTC (permalink / raw) To: Robert Yang; +Cc: openembedded-core On Sat, 2011-12-31 at 16:10 +0800, Robert Yang wrote: > Incremental image generation, the rootfs would be totally removed and > re-created in the second generation by default, but with INC_IMAGE_GEN = > "1", the rootfs would be kept, and will do update(remove/add some pkgs) > on it. > NOTE: This is not suggested when you want to create a productive rootfs > > For example: > 1) Add the follow config option to a conf file: > INC_IMAGE_GEN = "1" > > 2) bitbake core-image-sato > modify a package > bitbake core-image-sato > > The rootfs would not be totally removed and re-created in the second > generation, it would be simply updated based on the "package". > > Implatation: > 1) Figure out the pkg which need to be removed or re-installed, then use > 'rpm -e to remove the old one. Use the rpm's BUILDTIME to determine > which pkg has been rebuilt. > > 2) Figure out the pkg which is newly added, and use 'rpm -U' to install > it. > > This only for the rpm based rootfs, I don't know whether we also need > this for ipk or deb based rootfs. As Koen points out, this should be called INC_RPM_IMAGE_GEN since it only works for rpm at present. It would be good to make this work on the other package backends. If we keep the generic variable name, the other package backends could error out saying they don't support it. > [YOCTO #1651] > > Signed-off-by: Robert Yang <liezhi.yang@windriver.com> > --- > meta/classes/image.bbclass | 11 +++++- > meta/classes/package_rpm.bbclass | 72 ++++++++++++++++++++++++++++++++----- > 2 files changed, 71 insertions(+), 12 deletions(-) > > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index 865d430..71b60f3 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -134,15 +134,22 @@ do_rootfs[umask] = 022 > > fakeroot do_rootfs () { > #set -x > - rm -rf ${IMAGE_ROOTFS} > + # When use the incremental image generation, don't remove the rootfs > + if [ "${INC_IMAGE_GEN}" != "1" ]; then > + rm -rf ${IMAGE_ROOTFS} > + fi > rm -rf ${MULTILIB_TEMP_ROOTFS} > mkdir -p ${IMAGE_ROOTFS} > mkdir -p ${DEPLOY_DIR_IMAGE} > > cp ${COREBASE}/meta/files/deploydir_readme.txt ${DEPLOY_DIR_IMAGE}/README_-_DO_NOT_DELETE_FILES_IN_THIS_DIRECTORY.txt > > - if [ "${USE_DEVFS}" != "1" ]; then > + # If "${IMAGE_ROOTFS}/dev" exists, then the device had been made by > + # the previous build > + if [ "${USE_DEVFS}" != "1" -a ! -r "${IMAGE_ROOTFS}/dev" ]; then > for devtable in ${@get_devtable_list(d)}; do > + # Always return ture since there maybe already one when use the > + # incremental image generation > makedevs -r ${IMAGE_ROOTFS} -D $devtable > done > fi > diff --git a/meta/classes/package_rpm.bbclass b/meta/classes/package_rpm.bbclass > index d03dc3f..541d351 100644 > --- a/meta/classes/package_rpm.bbclass > +++ b/meta/classes/package_rpm.bbclass > @@ -147,6 +147,66 @@ resolve_package_rpm () { > echo $pkg_name > } > > +# rpm common command and options > +rpm_common_comand () { > + > + local target_rootfs="${INSTALL_ROOTFS_RPM}" > + local extra_args="$@" > + > + ${RPM} --root ${target_rootfs} \ > + --predefine "_rpmds_sysinfo_path ${target_rootfs}/etc/rpm/sysinfo" \ > + --predefine "_rpmrc_platform_path ${target_rootfs}/etc/rpm/platform" \ > + -D "_var ${localstatedir}" \ > + -D "_dbpath ${rpmlibdir}" \ > + --noparentdirs --nolinktos \ > + -D "__dbi_txn create nofsync private" \ > + -D "_cross_scriptlet_wrapper ${WORKDIR}/scriptlet_wrapper" $extra_args > +} > + > +# install or remove the pkg > +rpm_update_pkg () { > + > + local target_rootfs="${INSTALL_ROOTFS_RPM}" > + > + # Save the rpm's build time for incremental image generation, and the file > + # would be moved to ${T} > + rm -f ${target_rootfs}/install/total_solution_bt.manifest > + for i in `cat ${target_rootfs}/install/total_solution.manifest`; do > + # Use "rpm" rather than "${RPM}" here, since we don't need the > + # '--dbpath' option > + echo "$i `rpm -qp --qf '%{BUILDTIME}\n' $i`" >> \ > + ${target_rootfs}/install/total_solution_bt.manifest > + done > + > + # Only install the different pkgs if incremental image generation is set > + if [ "${INC_IMAGE_GEN}" = "1" -a -f ${T}/total_solution_bt.manifest ]; then > + cur_list="${target_rootfs}/install/total_solution_bt.manifest" > + pre_list="${T}/total_solution_bt.manifest" > + sort -u $cur_list -o $cur_list > + sort -u $pre_list -o $pre_list > + comm -1 -3 $cur_list $pre_list | sed 's#.*/\(.*\)\.rpm .*#\1#' > \ > + ${target_rootfs}/install/remove.manifest > + comm -2 -3 $cur_list $pre_list | awk '{print $1}' > \ > + ${target_rootfs}/install/incremental.manifest > + > + # Attempt to remove unwanted pkgs, the scripts(pre, post, etc.) has not > + # been run by now, so don't have to run them(preun, postun, etc.) when > + # erase the pkg Really? I'd have thought it possible the scripts have run during the previous do_rootfs? Otherwise the patch looks reasonable to me. Cheers, Richard ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] V2 Incremental image generation(rpm based rootfs) 2012-01-06 15:22 ` Richard Purdie @ 2012-01-10 10:40 ` Robert Yang 0 siblings, 0 replies; 6+ messages in thread From: Robert Yang @ 2012-01-10 10:40 UTC (permalink / raw) To: Richard Purdie; +Cc: openembedded-core Hi Richard, Please see my comments inline ... On 01/06/2012 11:22 PM, Richard Purdie wrote: > On Sat, 2011-12-31 at 16:10 +0800, Robert Yang wrote: >> Incremental image generation, the rootfs would be totally removed and >> re-created in the second generation by default, but with INC_IMAGE_GEN = >> "1", the rootfs would be kept, and will do update(remove/add some pkgs) >> on it. >> it. >> >> This only for the rpm based rootfs, I don't know whether we also need >> this for ipk or deb based rootfs. > > As Koen points out, this should be called INC_RPM_IMAGE_GEN since it > only works for rpm at present. It would be good to make this work on the > other package backends. If we keep the generic variable name, the other > package backends could error out saying they don't support it. > I did a rough investigation on deb these days, it seems not easy to do in M2, so I'd like to make it work for both deb and ipk in M3. I will change the name to INC_RPM_IMAGE_GEN at present, and send a V3 sooner. >> [YOCTO #1651] >> >> Signed-off-by: Robert Yang<liezhi.yang@windriver.com> >> --- >> + >> + # Attempt to remove unwanted pkgs, the scripts(pre, post, etc.) has not >> + # been run by now, so don't have to run them(preun, postun, etc.) when >> + # erase the pkg > > Really? I'd have thought it possible the scripts have run during the > previous do_rootfs? > Yes, I think so, I think these post scripts are put in /etc/rpm-postinsts/, and will be run when the system starts at the first time. The best solution is that remove the script when "rpm -e", but it is hard to know which pkg owns which script, so I just prevent running the preun and postun when "rpm -e", I think this is acceptable since the INC_RPM_IMAGE_GEN is not suggested in the productive rootfs. // Robert > Otherwise the patch looks reasonable to me. > > Cheers, > > Richard > > > ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 2/2] V2 Incremental image generation(Add config sample) 2011-12-31 8:10 [PATCH 0/2] V2 Incremental image generation(rpm based rootfs) Robert Yang 2011-12-31 8:10 ` [PATCH 1/2] " Robert Yang @ 2011-12-31 8:10 ` Robert Yang 1 sibling, 0 replies; 6+ messages in thread From: Robert Yang @ 2011-12-31 8:10 UTC (permalink / raw) To: mark.hatle, sgw, openembedded-core Add the config sample for incremental image generation to meta-yocto/conf/local.conf.sample.extended [YOCTO #1651] Signed-off-by: Robert Yang <liezhi.yang@windriver.com> --- meta-yocto/conf/local.conf.sample.extended | 7 +++++++ 1 files changed, 7 insertions(+), 0 deletions(-) diff --git a/meta-yocto/conf/local.conf.sample.extended b/meta-yocto/conf/local.conf.sample.extended index 7c26572..64efaa0 100644 --- a/meta-yocto/conf/local.conf.sample.extended +++ b/meta-yocto/conf/local.conf.sample.extended @@ -123,3 +123,10 @@ # The following is a list of classes to import to use in the generation of images # currently an example class is image_types_uboot # IMAGE_CLASSES = " image_types_uboot" + +# Incremental image generation(only for rpm based rootfs currently), the rootfs +# would be totally removed and re-created in the second generation by default, +# but with INC_IMAGE_GEN = "1", the rootfs would be kept, and will do +# update(remove/add some pkgs) on it. +# NOTE: This is not suggested when you want to create a productive rootfs +#INC_IMAGE_GEN = "1" -- 1.7.4.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-01-10 10:47 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-12-31 8:10 [PATCH 0/2] V2 Incremental image generation(rpm based rootfs) Robert Yang 2011-12-31 8:10 ` [PATCH 1/2] " Robert Yang 2011-12-31 8:48 ` Koen Kooi 2012-01-06 15:22 ` Richard Purdie 2012-01-10 10:40 ` Robert Yang 2011-12-31 8:10 ` [PATCH 2/2] V2 Incremental image generation(Add config sample) Robert Yang
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.