Re: [PATCH 00/49] Rootfs/image generation refactoring (cover letter only)

[Laurentiu Palcu <laurentiu.palcu@intel.com>]

In my ignorance, I forgot to thank Hongxu, from WindRiver, for his help
on implementing the rpm related part.

Thanks Hongxu,
laurentiu

On Mon, Feb 03, 2014 at 01:38:35PM +0200, Laurentiu Palcu wrote:
> Hi all,
> 
> First off: this large patchset adds no change in the rootfs/image creation
> functionality. The main logic is supposed to be the same as before. Also, I
> decided not to spam the list with this large patchset (49 patches) and I sent
> only the cover-letter. Those who want to take a peak at the code, can have a
> look by clicking the link below.
> 
> If people would like the patches sent to the list too, just say so.
> 
> Ok, what's it all about then? Here are the main changes:
>  * all the old bash shell code (more then 90% of it anyway) has been ported to
>    Python. This would, hopefully, make the code more readable and allows us to
>    use some of the Python powerful features. The following new files were born,
>    together with a fair amount of changes in some bbclass'es:
> 
>     > meta/lib/oe/manifest.py - implements the initial manifest creation (for now).
>       This is a file containing the list of packages going into the final image.
>       In theory, we should be able to use this file later to re-generate the
>       rootfs from a package feed, outside bitbake environment;
> 
>     > meta/lib/oe/package_manager.py - implements the basic package management
>       API: install/remove packages, update index DB/index file creation. This
>       is mainly used by Rootfs/Sdk classes in order to generate the rootfs;
> 
>     > meta/lib/oe/rootfs.py - contains code for rootfs generation;
> 
>     > meta/lib/oe/image.py - the image creation part;
> 
>     > meta/lib/oe/sdk.py - populate the SDK target/host rootfs.
> 
>    With the new code, the entire do_rootfs routine in image.bbclass has now just
>    3 function calls: create_manifest(), create_rootfs(), create_image(). In this
>    order. Easier to debug just a certain part of the rootfs/image creation process.
> 
>  * Some performance improvement has been obtained in 2 areas: index file creation
>    and image generation. These operations were serial in the old code. Now, they're
>    done in parallel.
> 
>  * People complained about intercept scripts being hard to debug. That was because
>    the old code used a log checking routine that scanned the logs for some
>    standart error strings which were not necessarily errors. Hence, the intercepts
>    had their stderr redirected to /dev/null. This was now removed since we can
>    easily handle intercepts errors in python. So, errors will go to the log file.
> 
> That's about it. There is room for even further improvement, in the index creation
> part. I had some code in place for that but I had to remove it, temporarily, due
> to some synchronization issues. Hopefully, I'll find the issue and include that
> piece back.
> 
> The code was tested on my machine and on a local AB for x86, x86-64, arm, ppc,
> mips and build appliance. The builds and sanity tests finished successfully.
> 
> laurentiu
> 
> The following changes since commit b37dd451a52622d5b570183a81583cc34c2ff555:
> 
>   rootfs_ipk: Ensure that BAD_RECOMMENDATIONS are honoured for all architectures (2014-02-02 22:37:42 +0000)
> 
> are available in the git repository at:
> 
>   git://git.yoctoproject.org/poky-contrib lpalcu/rootfs_refactoring_ship_oecore
>   http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lpalcu/rootfs_refactoring_ship_oecore
> 
> Hongxu Jia (18):
>   lib/oe/manifest.py: add rpm image manifest creation
>   lib/oe/package_manager.py: support RpmPM
>   lib/oe/rootfs.py: support RpmRootfs
>   lib/oe/rootfs.py: Tweak the rootfs removing operation
>   rootfs.py: fix uninstall uneeded pkgs failed
>   rootfs.py: strip kernel-abiversion useless readline
>   lib/oe/image.py: fix get rootfs_extra_space failed
>   image.bbclass/license.bbclass: ajustment list_installed_packages
>     invoking
>   license.bbclass: fix license_create_manifest failed
>   lib/oe/sdk.py: support RpmRootfs
>   image.bbclass/populate_sdk_base.bbclass: shift position of sdk
>     complementary install definition
>   Cleanup rootfs_rpm,package_rpm bbclass files
>   lib/oe/rootfs.py: fix invoking rpm _handle_intercept_failure failed
>   lib/oe/package_manager.py: fix invoking rpm save_rpmpostinist failed
>   lib/oe/rootfs.py: fix invoking _run_intercepts failed
>   populate_sdk_rpm.bbclass: remove old shell code
>   package_manger.py/rootfs.py: rename save_rpmpostinist with
>     save_rpmpostinst
>   lib/oe/package_manager.py: moving global SCRIPTLET_FORMAT and
>     DB_CONFIG_CONTENT to RpmPM
> 
> Laurentiu Palcu (31):
>   lib/oe/utils.py: add pre/post process helper function
>   lib/oe/image.py: add new image creation library
>   lib/oe/manifest.py: add library for image manifest creation
>   lib/oe/package_manager.py: add new package management library
>   lib/oe/rootfs.py: add new library for rootfs creation
>   Activate the new python rootfs/image creation routines
>   oe/manifest.py: Add manifest creation for opkg
>   package_(deb|ipk).bbclass: remove the stamp when creating package
>     from cache
>   lib/oe/manifest.py: create global variables for package types
>   lib/oe/package_manager.py: add deploy dir locking mechanism
>   lib/oe/package_manager.py: use the bitbake APT_ARGS variable
>   lib/oe/package_manager.py: add support for opkg backend
>   lib/oe/rootfs.py: add support for opkg backend
>   Cleanup image,rootfs_ipk,package_ipk bbclass files
>   image.bbclass, lib/oe/rootfs.py: remove intercept_scripts directory
>   lib/oe/manifest.py: create manifests for SDK too
>   lib/oe/rootfs.py: use Manifest.INSTALL_ORDER
>   lib/oe/package_manager.py: fixes for dpkg backend
>   lib/oe/sdk.py: add SDK class
>   populate_sdk_base.bbclass: activate the new python routines
>   image.bbclass: change POPULATE_SDK_POST_TARGET_COMMAND
>   populate_sdk_(deb|ipk).bbclass: remove old shell code
>   lib/oe/package_manager.py: RpmPM fixes
>   lib/oe/rootfs.py: warn the user if intercept hooks fail
>   scripts/postinst-intercepts: do not redirect errors to /dev/null
>   lib/oe/package_manager.py: DpkgPM, use lock on deploy dir
>   lib/oe/image.py: fix image size calculation routine
>   lib/oe/package_manager.py: OpkgPM/DpkgPM fixes
>   lib/oe/rootfs.py: OpkgRootfs, fix issue in _get_delayed_postinsts()
>   lib/oe/package_manager.py: do not use stamps
>   lib/oe/package_manager.py: OpkgPM, bad recommendation handling change
> 
>  meta/classes/image.bbclass                      |  329 +-----
>  meta/classes/image_types.bbclass                |  113 --
>  meta/classes/license.bbclass                    |   16 +-
>  meta/classes/package_deb.bbclass                |  151 +--
>  meta/classes/package_ipk.bbclass                |  202 +---
>  meta/classes/package_rpm.bbclass                |  472 --------
>  meta/classes/populate_sdk_base.bbclass          |   63 +-
>  meta/classes/populate_sdk_deb.bbclass           |   59 -
>  meta/classes/populate_sdk_ipk.bbclass           |   54 -
>  meta/classes/populate_sdk_rpm.bbclass           |  152 ---
>  meta/classes/rootfs_deb.bbclass                 |  115 --
>  meta/classes/rootfs_ipk.bbclass                 |  109 +-
>  meta/classes/rootfs_rpm.bbclass                 |  184 ----
>  meta/lib/oe/image.py                            |  239 ++++
>  meta/lib/oe/manifest.py                         |  291 +++++
>  meta/lib/oe/package_manager.py                  | 1326 +++++++++++++++++++++++
>  meta/lib/oe/rootfs.py                           |  680 ++++++++++++
>  meta/lib/oe/sdk.py                              |  343 ++++++
>  meta/lib/oe/utils.py                            |   12 +-
>  scripts/postinst-intercepts/update_font_cache   |    2 +-
>  scripts/postinst-intercepts/update_pixbuf_cache |    2 +-
>  21 files changed, 2965 insertions(+), 1949 deletions(-)
>  create mode 100644 meta/lib/oe/image.py
>  create mode 100644 meta/lib/oe/manifest.py
>  create mode 100644 meta/lib/oe/package_manager.py
>  create mode 100644 meta/lib/oe/rootfs.py
>  create mode 100644 meta/lib/oe/sdk.py
> 
> -- 
> 1.7.9.5
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core