[PATCH V3 0/5] manifest.py/package_manager.py/rootfs.py: support ipk incremental image generation

[Hongxu Jia <hongxu.jia@windriver.com>]

Change in V2:
- Move dump_install_solution to Manifest class as create_full;
- Move load_old_install_solution to Manifest class as parse_full_manifest;
- Handle the output of dummy install in Python rather than shell;
- Fix typos
- Rename _file_duplicate with _file_equal to avoid confusion;

Change in V3:
- While create_full calls dummy_install, pass an instance to PM
  as an argument rather than instantiate another PM object;
- Rename _create_incremental with _remove_extra_packages;
- Move the call of _remove_extra_packages below the call of
  self.pm.handle_bad_recommendations();
- Remove redundant self.pm.update in _remove_extra_packages;
- Rename _remove_existing_image with _remove_old_rootfs;

Test Cases

---------------------------------------------
Case 1: Rebuild in place
1. Edit local.conf, enable multilib and ipk incremental image generation
...
 46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257 
258 IMAGE_INSTALL_append = " lib32-gzip"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear "
262 
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

2. Edit local.conf, add some spaces in IMAGE_INSTALL_append
vim local.conf
...
258 IMAGE_INSTALL_append = " lib32-gzip "
...

3. bb core-image-minimal, the image should not be rebuilt
vim log.do_rootfs:
...
 16 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date.
 17 Package opkg-collateral (1.0-r2) installed in root is up to date.
 18 Package run-postinsts (1.0-r9) installed in root is up to date.
 19 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
 20 Package opkg (1:0.2.0-r0) installed in root is up to date.
 21 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
 ...
 53 NOTE: Package lib32-gzip (1.6-r7) installed in root is up to date.
...

vim installed_pkgs.txt
...
lib32-gzip x86
...

4. runqemu qemux86-64, ssh could work and lib32-gzip is installed
On host:
runqemu qemux86-64
ssh root@192.168.7.2 and run 'opkg-cl list-installed'
...
lib32-gzip - 1.6-r7
...

---------------------------------------------
Case 2: Decremental Rebuild
1. Edit local.conf, enable multilib and ipk incremental image generation
...
 46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257 
258 IMAGE_INSTALL_append = " lib32-gzip openssl"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear "
262 
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

2. Edit local.conf, remove 'lib32-gzip openssl'
vim local.conf
...
258 # IMAGE_INSTALL_append = " lib32-gzip openssl"
...

3. bb core-image-minimal, lib32-gzip and openssl should be removed.
vim log.do_rootfs:
...
 20 NOTE: Removing package lib32-eglibc from root...
 21 Removing package lib32-gzip from root...
 22 Removing package lib32-update-alternatives-opkg from root...
 23 Removing package libcrypto1.0.0 from root...
 24 Removing package libssl1.0.0 from root...
 25 Removing package openssl from root...
 26 Removing package openssl-conf from root...
 ...
 31 Package opkg-collateral (1.0-r2) installed in root is up to date.
 32 Package run-postinsts (1.0-r9) installed in root is up to date.
 33 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
 34 Package opkg (1:0.2.0-r0) installed in root is up to date.
 35 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
...

---------------------------------------------
Case 3: Incremental Rebuild
1. Edit local.conf, enable multilib and ipk incremental image generation
...
 46 MACHINE ?= "qemux86-64"
254 require conf/multilib.conf
255 MULTILIBS = "multilib:lib32"
256 DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
257 
258 # IMAGE_INSTALL_append = " lib32-gzip openssl"
260 IMAGE_FEATURES_append = " package-management ssh-server-dropbear "
262 
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

2. Edit local.conf, append 'lib32-gzip openssl' to IMAGE_INSTALL
vim local.conf
...
258 IMAGE_INSTALL_append = " lib32-gzip openssl"
...

3. bb core-image-minimal, lib32-gzip and openssl should be added.
vim log.do_rootfs:
...
 20 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date.
 21 Package opkg-collateral (1.0-r2) installed in root is up to date.
 22 Package run-postinsts (1.0-r9) installed in root is up to date.
 23 Installing openssl (1.0.1e-r15.0) to root...
 ...
 31 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
 32 Package opkg (1:0.2.0-r0) installed in root is up to date.
 33 Package poky-feed-config-opkg (1.0-r2) installed in root is up to date.
 ...
 54 NOTE: Installing the following packages: lib32-gzip
...

---------------------------------------------
Case 4: Rebuild in place and Upgrade package
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " gzip"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

2. Edit local.conf, upgrade gzip's PR
...
258 PR_pn-gzip = "r1"
...

3. bb core-image-minimal, gzip should be upgraded to r1
vim log.do_rootfs:
...
 21 NOTE: Package packagegroup-core-ssh-dropbear (1.0-r1) installed in root is up to date.
 22 Package opkg-collateral (1.0-r2) installed in root is up to date.
 23 Package run-postinsts (1.0-r9) installed in root is up to date.
 24 Package packagegroup-core-boot (1.0-r11) installed in root is up to date.
 25 Package opkg (1:0.2.0-r0) installed in root is up to date.
 26 Upgrading gzip on root from 1.6-r0 to 1.6-r1...
...

---------------------------------------------
Case 5: Test BAD_RECOMMENDATIONS
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " util-linux"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

The util-linux-fdisk is in util-linux's RRECOMMENDS
vim installed_pkgs.txt
...
41 util-linux-fdisk core2-64
...

2. Edit local.conf, add some spaces to IMAGE_INSTALL and add
util-linux-fdisk to core-image-minimal's BAD_RECOMMENDATIONS
...
260 IMAGE_INSTALL_append = " util-linux  "
265 BAD_RECOMMENDATIONS_pn-core-image-minimal += "util-linux-fdisk"
...

3. bb core-image-minimal, util-linux-fdisk should not be installed
vim log.do_rootfs:
...
34 NOTE: Requested ignored recommendation util-linux-fdisk is not a package
    ...
119 util-linux: ignoring recommendation for util-linux-fdisk at user request
120 util-linux: unsatisfied recommendation for util-linux-fdisk
...

The util-linux-fdisk doesn't exist in installed_pkgs.txt any more.

---------------------------------------------
Case 6: Test NO_RECOMMENDATIONS
1. Edit local.conf, enable ipk incremental image generation
...
258 IMAGE_INSTALL_append = " util-linux"
263 INC_IPK_IMAGE_GEN = "1"
...
bb core-image-minimal

The util-linux-fdisk util-linux-cfdisk util-linux-sfdisk
util-linux-mount util-linux-readprofile util-linux-mkfs are
in util-linux's RRECOMMENDS

vim installed_pkgs.txt
...
4 util-linux-mkfs core2-64
18 util-linux-sfdisk core2-64
20 util-linux-mount core2-64
38 util-linux-cfdisk core2-64
41 util-linux-fdisk core2-64
46 util-linux-readprofile core2-64
...

2. Edit local.conf, add some spaces to IMAGE_INSTALL and
   assign core-image-minimal's NO_RECOMMENDATIONS with 1
...
260 IMAGE_INSTALL_append = " util-linux  "
267 NO_RECOMMENDATIONS_pn-core-image-minimal = "1"
...

3. bb core-image-minimal, The six packages should not be installed
vim log.do_rootfs:
...
34 NOTE: Requested ignored recommendation util-linux-fdisk is not a package
    ...
113 util-linux: ignoring recommendation for util-linux-sfdisk at user request
114 util-linux: unsatisfied recommendation for util-linux-sfdisk
115 util-linux: ignoring recommendation for util-linux-cfdisk at user request
116 util-linux: unsatisfied recommendation for util-linux-cfdisk
117 util-linux: ignoring recommendation for util-linux-mount at user request
118 util-linux: unsatisfied recommendation for util-linux-mount
119 util-linux: ignoring recommendation for util-linux-fdisk at user request
120 util-linux: unsatisfied recommendation for util-linux-fdisk
121 util-linux: ignoring recommendation for util-linux-mkfs at user request
122 util-linux: unsatisfied recommendation for util-linux-mkfs
123 util-linux: ignoring recommendation for util-linux-readprofile at user request
124 util-linux: unsatisfied recommendation for util-linux-readprofile
...

They don't exist in installed_pkgs.txt any more.

//Hongxu

The following changes since commit bb0c26960d3b05070346cdfdfd05d6fd4ad0a7b1:

  wic: Make exec_native_command() fail if a command isn't found (2014-02-20 14:28:13 +0000)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib hongxu/ipk-inc-gen
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/ipk-inc-gen

Hongxu Jia (5):
  manifest.py: add create_full for OpkgManifest class
  package_manager.py: support ipk incremental image generation
  rootfs.py: support ipk incremental image generation
  rootfs.py: support BAD_RECOMMENDATIONS for ipk incremental image
    generation
  rootfs.py: tweak _multilib_sanity_test for ipk incremental image
    generation

 meta/lib/oe/manifest.py        |  57 +++++++++++++++++-
 meta/lib/oe/package_manager.py |  81 ++++++++++++++++++++++++--
 meta/lib/oe/rootfs.py          | 128 +++++++++++++++++++++++++++++++++++++++--
 3 files changed, 254 insertions(+), 12 deletions(-)

-- 
1.8.1.2