[PATCH 0/5] Fixes for image construction

[PATCH 0/5] Fixes for image construction

[Paul Eggleton]

The following changes since commit 2a775ebbb175dd70fc7228607c306d4ccb9e4ba4:

  net-tools: Fix rerunning of do_patch task (2015-01-23 14:32:40 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/imagefixes
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/imagefixes

Paul Eggleton (5):
  classes/image: drop unused default of MACHINE_POSTPROCESS_COMMAND
  lib/oe/rootfs.py: rename _uninstall_uneeded to fix spelling error
  classes/image: skip recipe on invalid IMAGE_FEATURES item
  classes/image: ensure uninstalled packages do not appear in manifests
  lib/oe/image.py: add error checking for missing IMAGE_CMD

 meta/classes/buildhistory.bbclass |  5 ++--
 meta/classes/image.bbclass        |  8 ++++--
 meta/lib/oe/image.py              |  6 +++-
 meta/lib/oe/rootfs.py             | 59 +++++++++++++++++++++------------------
 4 files changed, 45 insertions(+), 33 deletions(-)

-- 
1.9.3

[PATCH 1/5] classes/image: drop unused default of MACHINE_POSTPROCESS_COMMAND

[Paul Eggleton]

In the daisy (1.6) timeframe, when we rewrote the image construction in
Python, we neglected to reimplement the support for the little used and
undocumented variable MACHINE_POSTPROCESS_COMMAND, and apparently nobody
noticed. We have a better method for implementing machine-specific image
formats that is in wider use (i.e. add a custom class which implements
the new image type, add the class to IMAGE_CLASSES and the type to
IMAGE_FSTYPES), and we now also have wic. Thus it makes more sense to
just call this variable unsupported now and drop the sole remaining
reference to it.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/image.bbclass | 1 -
 1 file changed, 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 22b6970..6bd3e03 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -156,7 +156,6 @@ IMAGE_CLASSES += "image_types"
 inherit ${IMAGE_CLASSES}
 
 IMAGE_POSTPROCESS_COMMAND ?= ""
-MACHINE_POSTPROCESS_COMMAND ?= ""
 # Allow dropbear/openssh to accept logins from accounts with an empty password string if debug-tweaks is enabled
 ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "debug-tweaks", "ssh_allow_empty_password; ", "",d)}'
 # Enable postinst logging if debug-tweaks is enabled
-- 
1.9.3

[PATCH 2/5] lib/oe/rootfs.py: rename _uninstall_uneeded to fix spelling error

[Paul Eggleton]

Rename it to _uninstall_unneeded.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/rootfs.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 72d32f7..536efcc 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -118,7 +118,7 @@ class Rootfs(object):
         if self.d.getVar('USE_DEVFS', True) != "1":
             self._create_devfs()
 
-        self._uninstall_uneeded()
+        self._uninstall_unneeded()
 
         self._insert_feed_uris()
 
@@ -128,7 +128,7 @@ class Rootfs(object):
 
         self._cleanup()
 
-    def _uninstall_uneeded(self):
+    def _uninstall_unneeded(self):
         # Remove unneeded init script symlinks
         delayed_postinsts = self._get_delayed_postinsts()
         if delayed_postinsts is None:
-- 
1.9.3

[PATCH 3/5] classes/image: skip recipe on invalid IMAGE_FEATURES item

[Paul Eggleton]

If you add an item to EXTRA_IMAGE_FEATURES in your local.conf that is
not supported by image.bbclass itself (such as "tools-sdk" which is
implemented in core-image.bbclass), it can be somewhat annoying to have
the parse fall over if you have a recipe that inherits image.bbclass
only. Change the error from bb.fatal to skip the recipe instead so that
you only see the error when attempting to build the recipe, plus add a
bit of logic to report if the feature is coming in via
EXTRA_IMAGE_FEATURES.

Fixes [YOCTO #5023].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/image.bbclass | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 6bd3e03..62b214a 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -52,7 +52,10 @@ def check_image_features(d):
     features = set(oe.data.typed_value('IMAGE_FEATURES', d))
     for feature in features:
         if feature not in valid_features:
-            bb.fatal("'%s' in IMAGE_FEATURES is not a valid image feature. Valid features: %s" % (feature, ' '.join(valid_features)))
+            if bb.utils.contains('EXTRA_IMAGE_FEATURES', feature, True, False, d):
+                raise bb.parse.SkipRecipe("'%s' in IMAGE_FEATURES (added via EXTRA_IMAGE_FEATURES) is not a valid image feature. Valid features: %s" % (feature, ' '.join(valid_features)))
+            else:
+                raise bb.parse.SkipRecipe("'%s' in IMAGE_FEATURES is not a valid image feature. Valid features: %s" % (feature, ' '.join(valid_features)))
 
 IMAGE_INSTALL ?= ""
 IMAGE_INSTALL[type] = "list"
-- 
1.9.3

[PATCH 4/5] classes/image: ensure uninstalled packages do not appear in manifests

[Paul Eggleton]

Since the rewrite of the image construction code in python a few
releases ago, we remove a couple of packages from the image as one of
the final steps when constructing the image (notably update-rc.d and
run-postinsts).  However, because of the order of operations, these
packages are still listed both in the buildhistory
installed_package*.txt files and in the manifest file created next to
the image, which is wrong.

There were two possible solutions to this: (1) change the order such
that the uninstallation occurs before calling ROOTFS_POSTPROCESS_COMMAND
or (2) add another hook variable in such that we can have the
package list collection code run at the right time. Because it's
currently possible (but very much not recommended) to install additional
packages within ROOTFS_POSTPROCESS_COMMAND, which may have postinstall
scripts and thus require the packages we would otherwise uninstall if we
were to take option 1, option 2 is really the least likely to cause
problems. Therefore, add ROOTFS_POSTUNINSTALL_COMMAND and make the image
and buildhistory classes use it.

Fixes [YOCTO #6479].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/buildhistory.bbclass |  5 ++--
 meta/classes/image.bbclass        |  2 +-
 meta/lib/oe/rootfs.py             | 55 +++++++++++++++++++++------------------
 3 files changed, 34 insertions(+), 28 deletions(-)

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 2b5f84a..90cfe4f 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -484,8 +484,9 @@ END
 	echo "SDKSIZE = $sdksize" >> ${BUILDHISTORY_DIR_SDK}/sdk-info.txt
 }
 
-# By prepending we get in before the removal of packaging files
-ROOTFS_POSTPROCESS_COMMAND =+ " buildhistory_list_installed_image ;\
+# By using ROOTFS_POSTUNINSTALL_COMMAND we get in after uninstallation of
+# unneeded packages but before the removal of packaging files
+ROOTFS_POSTUNINSTALL_COMMAND += " buildhistory_list_installed_image ;\
                                 buildhistory_get_image_installed ; "
 
 IMAGE_POSTPROCESS_COMMAND += " buildhistory_get_imageinfo ; "
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 62b214a..56dbf52 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -165,7 +165,7 @@ ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "debug-twe
 ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("IMAGE_FEATURES", "debug-tweaks", "postinst_enable_logging; ", "",d)}'
 # Write manifest
 IMAGE_MANIFEST = "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.manifest"
-ROOTFS_POSTPROCESS_COMMAND =+ "write_image_manifest ; "
+ROOTFS_POSTUNINSTALL_COMMAND =+ "write_image_manifest ; "
 # Set default postinst log file
 POSTINST_LOGFILE ?= "${localstatedir}/log/postinstall.log"
 # Set default target for systemd images
diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 536efcc..30d164c 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -137,34 +137,39 @@ class Rootfs(object):
                                       self.d.getVar('IMAGE_ROOTFS', True),
                                       "run-postinsts", "remove"])
 
-        # Remove unneeded package-management related components
-        if bb.utils.contains("IMAGE_FEATURES", "package-management",
-                         True, False, self.d):
-            return
+        runtime_pkgmanage = bb.utils.contains("IMAGE_FEATURES", "package-management",
+                         True, False, self.d)
+        if not runtime_pkgmanage:
+            # Remove components that we don't need if we're not going to install
+            # additional packages at runtime
+            if delayed_postinsts is None:
+                installed_pkgs_dir = self.d.expand('${WORKDIR}/installed_pkgs.txt')
+                pkgs_to_remove = list()
+                with open(installed_pkgs_dir, "r+") as installed_pkgs:
+                    pkgs_installed = installed_pkgs.read().split('\n')
+                    for pkg_installed in pkgs_installed[:]:
+                        pkg = pkg_installed.split()[0]
+                        if pkg in ["update-rc.d",
+                                "base-passwd",
+                                self.d.getVar("ROOTFS_BOOTSTRAP_INSTALL", True)
+                                ]:
+                            pkgs_to_remove.append(pkg)
+                            pkgs_installed.remove(pkg_installed)
+
+                if len(pkgs_to_remove) > 0:
+                    self.pm.remove(pkgs_to_remove, False)
+                    # Update installed_pkgs.txt
+                    open(installed_pkgs_dir, "w+").write('\n'.join(pkgs_installed))
 
-        if delayed_postinsts is None:
-            installed_pkgs_dir = self.d.expand('${WORKDIR}/installed_pkgs.txt')
-            pkgs_to_remove = list()
-            with open(installed_pkgs_dir, "r+") as installed_pkgs:
-                pkgs_installed = installed_pkgs.read().split('\n')
-                for pkg_installed in pkgs_installed[:]:
-                    pkg = pkg_installed.split()[0]
-                    if pkg in ["update-rc.d",
-                               "base-passwd",
-                               self.d.getVar("ROOTFS_BOOTSTRAP_INSTALL", True)
-                               ]:
-                        pkgs_to_remove.append(pkg)
-                        pkgs_installed.remove(pkg_installed)
-
-            if len(pkgs_to_remove) > 0:
-                self.pm.remove(pkgs_to_remove, False)
-                # Update installed_pkgs.txt
-                open(installed_pkgs_dir, "w+").write('\n'.join(pkgs_installed))
+            else:
+                self._save_postinsts()
 
-        else:
-            self._save_postinsts()
+        post_uninstall_cmds = self.d.getVar("ROOTFS_POSTUNINSTALL_COMMAND", True)
+        execute_pre_post_process(self.d, post_uninstall_cmds)
 
-        self.pm.remove_packaging_data()
+        if not runtime_pkgmanage:
+            # Remove the package manager data files
+            self.pm.remove_packaging_data()
 
     def _run_intercepts(self):
         intercepts_dir = os.path.join(self.d.getVar('WORKDIR', True),
-- 
1.9.3

[PATCH 5/5] lib/oe/image.py: add error checking for missing IMAGE_CMD

[Paul Eggleton]

An invalid value in IMAGE_FSTYPES was triggering a traceback. Add a
check and a reasonable error message instead.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/lib/oe/image.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/meta/lib/oe/image.py b/meta/lib/oe/image.py
index f9c8f84..0ce303d 100644
--- a/meta/lib/oe/image.py
+++ b/meta/lib/oe/image.py
@@ -296,7 +296,11 @@ class Image(ImageDepGraph):
                 bb.data.update_data(localdata)
                 localdata.setVar('type', type)
 
-                cmds.append("\t" + localdata.getVar("IMAGE_CMD", True))
+                image_cmd = localdata.getVar("IMAGE_CMD", True)
+                if image_cmd:
+                    cmds.append("\t" + image_cmd)
+                else:
+                    bb.fatal("No IMAGE_CMD defined for IMAGE_FSTYPES entry '%s' - possibly invalid type name or missing support class" % type)
                 cmds.append(localdata.expand("\tcd ${DEPLOY_DIR_IMAGE}"))
 
                 if type in cimages:
-- 
1.9.3