[PATCH 0/2] Don't fool around with rpm __db.00* files

[PATCH 0/2] Don't fool around with rpm __db.00* files

[Laurentiu Palcu]

We were deleting/creating those files too many times during the rootfs creation
process. That led to various rpmdb related issues. Instead, remove those files
at the end of the rootfs creation process which seems the most sane thing to do.

laurentiu

The following changes since commit 8c3eb5ee4582b6f6d489549290937657f37fc19e:

  packagegroup-toolset-native: Update after ocf-linux -> cryptodev-linux change (2014-03-27 19:53:30 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib lpalcu/b6049_do_rootfs_crash
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lpalcu/b6049_do_rootfs_crash

Laurentiu Palcu (2):
  package_manager.py: leave the __db.00* files in place
  rootfs.py: add new cleanup method

 meta/lib/oe/package_manager.py |    8 +-------
 meta/lib/oe/rootfs.py          |   25 +++++++++++++++++++++++++
 2 files changed, 26 insertions(+), 7 deletions(-)

-- 
1.7.9.5

[PATCH 1/2] package_manager.py: leave the __db.00* files in place

[Laurentiu Palcu]

Do not delete the __db.00* files in the PackageManager class. Leave this
operation up to the client classes. One side effect of this deletion was
the following message appearing in the output of the next rpm command
executed:

rpmdb: BDB1540 configured environment flags incompatible with existing
environment

We might also gain some time here by not deleting/creating those files
very often.

[YOCTO #6049]

Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
---
 meta/lib/oe/package_manager.py |    8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 764ab72..a8360fe 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -299,9 +299,6 @@ class RpmPkgsList(PkgsList):
             # bb.note(cmd)
             tmp_output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).strip()
 
-            rpm_db_locks = glob.glob('%s/var/lib/rpm/__db.*' % self.rootfs_dir)
-            for f in rpm_db_locks:
-                bb.utils.remove(f, True)
         except subprocess.CalledProcessError as e:
             bb.fatal("Cannot get the installed packages list. Command '%s' "
                      "returned %d:\n%s" % (cmd, e.returncode, e.output))
@@ -1101,7 +1098,6 @@ class RpmPM(PackageManager):
             output = subprocess.check_output(cmd, stderr=subprocess.STDOUT, shell=True).strip()
             bb.note(output)
             os.chmod(saved_dir, 0755)
-            self._unlock_rpm_db()
         except subprocess.CalledProcessError as e:
             bb.fatal("Invoke save_rpmpostinst failed. Command '%s' "
                      "returned %d:\n%s" % (cmd, e.returncode, e.output))
@@ -1117,14 +1113,12 @@ class RpmPM(PackageManager):
             self._invoke_smart('flag --set ignore-recommends %s' % i)
         self._invoke_smart('channel --add rpmsys type=rpm-sys -y')
 
-        self._unlock_rpm_db()
-
     '''
     The rpm db lock files were produced after invoking rpm to query on
     build system, and they caused the rpm on target didn't work, so we
     need to unlock the rpm db by removing the lock files.
     '''
-    def _unlock_rpm_db(self):
+    def unlock_rpm_db(self):
         # Remove rpm db lock files
         rpm_db_locks = glob.glob('%s/var/lib/rpm/__db.*' % self.target_rootfs)
         for f in rpm_db_locks:
-- 
1.7.9.5

[PATCH 2/2] rootfs.py: add new cleanup method

[Laurentiu Palcu]

This commit adds a new _cleanup() internal method that will be called at
the end of rootfs creation, so that each backend can delete various
files that were probably generated during rootfs postprocess execution,
etc.

[YOCTO #6049]

Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
---
 meta/lib/oe/rootfs.py |   25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/meta/lib/oe/rootfs.py b/meta/lib/oe/rootfs.py
index 0e6c8bc..3eac3c9 100644
--- a/meta/lib/oe/rootfs.py
+++ b/meta/lib/oe/rootfs.py
@@ -50,6 +50,15 @@ class Rootfs(object):
     def _handle_intercept_failure(self, failed_script):
         pass
 
+    """
+    The _cleanup() method should be used to clean-up stuff that we don't really
+    want to end up on target. For example, in the case of RPM, the DB locks.
+    The method is called, once, at the end of create() method.
+    """
+    @abstractmethod
+    def _cleanup(self):
+        pass
+
     def _exec_shell_cmd(self, cmd):
         fakerootcmd = self.d.getVar('FAKEROOT', True)
         if fakerootcmd is not None:
@@ -117,6 +126,8 @@ class Rootfs(object):
 
         self._generate_kernel_module_deps()
 
+        self._cleanup()
+
     def _uninstall_uneeded(self):
         if base_contains("IMAGE_FEATURES", "package-management",
                          True, False, self.d):
@@ -358,6 +369,13 @@ class RpmRootfs(Rootfs):
         for pkg in registered_pkgs.split():
             self.pm.save_rpmpostinst(pkg)
 
+    def _cleanup(self):
+        # during the execution of postprocess commands, rpm is called several
+        # times to get the files installed, dependencies, etc. This creates the
+        # __db.00* (Berkeley DB files that hold locks, rpm specific environment
+        # settings, etc.), that should not get into the final rootfs
+        self.pm.unlock_rpm_db()
+
 
 class DpkgRootfs(Rootfs):
     def __init__(self, d, manifest_dir):
@@ -431,6 +449,9 @@ class DpkgRootfs(Rootfs):
     def _log_check(self):
         pass
 
+    def _cleanup(self):
+        pass
+
 
 class OpkgRootfs(Rootfs):
     def __init__(self, d, manifest_dir):
@@ -694,6 +715,10 @@ class OpkgRootfs(Rootfs):
     def _log_check(self):
         pass
 
+    def _cleanup(self):
+        pass
+
+
 def create_rootfs(d, manifest_dir=None):
     env_bkp = os.environ.copy()
 
-- 
1.7.9.5