[OE-core][PATCH 0/5] fix populate_sdk for package_deb

[OE-core][PATCH 0/5] fix populate_sdk for package_deb

[Jan Luebbe]

This series fixes bugs in lib/oe/package_manager which are triggered
when using populate_sdk in a distro using PACKAGE_CLASSES = "package_ipk".

The main issue is that apt install will try to remove already installed
packages which conflict against those requested in the current install
step. Is is triggered here by the coreutils conflict in
target-sdk-provides-dummy, which provides perl (amongst others).
As apt and dpkg depend on perl, this causes apt to try to remove itself,
failing at the essential package check. As this happens during the
complementary package installation, which is allowed to fail with a
warning, this leads to an SDK which is missing the -dev/-dbg packages.

This issue also affects zeus, so I'd be grateful if they could be
applied there as well, to avoid having to maintain a modified OE-core
locally. I've tested them on master and zeus.

Jan Luebbe (5):
  lib/oe/package_manager: make sure to not remove packages in apt
    install
  lib/oe/package_manager: fix handling of last package
  lib/oe/package_manager: collect provided package names when using debs
  lib/oe/package_manager: avoid installing provided packages via apt
  lib/oe/package_manager: don't try to rm /var/lib/opkg

 meta/lib/oe/package_manager.py | 37 +++++++++++++++++++++-------------
 1 file changed, 23 insertions(+), 14 deletions(-)

-- 
2.26.0.rc2

[OE-core][PATCH 1/5] lib/oe/package_manager: make sure to not remove packages in apt install

[Jan Luebbe]

apt install can decide to remove already installed packages if there are
conflicts. Avoid this by explicitly specifying --no-remove. This will
then cause a "E: Packages need to be removed but remove is disabled."
message.

Signed-off-by: Jan Luebbe <jlu@pengutronix.de>
---
 meta/lib/oe/package_manager.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 2ea8046332af..dfe4197ce1fb 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -1624,7 +1624,7 @@ class DpkgPM(OpkgDpkgPM):
 
         os.environ['APT_CONFIG'] = self.apt_conf_file
 
-        cmd = "%s %s install --force-yes --allow-unauthenticated %s" % \
+        cmd = "%s %s install --force-yes --allow-unauthenticated --no-remove %s" % \
               (self.apt_get_cmd, self.apt_args, ' '.join(pkgs))
 
         try:
-- 
2.26.0.rc2

[OE-core][PATCH 2/5] lib/oe/package_manager: fix handling of last package

[Jan Luebbe]

In commit 7d214b34e11dc57316ed5c1c7747c4601286f6d2, only the code in the
for loop was modified to store the pkgarch value. The code used if there
was no empty line at the end was not modified.

Instead of fixing the duplicated code, remove it and just make sure that
a final empty line is processed.

Signed-off-by: Jan Luebbe <jlu@pengutronix.de>
---
 meta/lib/oe/package_manager.py | 8 +-------
 1 file changed, 1 insertion(+), 7 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index dfe4197ce1fb..072372892732 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -41,7 +41,7 @@ def opkg_query(cmd_output):
     filename = ""
     dep = []
     pkgarch = ""
-    for line in cmd_output.splitlines():
+    for line in cmd_output.splitlines()+['']:
         line = line.rstrip()
         if ':' in line:
             if line.startswith("Package: "):
@@ -80,12 +80,6 @@ def opkg_query(cmd_output):
             dep = []
             pkgarch = ""
 
-    if pkg:
-        if not filename:
-            filename = "%s_%s_%s.ipk" % (pkg, ver, arch)
-        output[pkg] = {"arch":arch, "ver":ver,
-                "filename":filename, "deps": dep }
-
     return output
 
 def failed_postinsts_abort(pkgs, log_path):
-- 
2.26.0.rc2

[OE-core][PATCH 3/5] lib/oe/package_manager: collect provided package names when using debs

[Jan Luebbe]

This is needed for a later change to avoid installing packages which are
already provided by an installed package.

Signed-off-by: Jan Luebbe <jlu@pengutronix.de>
---
 meta/lib/oe/package_manager.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 072372892732..e862915b1b75 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -40,6 +40,7 @@ def opkg_query(cmd_output):
     ver = ""
     filename = ""
     dep = []
+    prov = []
     pkgarch = ""
     for line in cmd_output.splitlines()+['']:
         line = line.rstrip()
@@ -64,6 +65,10 @@ def opkg_query(cmd_output):
                     dep.append("%s [REC]" % recommend)
             elif line.startswith("PackageArch: "):
                 pkgarch = line.split(": ")[1]
+            elif line.startswith("Provides: "):
+                provides = verregex.sub('', line.split(": ")[1])
+                for provide in provides.split(", "):
+                    prov.append(provide)
 
         # When there is a blank line save the package information
         elif not line:
@@ -72,12 +77,13 @@ def opkg_query(cmd_output):
                 filename = "%s_%s_%s.ipk" % (pkg, ver, arch)
             if pkg:
                 output[pkg] = {"arch":arch, "ver":ver,
-                        "filename":filename, "deps": dep, "pkgarch":pkgarch }
+                        "filename":filename, "deps": dep, "pkgarch":pkgarch, "provs": prov}
             pkg = ""
             arch = ""
             ver = ""
             filename = ""
             dep = []
+            prov = []
             pkgarch = ""
 
     return output
@@ -355,7 +361,7 @@ class DpkgPkgsList(PkgsList):
                "--admindir=%s/var/lib/dpkg" % self.rootfs_dir,
                "-W"]
 
-        cmd.append("-f=Package: ${Package}\nArchitecture: ${PackageArch}\nVersion: ${Version}\nFile: ${Package}_${Version}_${Architecture}.deb\nDepends: ${Depends}\nRecommends: ${Recommends}\n\n")
+        cmd.append("-f=Package: ${Package}\nArchitecture: ${PackageArch}\nVersion: ${Version}\nFile: ${Package}_${Version}_${Architecture}.deb\nDepends: ${Depends}\nRecommends: ${Recommends}\nProvides: ${Provides}\n\n")
 
         try:
             cmd_output = subprocess.check_output(cmd, stderr=subprocess.STDOUT).strip().decode("utf-8")
-- 
2.26.0.rc2

[OE-core][PATCH 4/5] lib/oe/package_manager: avoid installing provided packages via apt

[Jan Luebbe]

If there already is a package providing (and conflicting against)
packages what should be installed, apt will try remove the conflicting
package (target-sdk-provides-dummy) and any that depend on it (like apt
and dpkg). This usually fails because of the protection of essential
packages. In that case, no -dev/-dbg packages are installed to the SDK.

Avoid this problem by checking which packages are already provided and
removing them from the list to be installed. Also sort the list to make
it easier to read when debugging.

Signed-off-by: Jan Luebbe <jlu@pengutronix.de>
---
 meta/lib/oe/package_manager.py | 14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index e862915b1b75..2a5d7ef539e6 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -581,6 +581,11 @@ class PackageManager(object, metaclass=ABCMeta):
         # oe-pkgdata-util reads it from a file
         with tempfile.NamedTemporaryFile(mode="w+", prefix="installed-pkgs") as installed_pkgs:
             pkgs = self.list_installed()
+
+            provided_pkgs = set()
+            for pkg in pkgs.values():
+                provided_pkgs |= set(pkg.get('provs', []))
+
             output = oe.utils.format_pkg_list(pkgs, "arch")
             installed_pkgs.write(output)
             installed_pkgs.flush()
@@ -592,10 +597,15 @@ class PackageManager(object, metaclass=ABCMeta):
             if exclude:
                 cmd.extend(['--exclude=' + '|'.join(exclude.split())])
             try:
-                bb.note("Installing complementary packages ...")
                 bb.note('Running %s' % cmd)
                 complementary_pkgs = subprocess.check_output(cmd, stderr=subprocess.STDOUT).decode("utf-8")
-                self.install(complementary_pkgs.split(), attempt_only=True)
+                complementary_pkgs = set(complementary_pkgs.split())
+                skip_pkgs = sorted(complementary_pkgs & provided_pkgs)
+                install_pkgs = sorted(complementary_pkgs - provided_pkgs)
+                bb.note("Installing complementary packages ... %s (skipped already provided packages %s)" % (
+                    ' '.join(install_pkgs),
+                    ' '.join(skip_pkgs)))
+                self.install(install_pkgs, attempt_only=True)
             except subprocess.CalledProcessError as e:
                 bb.fatal("Could not compute complementary packages list. Command "
                          "'%s' returned %d:\n%s" %
-- 
2.26.0.rc2

[OE-core][PATCH 5/5] lib/oe/package_manager: don't try to rm /var/lib/opkg

[Jan Luebbe]

As opkglibdir starts with a /, os.path.join will ignore
self.target_rootfs, leading to an attempt to remove /var/lib/opkg.

This only fails if it exists on the host, explaining why this remained
undiscovered for long.

Signed-off-by: Jan Luebbe <jlu@pengutronix.de>
---
 meta/lib/oe/package_manager.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/lib/oe/package_manager.py b/meta/lib/oe/package_manager.py
index 2a5d7ef539e6..b0660411eaf5 100644
--- a/meta/lib/oe/package_manager.py
+++ b/meta/lib/oe/package_manager.py
@@ -1796,8 +1796,7 @@ class DpkgPM(OpkgDpkgPM):
             open(os.path.join(target_dpkg_dir, "available"), "w+").close()
 
     def remove_packaging_data(self):
-        bb.utils.remove(os.path.join(self.target_rootfs,
-                                     self.d.getVar('opkglibdir')), True)
+        bb.utils.remove(self.target_rootfs + self.d.getVar('opkglibdir'), True)
         bb.utils.remove(self.target_rootfs + "/var/lib/dpkg/", True)
 
     def fix_broken_dependencies(self):
-- 
2.26.0.rc2

Re: [OE-core][PATCH 5/5] lib/oe/package_manager: don't try to rm /var/lib/opkg

[Peter Kjellerstedt]

> -----Original Message-----
> From: openembedded-core@lists.openembedded.org <openembedded-
> core@lists.openembedded.org> On Behalf Of Jan Luebbe
> Sent: den 1 april 2020 16:58
> To: openembedded-core@lists.openembedded.org
> Cc: Jan Luebbe <jlu@pengutronix.de>
> Subject: [OE-core][PATCH 5/5] lib/oe/package_manager: don't try to rm
> /var/lib/opkg
> 
> As opkglibdir starts with a /, os.path.join will ignore
> self.target_rootfs, leading to an attempt to remove /var/lib/opkg.
> 
> This only fails if it exists on the host, explaining why this remained
> undiscovered for long.
> 
> Signed-off-by: Jan Luebbe <jlu@pengutronix.de>
> ---
>  meta/lib/oe/package_manager.py | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/meta/lib/oe/package_manager.py
> b/meta/lib/oe/package_manager.py
> index 2a5d7ef539e6..b0660411eaf5 100644
> --- a/meta/lib/oe/package_manager.py
> +++ b/meta/lib/oe/package_manager.py
> @@ -1796,8 +1796,7 @@ class DpkgPM(OpkgDpkgPM):
>              open(os.path.join(target_dpkg_dir, "available"), "w+").close()
> 
>      def remove_packaging_data(self):
> -        bb.utils.remove(os.path.join(self.target_rootfs,
> -                                     self.d.getVar('opkglibdir')), True)
> +        bb.utils.remove(self.target_rootfs + self.d.getVar('opkglibdir'), True)

Or you could change it to use oe.path.join() instead:

        bb.utils.remove(oe.path.join(self.target_rootfs,
                                     self.d.getVar('opkglibdir')), True)


>          bb.utils.remove(self.target_rootfs + "/var/lib/dpkg/", True)
> 
>      def fix_broken_dependencies(self):
> --
> 2.26.0.rc2

//Peter