[PATCH 0/1] python-smartpm-native: prefer same arch when install

[PATCH 0/1] python-smartpm-native: prefer same arch when install

[Robert Yang]

The following changes since commit f787b688f2884ce3fa888b4041030538c7d2bf55:

  oeqa/utils/decorators: fix missing keyword arguments on decorators (2015-10-27 07:22:22 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib rbt/smart
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=rbt/smart

Robert Yang (1):
  python-smartpm-native: prefer same arch when install

 .../python-smartpm/smart-prefer-same-arch.patch    |  119 ++++++++++++++++++++
 meta/recipes-devtools/python/python-smartpm_git.bb |    2 +
 2 files changed, 121 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python-smartpm/smart-prefer-same-arch.patch

-- 
1.7.9.5

[PATCH 1/1] python-smartpm-native: prefer same arch when install

[Robert Yang]

We had made smart install multilib RDEPENDS correctly from
package_manager.py, but it couldn't handle RRECOMMANDS, this patch fix
the issue from python-smartpm-native, and make it work well.

The logic is: when pkg_A rdepends/rrecommands pkg_B, then let pkg_B use
pkg_A's arch when possible.

This patch fixed:
require conf/multilib.conf
MULTILIBS = "multilib:lib32"
DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
RPM_PREFER_ELF_ARCH = "1"
IMAGE_INSTALL_append = " lib32-connman-gnome"
MACHINE = "qemux86-64"

$ bitbake core-image-sato

Only 64bit loaders like libgdk-pixbuf-2.0-loader-jpeg and
libgdk-pixbuf-2.0-loader-png were installed before this patch since they
are in RRECOMMANDS, now both 32bit and 64bit are installed.

This patch is for native only to minimize the impact for this release,
the main problem is the "if attempt:", otherwise, more packages like
lib32-libc6-dev will be installed when populate_sdk, but only libc6-dev
were installed before. We need reconstruct the way that we install the
complentary packages in future release, the "if attempt:" make they work
together atm.

[YOCTO #8570]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 .../python-smartpm/smart-prefer-same-arch.patch    |  119 ++++++++++++++++++++
 meta/recipes-devtools/python/python-smartpm_git.bb |    2 +
 2 files changed, 121 insertions(+)
 create mode 100644 meta/recipes-devtools/python/python-smartpm/smart-prefer-same-arch.patch

diff --git a/meta/recipes-devtools/python/python-smartpm/smart-prefer-same-arch.patch b/meta/recipes-devtools/python/python-smartpm/smart-prefer-same-arch.patch
new file mode 100644
index 0000000..b1305b5
--- /dev/null
+++ b/meta/recipes-devtools/python/python-smartpm/smart-prefer-same-arch.patch
@@ -0,0 +1,119 @@
+From 1749592a76425449c1f3119fb14db3f5e14226ed Mon Sep 17 00:00:00 2001
+From: Robert Yang <liezhi.yang@windriver.com>
+Date: Tue, 27 Oct 2015 06:20:45 -0700
+Subject: [PATCH] smart/transaction.py: prefer same arch when install
+
+We had made smart install multilib RDEPENDS correctly from
+package_manager.py, but it couldn't handle RRECOMMANDS, this patch fix
+the issue from python-smartpm-native, and make it work well.
+
+The logic is: when pkg_A rdepends/rrecommands pkg_B, then let pkg_B use
+pkg_A's arch when possible.
+
+This patch fixed:
+require conf/multilib.conf
+MULTILIBS = "multilib:lib32"
+DEFAULTTUNE_virtclass-multilib-lib32 = "x86"
+RPM_PREFER_ELF_ARCH = "1"
+IMAGE_INSTALL_append = " lib32-connman-gnome"
+MACHINE = "qemux86-64"
+
+$ bitbake core-image-sato
+
+Only 64bit loaders like libgdk-pixbuf-2.0-loader-jpeg and
+libgdk-pixbuf-2.0-loader-png were installed before this patch since they
+are in RRECOMMANDS, now both 32bit and 64bit are installed.
+
+This patch is for native only to minimize the impact for this release,
+the main problem is the "if attempt:", otherwise, more packages like
+lib32-libc6-dev will be installed when populate_sdk, but only libc6-dev
+were installed before. We need reconstruct the way that we install the
+complentary packages in future release, the "if attempt:" make they work
+together atm.
+
+Upstream-Status: Inappropriate [oe specific]
+
+Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
+---
+ smart/transaction.py |   28 +++++++++++++++++++++++++---
+ 1 file changed, 25 insertions(+), 3 deletions(-)
+
+diff --git a/smart/transaction.py b/smart/transaction.py
+index f022aae..63a837c 100644
+--- a/smart/transaction.py
++++ b/smart/transaction.py
+@@ -22,6 +22,7 @@
+ from smart.const import INSTALL, REMOVE, UPGRADE, FIX, REINSTALL, KEEP, LOCKED_EXCLUDE, LOCKED_INSTALL, LOCKED_CONFLICT, LOCKED_CONFLICT_BY, LOCKED_NO_COEXIST, LOCKED_SYSCONF, LOCKED_REMOVE
+ from smart.cache import PreRequires, Package
+ from smart import *
++from smart.backends.rpm.rpmver import splitarch
+ 
+ def lock_reason(pkg, lockvalue):
+     try:
+@@ -618,6 +619,7 @@ class Transaction(object):
+                     self._remove(namepkg, changeset, locked, pending, depth)
+ 
+         # Install packages required by this one.
++        pkgarch = splitarch(pkg.version)[1]
+         for req in pkg.requires + pkg.recommends:
+ 
+             reqrequired = req in pkg.requires
+@@ -626,6 +628,8 @@ class Transaction(object):
+             prvpkgs = {}
+             lockedpkgs = {}
+             found = False
++            found_same_arch = False
++            same_arch_in_provides = False
+             for prv in req.providedby:
+                 for prvpkg in prv.packages:
+                     if not reqrequired:
+@@ -633,9 +637,19 @@ class Transaction(object):
+                             continue
+                         elif pkgconf.testFlag("ignore-recommends", prvpkg):
+                             continue
++                    prvpkgarch = splitarch(prvpkg.version)[1]
++                    if pkgarch == prvpkgarch:
++                        # Found a same arch in provides
++                        same_arch_in_provides = True
+                     if isinst(prvpkg):
+                         found = True
+-                        break
++                        if attempt:
++                            break
++                        if pkgarch == prvpkgarch:
++                            # Found the best match
++                            found_same_arch = True
++                            break
++                        continue
+                     if prvpkg not in locked:
+                         prvpkgs[prvpkg] = True
+                     else:
+@@ -643,7 +657,7 @@ class Transaction(object):
+                 else:
+                     continue
+                 break
+-            if found:
++            if found_same_arch or (found and attempt) or (found and not same_arch_in_provides):
+                 # Someone is already providing it. Good.
+                 continue
+ 
+@@ -672,7 +686,15 @@ class Transaction(object):
+ 
+                     # It's only a recommend, skip
+                     continue
+-
++            if len(prvpkgs) > 1 and not attempt:
++                prvarchs = set()
++                for prv in prvpkgs:
++                    prvarchs.add(splitarch(prv.version)[1])
++                if pkgarch in prvarchs:
++                    # Prefer the one which has the same arch
++                    for prv in prvpkgs.copy():
++                        if splitarch(prv.version)[1] != pkgarch:
++                                del prvpkgs[prv]
+             if len(prvpkgs) == 1:
+                 # Don't check locked here. prvpkgs was
+                 # already filtered above.
+-- 
+1.7.9.5
+
diff --git a/meta/recipes-devtools/python/python-smartpm_git.bb b/meta/recipes-devtools/python/python-smartpm_git.bb
index d6c378b..1ebf590 100644
--- a/meta/recipes-devtools/python/python-smartpm_git.bb
+++ b/meta/recipes-devtools/python/python-smartpm_git.bb
@@ -26,6 +26,8 @@ SRC_URI = "\
           file://smart-cache.py-getPackages-matches-name-version.patch \
          "
 
+SRC_URI_append_class-native = "file://smart-prefer-same-arch.patch"
+
 SRCREV = "407a7eca766431257dcd1da15175cc36a1bb22d0"
 PV = "1.5+git${SRCPV}"
 
-- 
1.7.9.5

Re: [PATCH 1/1] python-smartpm-native: prefer same arch when install

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1418 bytes --]

On 27 October 2015 at 14:05, Robert Yang <liezhi.yang@windriver.com> wrote:

> We had made smart install multilib RDEPENDS correctly from
> package_manager.py, but it couldn't handle RRECOMMANDS, this patch fix
> the issue from python-smartpm-native, and make it work well.
>
> The logic is: when pkg_A rdepends/rrecommands pkg_B, then let pkg_B use
> pkg_A's arch when possible.
>

I added this patch and rebuilt core-image-sato, then ran buildhistory-diff:

Changes to images/intel_corei7_64/glibc/core-image-sato
(installed-package-names.txt):
  lib32-ncurses-terminfo-base was added
  lib32-libjpeg9 was added
  lib32-glibc-gconv-iso8859-1 was added
  lib32-shared-mime-info was added
  lib32-libxml2 was added
  lib32-wpa-supplicant-cli was added
  lib32-libgdk-pixbuf-2.0-loader-png was added
  lib32-wpa-supplicant-passphrase was added
  lib32-pango-module-basic-fc was added
  lib32-bash was added
  lib32-libgdk-pixbuf-2.0-loader-jpeg was added
  lib32-libgdk-pixbuf-2.0-loader-gif was added
  lib32-libgdk-pixbuf-2.0-loader-xpm was added
  lib32-glibc-gconv was added

Some of those are good (gdk-pixbuf and pango), some bad (bash,
wpa-supplicant-cli), some probably bugs in packages that should be allarch
(termino, shared-mime-info).

I'm undecided what we do about this.  Fixing the dependency chains is good,
but we're definitely pulling in more than expected.

Ross

[-- Attachment #2: Type: text/html, Size: 2567 bytes --]

Re: [PATCH 1/1] python-smartpm-native: prefer same arch when install

[Mark Hatle]

On 10/27/15 1:45 PM, Burton, Ross wrote:
> 
> On 27 October 2015 at 14:05, Robert Yang <liezhi.yang@windriver.com
> <mailto:liezhi.yang@windriver.com>> wrote:
> 
>     We had made smart install multilib RDEPENDS correctly from
>     package_manager.py, but it couldn't handle RRECOMMANDS, this patch fix
>     the issue from python-smartpm-native, and make it work well.
> 
>     The logic is: when pkg_A rdepends/rrecommands pkg_B, then let pkg_B use
>     pkg_A's arch when possible.
> 
> 
> I added this patch and rebuilt core-image-sato, then ran buildhistory-diff:
> 
> Changes to images/intel_corei7_64/glibc/core-image-sato
> (installed-package-names.txt):
>   lib32-ncurses-terminfo-base was added
>   lib32-libjpeg9 was added
>   lib32-glibc-gconv-iso8859-1 was added
>   lib32-shared-mime-info was added
>   lib32-libxml2 was added
>   lib32-wpa-supplicant-cli was added
>   lib32-libgdk-pixbuf-2.0-loader-png was added
>   lib32-wpa-supplicant-passphrase was added
>   lib32-pango-module-basic-fc was added
>   lib32-bash was added
>   lib32-libgdk-pixbuf-2.0-loader-jpeg was added
>   lib32-libgdk-pixbuf-2.0-loader-gif was added
>   lib32-libgdk-pixbuf-2.0-loader-xpm was added
>   lib32-glibc-gconv was added
> 
> Some of those are good (gdk-pixbuf and pango), some bad (bash,
> wpa-supplicant-cli), some probably bugs in packages that should be allarch
> (termino, shared-mime-info).

bash, wpa-supplicant-cli is a good example of why this is an issue.  RPM luckily
says, well we've got two versions of bash, lets resolve this by following the
ELF resolution rule.  So while two packages may exist in the manifest, you won't
get any conflicts and space is likely not duplicated.

But you can't automatically say "give me any bash, but give me arch specific
versions of everything else".  The default in RPM is if I ask for 'bash', give
me any version of bash.. if I ask for 'bash.i586' I was -the- i586 version.

This is why the fix for this is either adjust the behavior (as Robert has done)
to suck in more dependencies and consider the ELF resolver to be adequate -- or
to add some kind of a hint to the recipe mechanisms that select dependencies
must be of the same arch type.  One we handle in the resolver, and one we could
handle when we generate the packages themselves.

I'm not sure there is an obvious solution, I prefer the later with a variant of
the former myself.  (Variant being if all things are equal we weight
dependencies to match the arch.. but if something is already in with that
provide we use it..)  Then the hint would cause the produced packages to be able
to say they need explicit versions of the plugins.

> I'm undecided what we do about this.  Fixing the dependency chains is good, but
> we're definitely pulling in more than expected.

I agree, I have a feeling what I wrote about is the right answer... but I don't
have any code to prove it.. and I'm not sure that we should be making that big
of a change right now.

Robert's patch SHOULD resolve the problem at the expense of a slightly larger
resolution manifest, which may be acceptable.

--Mark

> Ross
> 
> 

Re: [PATCH 1/1] python-smartpm-native: prefer same arch when install

[Robert Yang]

On 10/28/2015 02:45 AM, Burton, Ross wrote:
>
> On 27 October 2015 at 14:05, Robert Yang <liezhi.yang@windriver.com
> <mailto:liezhi.yang@windriver.com>> wrote:
>
>     We had made smart install multilib RDEPENDS correctly from
>     package_manager.py, but it couldn't handle RRECOMMANDS, this patch fix
>     the issue from python-smartpm-native, and make it work well.
>
>     The logic is: when pkg_A rdepends/rrecommands pkg_B, then let pkg_B use
>     pkg_A's arch when possible.
>
>
> I added this patch and rebuilt core-image-sato, then ran buildhistory-diff:
>
> Changes to images/intel_corei7_64/glibc/core-image-sato
> (installed-package-names.txt):
>    lib32-ncurses-terminfo-base was added
>    lib32-libjpeg9 was added
>    lib32-glibc-gconv-iso8859-1 was added
>    lib32-shared-mime-info was added
>    lib32-libxml2 was added
>    lib32-wpa-supplicant-cli was added
>    lib32-libgdk-pixbuf-2.0-loader-png was added
>    lib32-wpa-supplicant-passphrase was added
>    lib32-pango-module-basic-fc was added
>    lib32-bash was added
>    lib32-libgdk-pixbuf-2.0-loader-jpeg was added
>    lib32-libgdk-pixbuf-2.0-loader-gif was added
>    lib32-libgdk-pixbuf-2.0-loader-xpm was added
>    lib32-glibc-gconv was added
>
> Some of those are good (gdk-pixbuf and pango), some bad (bash,
> wpa-supplicant-cli), some probably bugs in packages that should be allarch
> (termino, shared-mime-info).

I think that lacking something so that it doesn't work is worse than have more
things but works ?

// Robert

>
> I'm undecided what we do about this.  Fixing the dependency chains is good, but
> we're definitely pulling in more than expected.
>
> Ross