[PATCH 0/5] shlibs providers improvements

[PATCH 0/5] shlibs providers improvements

[Martin Jansa]

Fixes for [YOCTO #4628]

Richard had few more ideas to improve it even more, but this is good start.

Most important changes:
* PRIVATE_LIBS aren't duplicitly added to RDEPENDS (when recipe provides
  own libfoo it shouldn't get runtime dependency on global libfoo shlib provider)
* warning is shown for undeterministic providers (multiple packages
  recorded to provide the same library) - recipe maintainers can decide
  if such library should be considered private or make sure that only one
  package is built (most common case are gst plugins where 0.10 and 1.* provide
  the same gst modules, but as long as nobody tries to linke directly to some
  gst plugin we shouldn't list them as shlib providers).
* list files which require given shlib, often I was surprised to see that
  something got runtime dependency on libfoo, showing which binary is linked to
  it makes it easier to debug when you have only log.do_package and package
  directory is already gone (e.g. analyzing test-dependencies.sh script outputs)

The following changes since commit 8163854adf87ac42a8f08ee25685d0ce1efb4724:

  oe-selftest: separated the SStateBase and SStateTests in different modules (2014-01-16 12:18:44 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib jansa/shlib-providers
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=jansa/shlib-providers

Martin Jansa (5):
  package.bbclass: move reading shlibs providers to separate function
  package.bbclass: show warning when package is trying to provide
    already provided shlib
  package.bbclass: add SHLIBSSEARCHDIRS to define where to search for
    shlib providers
  package.bbclass: Don't search for prividers of PRIVATE_LIBS
  package.bbclass: Show which files require given dependency in debug
    output

 meta/classes/package.bbclass | 102 +++++++++++++++++++++++++++++++------------
 1 file changed, 75 insertions(+), 27 deletions(-)

-- 
1.8.5.2

[PATCH 1/5] package.bbclass: move reading shlibs providers to separate function

[Martin Jansa]

* prepare for reading shlibs providers only from dependency tree of
  current recipe

[YOCTO #4628]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/classes/package.bbclass | 43 +++++++++++++++++++++++--------------------
 1 file changed, 23 insertions(+), 20 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 768047c..7dcec5e 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1352,6 +1352,28 @@ python package_do_shlibs() {
     # Take shared lock since we're only reading, not writing
     lf = bb.utils.lockfile(d.expand("${PACKAGELOCK}"))
 
+    def read_shlib_providers():
+        list_re = re.compile('^(.*)\.list$')
+        # Go from least to most specific since the last one found wins
+        for dir in reversed(shlibs_dirs):
+            if not os.path.exists(dir):
+                continue
+            for file in os.listdir(dir):
+                m = list_re.match(file)
+                if m:
+                    dep_pkg = m.group(1)
+                    fd = open(os.path.join(dir, file))
+                    lines = fd.readlines()
+                    fd.close()
+                    ver_file = os.path.join(dir, dep_pkg + '.ver')
+                    lib_ver = None
+                    if os.path.exists(ver_file):
+                        fd = open(ver_file)
+                        lib_ver = fd.readline().rstrip()
+                        fd.close()
+                    for l in lines:
+                        shlib_provider[l.rstrip()] = (dep_pkg, lib_ver)
+
     def linux_so(file):
         needs_ldconfig = False
         cmd = d.getVar('OBJDUMP', True) + " -p " + pipes.quote(file) + " 2>/dev/null"
@@ -1496,26 +1518,7 @@ python package_do_shlibs() {
             postinst += d.getVar('ldconfig_postinst_fragment', True)
             d.setVar('pkg_postinst_%s' % pkg, postinst)
 
-    list_re = re.compile('^(.*)\.list$')
-    # Go from least to most specific since the last one found wins
-    for dir in reversed(shlibs_dirs):
-        if not os.path.exists(dir):
-            continue
-        for file in os.listdir(dir):
-            m = list_re.match(file)
-            if m:
-                dep_pkg = m.group(1)
-                fd = open(os.path.join(dir, file))
-                lines = fd.readlines()
-                fd.close()
-                ver_file = os.path.join(dir, dep_pkg + '.ver')
-                lib_ver = None
-                if os.path.exists(ver_file):
-                    fd = open(ver_file)
-                    lib_ver = fd.readline().rstrip()
-                    fd.close()
-                for l in lines:
-                    shlib_provider[l.rstrip()] = (dep_pkg, lib_ver)
+    read_shlib_providers()
 
     bb.utils.unlockfile(lf)
 
-- 
1.8.5.2

[PATCH 2/5] package.bbclass: show warning when package is trying to provide already provided shlib

[Martin Jansa]

* move read_shlib_providers before registering package as provider
  and don't change provider if it already exists, show warning instead

[YOCTO #4628]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/classes/package.bbclass | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 7dcec5e..aa8156d 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1356,6 +1356,7 @@ python package_do_shlibs() {
         list_re = re.compile('^(.*)\.list$')
         # Go from least to most specific since the last one found wins
         for dir in reversed(shlibs_dirs):
+            bb.debug(2, "Reading shlib providers in %s" % (dir))
             if not os.path.exists(dir):
                 continue
             for file in os.listdir(dir):
@@ -1471,6 +1472,8 @@ python package_do_shlibs() {
 
     needed = {}
     shlib_provider = {}
+    read_shlib_providers()
+
     for pkg in packages.split():
         private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or d.getVar('PRIVATE_LIBS', True)
         needs_ldconfig = False
@@ -1504,6 +1507,12 @@ python package_do_shlibs() {
         if len(sonames):
             fd = open(shlibs_file, 'w')
             for s in sonames:
+                if s in shlib_provider:
+                    (old_pkg, old_pkgver) = shlib_provider[s]
+                    if old_pkg != pkg:
+                        bb.warn('%s-%s is already registered as shlib provider for %s, ignoring %s-%s trying to register the same' % (old_pkg, old_pkgver, s, pkg, pkgver))
+                        continue
+                bb.debug(1, 'registering %s-%s as shlib provider for %s' % (pkg, pkgver, s))
                 fd.write(s + '\n')
                 shlib_provider[s] = (pkg, pkgver)
             fd.close()
@@ -1517,8 +1526,7 @@ python package_do_shlibs() {
                 postinst = '#!/bin/sh\n'
             postinst += d.getVar('ldconfig_postinst_fragment', True)
             d.setVar('pkg_postinst_%s' % pkg, postinst)
-
-    read_shlib_providers()
+        bb.debug(1, 'LIBNAMES: pkg %s sonames %s' % (pkg, sonames))
 
     bb.utils.unlockfile(lf)
 
-- 
1.8.5.2

Re: [PATCH 2/5] package.bbclass: show warning when package is trying to provide already provided shlib

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2865 bytes --]

On Sat, Jan 18, 2014 at 03:02:07PM +0100, Martin Jansa wrote:
> * move read_shlib_providers before registering package as provider
>   and don't change provider if it already exists, show warning instead
> 
> [YOCTO #4628]
> 
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  meta/classes/package.bbclass | 12 ++++++++++--
>  1 file changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index 7dcec5e..aa8156d 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -1356,6 +1356,7 @@ python package_do_shlibs() {
>          list_re = re.compile('^(.*)\.list$')
>          # Go from least to most specific since the last one found wins
>          for dir in reversed(shlibs_dirs):
> +            bb.debug(2, "Reading shlib providers in %s" % (dir))
>              if not os.path.exists(dir):
>                  continue
>              for file in os.listdir(dir):
> @@ -1471,6 +1472,8 @@ python package_do_shlibs() {
>  
>      needed = {}
>      shlib_provider = {}
> +    read_shlib_providers()
> +
>      for pkg in packages.split():
>          private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or d.getVar('PRIVATE_LIBS', True)
>          needs_ldconfig = False
> @@ -1504,6 +1507,12 @@ python package_do_shlibs() {
>          if len(sonames):
>              fd = open(shlibs_file, 'w')
>              for s in sonames:
> +                if s in shlib_provider:
> +                    (old_pkg, old_pkgver) = shlib_provider[s]
> +                    if old_pkg != pkg:
> +                        bb.warn('%s-%s is already registered as shlib provider for %s, ignoring %s-%s trying to register the same' % (old_pkg, old_pkgver, s, pkg, pkgver))
> +                        continue

I'll drop continue and update warning message, changing registered
provider for last built is still undeterministic but slightly better
e.g. when changing packaging and the shlib is moved between packages
(I've run into this in efl, when moving to split packages and they were
still getting runtime dependency on old efl package).

> +                bb.debug(1, 'registering %s-%s as shlib provider for %s' % (pkg, pkgver, s))
>                  fd.write(s + '\n')
>                  shlib_provider[s] = (pkg, pkgver)
>              fd.close()
> @@ -1517,8 +1526,7 @@ python package_do_shlibs() {
>                  postinst = '#!/bin/sh\n'
>              postinst += d.getVar('ldconfig_postinst_fragment', True)
>              d.setVar('pkg_postinst_%s' % pkg, postinst)
> -
> -    read_shlib_providers()
> +        bb.debug(1, 'LIBNAMES: pkg %s sonames %s' % (pkg, sonames))
>  
>      bb.utils.unlockfile(lf)
>  
> -- 
> 1.8.5.2
> 

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

[PATCHv2 2/5] package.bbclass: show warning when package is trying to provide already provided shlib

[Martin Jansa]

* move read_shlib_providers before registering package as provider
  and don't change provider if it already exists, show warning instead

[YOCTO #4628]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/classes/package.bbclass | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 7dcec5e..9b511a6 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1356,6 +1356,7 @@ python package_do_shlibs() {
         list_re = re.compile('^(.*)\.list$')
         # Go from least to most specific since the last one found wins
         for dir in reversed(shlibs_dirs):
+            bb.debug(2, "Reading shlib providers in %s" % (dir))
             if not os.path.exists(dir):
                 continue
             for file in os.listdir(dir):
@@ -1471,6 +1472,8 @@ python package_do_shlibs() {
 
     needed = {}
     shlib_provider = {}
+    read_shlib_providers()
+
     for pkg in packages.split():
         private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or d.getVar('PRIVATE_LIBS', True)
         needs_ldconfig = False
@@ -1504,6 +1507,11 @@ python package_do_shlibs() {
         if len(sonames):
             fd = open(shlibs_file, 'w')
             for s in sonames:
+                if s in shlib_provider:
+                    (old_pkg, old_pkgver) = shlib_provider[s]
+                    if old_pkg != pkg:
+                        bb.warn('%s-%s was registered as shlib provider for %s, changing it to %s-%s because it was built later' % (old_pkg, old_pkgver, s, pkg, pkgver))
+                bb.debug(1, 'registering %s-%s as shlib provider for %s' % (pkg, pkgver, s))
                 fd.write(s + '\n')
                 shlib_provider[s] = (pkg, pkgver)
             fd.close()
@@ -1517,8 +1525,7 @@ python package_do_shlibs() {
                 postinst = '#!/bin/sh\n'
             postinst += d.getVar('ldconfig_postinst_fragment', True)
             d.setVar('pkg_postinst_%s' % pkg, postinst)
-
-    read_shlib_providers()
+        bb.debug(1, 'LIBNAMES: pkg %s sonames %s' % (pkg, sonames))
 
     bb.utils.unlockfile(lf)
 
-- 
1.8.5.2

Re: [PATCHv2 2/5] package.bbclass: show warning when package is trying to provide already provided shlib

[Phil Blundell]

On Sun, 2014-01-19 at 14:28 +0100, Martin Jansa wrote:
> * move read_shlib_providers before registering package as provider
>   and don't change provider if it already exists, show warning instead

This new patch doesn't seem to match the commit message any more.

p.

> +                if s in shlib_provider:
> +                    (old_pkg, old_pkgver) = shlib_provider[s]
> +                    if old_pkg != pkg:
> +                        bb.warn('%s-%s was registered as shlib provider for %s, changing it to %s-%s because it was built later' % (old_pkg, old_pkgver, s, pkg, pkgver))
> +                bb.debug(1, 'registering %s-%s as shlib provider for %s' % (pkg, pkgver, s))

[PATCHv2 3/5] package.bbclass: add SHLIBSSEARCHDIRS to define where to search for shlib providers

[Martin Jansa]

* when package contains some files matching "^.*\.so", but in directory
  not default linker search paths (e.g. /opt/package/bundled-lib/libfoo.so)
  don't register it as libfoo provider, because it's possible that there
  is different package providing libfoo.so in ${libdir} and that would
  be better shlib provider for other packages to depend on
* recipes providing libs intentionally in some other directory can
  define own SHLIBSSEARCHDIRS value

[YOCTO #4628]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/classes/package.bbclass | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 9b511a6..a9e32f2 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1322,6 +1322,9 @@ python package_do_filedeps() {
 SHLIBSDIRS = "${PKGDATA_DIR}/${MLPREFIX}shlibs"
 SHLIBSWORKDIR = "${PKGDESTWORK}/${MLPREFIX}shlibs"
 
+# default search path when searching for shlibs provided by package
+SHLIBSSEARCHDIRS ?= "${baselib} ${libdir}"
+
 python package_do_shlibs() {
     import re, pipes
 
@@ -1332,6 +1335,20 @@ python package_do_shlibs() {
 
     lib_re = re.compile("^.*\.so")
     libdir_re = re.compile(".*/%s$" % d.getVar('baselib', True))
+    
+    shlibs_search_dirs = d.getVar('SHLIBSSEARCHDIRS', True)
+    shlibs_search_dirs_re_txt = ""
+    for dir in shlibs_search_dirs.split(' '):
+        # strip leading and trailing slash, it's added in regexp
+        if dir.endswith("/"):
+            dir = dir[:-1]
+        if dir.startswith("/"):
+            dir = dir[1:]
+        if shlibs_search_dirs_re_txt:
+            shlibs_search_dirs_re_txt += "|"
+        shlibs_search_dirs_re_txt += "(^.*/%s/.*$)" % dir
+    shlibs_search_dirs_re = re.compile(shlibs_search_dirs_re_txt)
+    bb.debug(2, "will use following RE to search for provides sonames %s" % shlibs_search_dirs_re_txt)
 
     packages = d.getVar('PACKAGES', True)
     targetos = d.getVar('TARGET_OS', True)
@@ -1390,9 +1407,12 @@ python package_do_shlibs() {
             if m:
                 this_soname = m.group(1)
                 if not this_soname in sonames:
-                    # if library is private (only used by package) then do not build shlib for it
-                    if not private_libs or -1 == private_libs.find(this_soname):
-                        sonames.append(this_soname)
+                    if shlibs_search_dirs_re.match(file):
+                        # if library is private (only used by package) then do not build shlib for it
+                        if not private_libs or -1 == private_libs.find(this_soname):
+                            sonames.append(this_soname)
+                    else:
+                        bb.debug(2, "ignoring soname %s from %s, because path doesn't match %s" % (this_soname, file, shlibs_search_dirs_re_txt))
                 if libdir_re.match(os.path.dirname(file)):
                     needs_ldconfig = True
                 if snap_symlinks and (os.path.basename(file) != this_soname):
-- 
1.8.5.2

Re: [PATCHv2 3/5] package.bbclass: add SHLIBSSEARCHDIRS to define where to search for shlib providers

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3505 bytes --]

On Sun, Jan 19, 2014 at 04:23:35PM +0100, Martin Jansa wrote:
> * when package contains some files matching "^.*\.so", but in directory
>   not default linker search paths (e.g. /opt/package/bundled-lib/libfoo.so)
>   don't register it as libfoo provider, because it's possible that there
>   is different package providing libfoo.so in ${libdir} and that would
>   be better shlib provider for other packages to depend on
> * recipes providing libs intentionally in some other directory can
>   define own SHLIBSSEARCHDIRS value

This one is the same as v1, sent by accident when sending v3 of 2/5.

> [YOCTO #4628]
> 
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  meta/classes/package.bbclass | 26 +++++++++++++++++++++++---
>  1 file changed, 23 insertions(+), 3 deletions(-)
> 
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index 9b511a6..a9e32f2 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -1322,6 +1322,9 @@ python package_do_filedeps() {
>  SHLIBSDIRS = "${PKGDATA_DIR}/${MLPREFIX}shlibs"
>  SHLIBSWORKDIR = "${PKGDESTWORK}/${MLPREFIX}shlibs"
>  
> +# default search path when searching for shlibs provided by package
> +SHLIBSSEARCHDIRS ?= "${baselib} ${libdir}"
> +
>  python package_do_shlibs() {
>      import re, pipes
>  
> @@ -1332,6 +1335,20 @@ python package_do_shlibs() {
>  
>      lib_re = re.compile("^.*\.so")
>      libdir_re = re.compile(".*/%s$" % d.getVar('baselib', True))
> +    
> +    shlibs_search_dirs = d.getVar('SHLIBSSEARCHDIRS', True)
> +    shlibs_search_dirs_re_txt = ""
> +    for dir in shlibs_search_dirs.split(' '):
> +        # strip leading and trailing slash, it's added in regexp
> +        if dir.endswith("/"):
> +            dir = dir[:-1]
> +        if dir.startswith("/"):
> +            dir = dir[1:]
> +        if shlibs_search_dirs_re_txt:
> +            shlibs_search_dirs_re_txt += "|"
> +        shlibs_search_dirs_re_txt += "(^.*/%s/.*$)" % dir
> +    shlibs_search_dirs_re = re.compile(shlibs_search_dirs_re_txt)
> +    bb.debug(2, "will use following RE to search for provides sonames %s" % shlibs_search_dirs_re_txt)
>  
>      packages = d.getVar('PACKAGES', True)
>      targetos = d.getVar('TARGET_OS', True)
> @@ -1390,9 +1407,12 @@ python package_do_shlibs() {
>              if m:
>                  this_soname = m.group(1)
>                  if not this_soname in sonames:
> -                    # if library is private (only used by package) then do not build shlib for it
> -                    if not private_libs or -1 == private_libs.find(this_soname):
> -                        sonames.append(this_soname)
> +                    if shlibs_search_dirs_re.match(file):
> +                        # if library is private (only used by package) then do not build shlib for it
> +                        if not private_libs or -1 == private_libs.find(this_soname):
> +                            sonames.append(this_soname)
> +                    else:
> +                        bb.debug(2, "ignoring soname %s from %s, because path doesn't match %s" % (this_soname, file, shlibs_search_dirs_re_txt))
>                  if libdir_re.match(os.path.dirname(file)):
>                      needs_ldconfig = True
>                  if snap_symlinks and (os.path.basename(file) != this_soname):
> -- 
> 1.8.5.2
> 

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

[PATCHv3 2/5] package.bbclass: show warning when package is providing already provided shlib

[Martin Jansa]

* move read_shlib_providers before registering package as provider
  and show warning when different package tries to provide something
  already provided.

[YOCTO #4628]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/classes/package.bbclass | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 7dcec5e..9b511a6 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1356,6 +1356,7 @@ python package_do_shlibs() {
         list_re = re.compile('^(.*)\.list$')
         # Go from least to most specific since the last one found wins
         for dir in reversed(shlibs_dirs):
+            bb.debug(2, "Reading shlib providers in %s" % (dir))
             if not os.path.exists(dir):
                 continue
             for file in os.listdir(dir):
@@ -1471,6 +1472,8 @@ python package_do_shlibs() {
 
     needed = {}
     shlib_provider = {}
+    read_shlib_providers()
+
     for pkg in packages.split():
         private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or d.getVar('PRIVATE_LIBS', True)
         needs_ldconfig = False
@@ -1504,6 +1507,11 @@ python package_do_shlibs() {
         if len(sonames):
             fd = open(shlibs_file, 'w')
             for s in sonames:
+                if s in shlib_provider:
+                    (old_pkg, old_pkgver) = shlib_provider[s]
+                    if old_pkg != pkg:
+                        bb.warn('%s-%s was registered as shlib provider for %s, changing it to %s-%s because it was built later' % (old_pkg, old_pkgver, s, pkg, pkgver))
+                bb.debug(1, 'registering %s-%s as shlib provider for %s' % (pkg, pkgver, s))
                 fd.write(s + '\n')
                 shlib_provider[s] = (pkg, pkgver)
             fd.close()
@@ -1517,8 +1525,7 @@ python package_do_shlibs() {
                 postinst = '#!/bin/sh\n'
             postinst += d.getVar('ldconfig_postinst_fragment', True)
             d.setVar('pkg_postinst_%s' % pkg, postinst)
-
-    read_shlib_providers()
+        bb.debug(1, 'LIBNAMES: pkg %s sonames %s' % (pkg, sonames))
 
     bb.utils.unlockfile(lf)
 
-- 
1.8.5.2

[PATCH 3/5] package.bbclass: add SHLIBSSEARCHDIRS to define where to search for shlib providers

[Martin Jansa]

* when package contains some files matching "^.*\.so", but in directory
  not default linker search paths (e.g. /opt/package/bundled-lib/libfoo.so)
  don't register it as libfoo provider, because it's possible that there
  is different package providing libfoo.so in ${libdir} and that would
  be better shlib provider for other packages to depend on
* recipes providing libs intentionally in some other directory can
  define own SHLIBSSEARCHDIRS value

[YOCTO #4628]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/classes/package.bbclass | 26 +++++++++++++++++++++++---
 1 file changed, 23 insertions(+), 3 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index aa8156d..fe25a6b 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1322,6 +1322,9 @@ python package_do_filedeps() {
 SHLIBSDIRS = "${PKGDATA_DIR}/${MLPREFIX}shlibs"
 SHLIBSWORKDIR = "${PKGDESTWORK}/${MLPREFIX}shlibs"
 
+# default search path when searching for shlibs provided by package
+SHLIBSSEARCHDIRS ?= "${baselib} ${libdir}"
+
 python package_do_shlibs() {
     import re, pipes
 
@@ -1332,6 +1335,20 @@ python package_do_shlibs() {
 
     lib_re = re.compile("^.*\.so")
     libdir_re = re.compile(".*/%s$" % d.getVar('baselib', True))
+    
+    shlibs_search_dirs = d.getVar('SHLIBSSEARCHDIRS', True)
+    shlibs_search_dirs_re_txt = ""
+    for dir in shlibs_search_dirs.split(' '):
+        # strip leading and trailing slash, it's added in regexp
+        if dir.endswith("/"):
+            dir = dir[:-1]
+        if dir.startswith("/"):
+            dir = dir[1:]
+        if shlibs_search_dirs_re_txt:
+            shlibs_search_dirs_re_txt += "|"
+        shlibs_search_dirs_re_txt += "(^.*/%s/.*$)" % dir
+    shlibs_search_dirs_re = re.compile(shlibs_search_dirs_re_txt)
+    bb.debug(2, "will use following RE to search for provides sonames %s" % shlibs_search_dirs_re_txt)
 
     packages = d.getVar('PACKAGES', True)
     targetos = d.getVar('TARGET_OS', True)
@@ -1390,9 +1407,12 @@ python package_do_shlibs() {
             if m:
                 this_soname = m.group(1)
                 if not this_soname in sonames:
-                    # if library is private (only used by package) then do not build shlib for it
-                    if not private_libs or -1 == private_libs.find(this_soname):
-                        sonames.append(this_soname)
+                    if shlibs_search_dirs_re.match(file):
+                        # if library is private (only used by package) then do not build shlib for it
+                        if not private_libs or -1 == private_libs.find(this_soname):
+                            sonames.append(this_soname)
+                    else:
+                        bb.debug(2, "ignoring soname %s from %s, because path doesn't match %s" % (this_soname, file, shlibs_search_dirs_re_txt))
                 if libdir_re.match(os.path.dirname(file)):
                     needs_ldconfig = True
                 if snap_symlinks and (os.path.basename(file) != this_soname):
-- 
1.8.5.2

Re: [PATCH 3/5] package.bbclass: add SHLIBSSEARCHDIRS to define where to search for shlib providers

[Richard Purdie]

On Sat, 2014-01-18 at 15:02 +0100, Martin Jansa wrote:
> * when package contains some files matching "^.*\.so", but in directory
>   not default linker search paths (e.g. /opt/package/bundled-lib/libfoo.so)
>   don't register it as libfoo provider, because it's possible that there
>   is different package providing libfoo.so in ${libdir} and that would
>   be better shlib provider for other packages to depend on
> * recipes providing libs intentionally in some other directory can
>   define own SHLIBSSEARCHDIRS value
> 
> [YOCTO #4628]
> 
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  meta/classes/package.bbclass | 26 +++++++++++++++++++++++---
>  1 file changed, 23 insertions(+), 3 deletions(-)

Doesn't this mean we lose automatic dependencies for libraries not in
the default system paths after this change?

We should really be respecting the libraries RPATH in this code and
searching the places it will look...

Cheers,

Richard

Re: [PATCH 3/5] package.bbclass: add SHLIBSSEARCHDIRS to define where to search for shlib providers

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1759 bytes --]

On Sun, Jan 19, 2014 at 05:13:45PM +0000, Richard Purdie wrote:
> On Sat, 2014-01-18 at 15:02 +0100, Martin Jansa wrote:
> > * when package contains some files matching "^.*\.so", but in directory
> >   not default linker search paths (e.g. /opt/package/bundled-lib/libfoo.so)
> >   don't register it as libfoo provider, because it's possible that there
> >   is different package providing libfoo.so in ${libdir} and that would
> >   be better shlib provider for other packages to depend on
> > * recipes providing libs intentionally in some other directory can
> >   define own SHLIBSSEARCHDIRS value
> > 
> > [YOCTO #4628]
> > 
> > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > ---
> >  meta/classes/package.bbclass | 26 +++++++++++++++++++++++---
> >  1 file changed, 23 insertions(+), 3 deletions(-)
> 
> Doesn't this mean we lose automatic dependencies for libraries not in
> the default system paths after this change?

True, recipes with such libraries would need to append to
SHLIBSSEARCHDIRS

> We should really be respecting the libraries RPATH in this code and
> searching the places it will look...

I agree we should respect RPATH, but sometimes it is more complicated,
because your own binaries can have RPATH pointing to
/opt/package/bundled-lib/libfoo.so, but you don't want any other recipe
to find your libfoo.so, because there is preferred "system" /usr/lib/libfoo.so

So in ideal case we should respect combination of default search path,
ld.so.conf, RPATH and PRIVATE_LIBS :/.

I don't mind if you drop this one from patchset, it's the most controversial
one and can be work-arounded by carefully maintained PRIVATE_LIBS list.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

[PATCH 4/5] package.bbclass: Don't search for prividers of PRIVATE_LIBS

[Martin Jansa]

* split PRIVATE_LIBS and don't use find(), so that libfoo cannot be
  found in PRIVATE_LIBS = "libfoobar"

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/classes/package.bbclass | 13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index fe25a6b..7820955 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1409,7 +1409,7 @@ python package_do_shlibs() {
                 if not this_soname in sonames:
                     if shlibs_search_dirs_re.match(file):
                         # if library is private (only used by package) then do not build shlib for it
-                        if not private_libs or -1 == private_libs.find(this_soname):
+                        if not private_libs or this_soname not in private_libs:
                             sonames.append(this_soname)
                     else:
                         bb.debug(2, "ignoring soname %s from %s, because path doesn't match %s" % (this_soname, file, shlibs_search_dirs_re_txt))
@@ -1495,7 +1495,8 @@ python package_do_shlibs() {
     read_shlib_providers()
 
     for pkg in packages.split():
-        private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or d.getVar('PRIVATE_LIBS', True)
+        private_libs = d.getVar('PRIVATE_LIBS_' + pkg, True) or d.getVar('PRIVATE_LIBS', True) or ""
+        private_libs = private_libs.split()
         needs_ldconfig = False
         bb.debug(2, "calculating shlib provides for %s" % pkg)
 
@@ -1566,6 +1567,14 @@ python package_do_shlibs() {
 
         deps = list()
         for n in needed[pkg]:
+            # if n is in private libraries, don't try to search provider for it
+            # this could cause problem in case some abc.bb provides private
+            # /opt/abc/lib/libfoo.so.1 and contains /usr/bin/abc depending on system library libfoo.so.1
+            # but skipping it is still better alternative than providing own
+            # version and then adding runtime dependency for the same system library
+            if private_libs and n in private_libs:
+                bb.debug(2, '%s: Dependency %s covered by PRIVATE_LIBS' % (pkg, n))
+                continue
             if n in shlib_provider.keys():
                 (dep_pkg, ver_needed) = shlib_provider[n]
 
-- 
1.8.5.2

[PATCH 5/5] package.bbclass: Show which files require given dependency in debug output

[Martin Jansa]

* when log.do_package shows some unexpected dependency, people usually
  need to grep package directory to find which binary was creating that
  dependency, show it directly in the debug output

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/classes/package.bbclass | 12 ++++++++++--
 1 file changed, 10 insertions(+), 2 deletions(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index 7820955..854a120 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1403,6 +1403,9 @@ python package_do_shlibs() {
             if m:
                 if m.group(1) not in needed[pkg]:
                     needed[pkg].append(m.group(1))
+                if m.group(1) not in needed_from:
+                    needed_from[m.group(1)] = []
+                needed_from[m.group(1)].append(file)
             m = re.match("\s+SONAME\s+([^\s]*)", l)
             if m:
                 this_soname = m.group(1)
@@ -1478,6 +1481,10 @@ python package_do_shlibs() {
                                 needed[pkg] = []
                             if name and name not in needed[pkg]:
                                 needed[pkg].append(name)
+                            if name not in needed_from:
+                                needed_from[name] = []
+                            if lafile and lafile not in needed_from[name]:
+                                needed_from[name].append(lafile)
                                 #bb.note("Adding %s for %s" % (name, pkg))
 
     if d.getVar('PACKAGE_SNAP_LIB_SYMLINKS', True) == "1":
@@ -1491,6 +1498,7 @@ python package_do_shlibs() {
         use_ldconfig = False
 
     needed = {}
+    needed_from = {}
     shlib_provider = {}
     read_shlib_providers()
 
@@ -1578,7 +1586,7 @@ python package_do_shlibs() {
             if n in shlib_provider.keys():
                 (dep_pkg, ver_needed) = shlib_provider[n]
 
-                bb.debug(2, '%s: Dependency %s requires package %s' % (pkg, n, dep_pkg))
+                bb.debug(2, '%s: Dependency %s requires package %s (used by files: %s)' % (pkg, n, dep_pkg, needed_from[n]))
 
                 if dep_pkg == pkg:
                     continue
@@ -1590,7 +1598,7 @@ python package_do_shlibs() {
                 if not dep in deps:
                     deps.append(dep)
             else:
-                bb.note("Couldn't find shared library provider for %s" % n)
+                bb.note("Couldn't find shared library provider for %s, used by files: %s" % (n, needed_from[n]))
 
         deps_file = os.path.join(pkgdest, pkg + ".shlibdeps")
         if os.path.exists(deps_file):
-- 
1.8.5.2