debian.bbclass issues when some package is substring of another

debian.bbclass issues when some package is substring of another

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 4237 bytes --]

After last rebuild from scratch I got
Package libncursesw (5.7+20110115-r16.6)
instead of expected
Package libncursesw5 (5.7+20110115-r16.6)
which I had before (and which is _still_ in depends of packages needing
it - like ie screen)

I started to debug debian.bbclass to see why
libncursesw is no longer renamed to libncursesw5 and found that it's
because also libncurses exists and it's replaced in both and then
replacement for libncursesw doesn't match anymore. Same issue with
libtic/libticw.

Problem is that when checking libtic:
./log:DEBUG: LIBNAMES: pkg ncurses-libtic socrunc ('libtic5', 'libtic')
./log:DEBUG: LIBNAMES: pkgname libtic5 devname libtic

./log:DEBUG: LIBNAMES: pkg ncurses-libtic == orig_pkg ncurses-libtic
./log:DEBUG: LIBNAMES: pkgname libtic5 devname libtic pkg ncurses-libtic orig_pkg ncurses-libtic debian_pn None newpkg libtic5
./log:DEBUG: setVar PKG_ncurses-libtic, libtic5
but also
./log:DEBUG: LIBNAMES: else pkg ncurses-libticw.replace(ncurses-libtic,libtic)
./log:DEBUG: LIBNAMES: pkgname libtic5 devname libtic pkg ncurses-libticw orig_pkg ncurses-libtic debian_pn None newpkg libticw
./log:DEBUG: setVar PKG_ncurses-libticw, libticw

so later when checking libticw
./log:DEBUG: LIBNAMES: pkg ncurses-libticw socrunc ('libticw5','libticw')
./log:DEBUG: LIBNAMES: pkgname libticw5 devname libticw
it won't find ncurses-libticw to replace it with libticw5 (because it's
already named libticw)

So can someone explain why we need that 'newpkg = pkg.replace(orig_pkg, devname, 1)' in else?

Or any hint which change caused this (as debian.bbclass wasn't changed in this aspect).

Reordering PACKAGES seems like really error prone and even not really possible with dynamic packages etc.

Here is added debug I've used to find it:

diff --git a/classes/debian.bbclass b/classes/debian.bbclass
index 1a2c7da..9239631 100644
--- a/classes/debian.bbclass
+++ b/classes/debian.bbclass
@@ -84,10 +84,13 @@ python debian_package_name_hook () {
                        else:
                                bb.note("Multiple libraries (%s) found and LEAD_SONAME not defined" % ", ".join(sonames))

+               bb.debug(1, 'LIBNAMES: pkg %s soname %s' % (orig_pkg, soname))
                if has_libs and not has_bins and soname:
                        soname_result = socrunch(soname)
+                       bb.debug(1, 'LIBNAMES: pkg %s socrunc %s' % (orig_pkg, soname_result))
                        if soname_result:
                                (pkgname, devname) = soname_result
+                               bb.debug(1, 'LIBNAMES: pkgname %s devname %s' % (pkgname, devname))
                                for pkg in packages.split():
                                        if (bb.data.getVar('PKG_' + pkg, d) or bb.data.getVar('DEBIAN_NOAUTONAME_' + pkg, d)):
                                                continue
@@ -95,11 +98,15 @@ python debian_package_name_hook () {
                                        if debian_pn:
                                                newpkg = debian_pn
                                        elif pkg == orig_pkg:
+                                               bb.debug(1, 'LIBNAMES: pkg %s == orig_pkg %s' % (pkg, orig_pkg))
                                                newpkg = pkgname
                                        else:
+                                               bb.debug(1, 'LIBNAMES: else pkg %s.replace(%s, %s)' % (pkg, orig_pkg, devname))
                                                newpkg = pkg.replace(orig_pkg, devname, 1)
+                                       bb.debug(1, 'LIBNAMES: pkgname %s devname %s pkg %s orig_pkg %s debian_pn %s newpkg %s' % (pkgname, devname, pkg, orig_pkg, debian_pn, newpkg))
                                        if newpkg != pkg:
                                                bb.data.setVar('PKG_' + pkg, newpkg, d)
+                                               bb.debug(1, 'setVar PKG_%s, %s' % (pkg, newpkg))

        for pkg in (bb.data.getVar('AUTO_LIBNAME_PKGS', d, 1) or "").split():
                auto_libname(packages, pkg)


-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

Re: debian.bbclass issues when some package is substring of another

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3043 bytes --]

On Mon, Feb 14, 2011 at 11:22:32AM +0100, Martin Jansa wrote:
> After last rebuild from scratch I got
> Package libncursesw (5.7+20110115-r16.6)
> instead of expected
> Package libncursesw5 (5.7+20110115-r16.6)
> which I had before (and which is _still_ in depends of packages needing
> it - like ie screen)

To be more clear about that issue:

second buildhost
DEBUG: setVar PKG_ncurses-libpanelw, libpanelw5            <= ok
DEBUG: setVar PKG_ncurses-libmenu, libmenu5
DEBUG: setVar PKG_ncurses-libmenuw, libmenuw               <= bad
DEBUG: setVar PKG_ncurses-libncursesw, libncursesw5        <= ok
DEBUG: setVar PKG_ncurses-libformw, libformw5              <= ok
DEBUG: setVar PKG_ncurses-libtic, libtic5
DEBUG: setVar PKG_ncurses-libticw, libticw                 <= bad
DEBUG: setVar PKG_ncurses-libtinfo, libtinfo5
DEBUG: setVar PKG_ncurses-libncurses, libncurses5
DEBUG: setVar PKG_ncurses-libform, libform5
DEBUG: setVar PKG_ncurses-libpanel, libpanel5

my buildhost
DEBUG: setVar PKG_ncurses-libformw, libformw5             <= ok
DEBUG: setVar PKG_ncurses-libncurses, libncurses5
DEBUG: setVar PKG_ncurses-libncursesw, libncursesw        <= bad
DEBUG: setVar PKG_ncurses-libpanelw, libpanelw5           <= ok
DEBUG: setVar PKG_ncurses-libmenuw, libmenuw5             <= ok
DEBUG: setVar PKG_ncurses-libtinfo, libtinfo5
DEBUG: setVar PKG_ncurses-libpanel, libpanel5
DEBUG: setVar PKG_ncurses-libtic, libtic5
DEBUG: setVar PKG_ncurses-libticw, libticw                <= bad
DEBUG: setVar PKG_ncurses-libform, libform5
DEBUG: setVar PKG_ncurses-libmenu, libmenu5

my buildhost after removing prepend=True in ncurses recipe
-        do_split_packages(d, libdir, '^lib(.*)\.so\..*', pnbase, 'ncurses %s library', prepend=True, extra_depends = '', allow_links=True)
+        do_split_packages(d, libdir, '^lib(.*)\.so\..*', pnbase, 'ncurses %s library', extra_depends = '', allow_links=True)

DEBUG: setVar PKG_ncurses-libticw, libticw5              <= ok
DEBUG: setVar PKG_ncurses-libmenu, libmenu5
DEBUG: setVar PKG_ncurses-libmenuw, libmenuw             <= bad
DEBUG: setVar PKG_ncurses-libform, libform5
DEBUG: setVar PKG_ncurses-libformw, libformw             <= bad
DEBUG: setVar PKG_ncurses-libtic, libtic5
DEBUG: setVar PKG_ncurses-libpanel, libpanel5
DEBUG: setVar PKG_ncurses-libpanelw, libpanelw           <= bad
DEBUG: setVar PKG_ncurses-libtinfo, libtinfo5
DEBUG: setVar PKG_ncurses-libncursesw, libncursesw5      <= ok
DEBUG: setVar PKG_ncurses-libncurses, libncurses5

so it really depends on order in PACKAGES which in this case also depends on
for walkroot, dirs, files in os.walk(dvar + root) (from package.bbclass)
which isn't consistene even between 2 buildhosts.

Alphabetical descending sort should give better results (substrings later),
but is it right thing to do? And can we do it in do_split_packages as sorting
whole PACKAGES var is not possible due to order how FILES_ are used.

Regards,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

Re: debian.bbclass issues when some package is substring of another

[Phil Blundell]

On Mon, 2011-02-14 at 11:22 +0100, Martin Jansa wrote:
> So can someone explain why we need that 'newpkg = pkg.replace(orig_pkg, devname, 1)' in else?

If that line was removed altogether then the -dev packages (and -doc,
-bin, ...) would stop being renamed, which wouldn't be a very good
thing.  In other words, you'd get libncurses5 but ncurses-dev (not
libncurses5-dev) etc.

However you're right that the logic as it stands does look fairly
broken.  Two ways of fixing it come to mind:

a) have it explicitly search for (and rename) [orig_pkg]-dev,
[orig_pkg]-doc etc, and remove the catch-all altogether; or

b) switch to a two-phase approach where it first renames all the
libraries themselves, then uses some heuristic to match the remaining
packages against the libraries and apply an appropriate renaming to
them.  Probably just matching on longest substring would be good enough
here.

I think (b) is probably a better solution although it would be a little
harder to implement.

p.

Re: debian.bbclass issues when some package is substring of another

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2235 bytes --]

On Mon, Feb 14, 2011 at 03:27:23PM +0000, Phil Blundell wrote:
> On Mon, 2011-02-14 at 11:22 +0100, Martin Jansa wrote:
> > So can someone explain why we need that 'newpkg = pkg.replace(orig_pkg, devname, 1)' in else?
> 
> If that line was removed altogether then the -dev packages (and -doc,
> -bin, ...) would stop being renamed, which wouldn't be a very good
> thing.  In other words, you'd get libncurses5 but ncurses-dev (not
> libncurses5-dev) etc.

That's not really what you get currently, but ok I see your point in keeping
that "else".

Currently you would get libncurses-dev without 5, because it replaces
with devname not pkgname.

> 
> However you're right that the logic as it stands does look fairly
> broken.  Two ways of fixing it come to mind:
> 
> a) have it explicitly search for (and rename) [orig_pkg]-dev,
> [orig_pkg]-doc etc, and remove the catch-all altogether; or
> 
> b) switch to a two-phase approach where it first renames all the
> libraries themselves, then uses some heuristic to match the remaining
> packages against the libraries and apply an appropriate renaming to
> them.  Probably just matching on longest substring would be good enough
> here.
> 
> I think (b) is probably a better solution although it would be a little
> harder to implement.

I'll try (b), but imho you have same problem there (if you have
-dev/-dbg for such package names and they are in wrong order, even worse ie:
PKG_ncurses-libncurses-dev, libncurses-dev    <= ok
PKG_ncurses-libncursesw-dev, libncursesw-dev  <= works only because
  ncurses-libncursesw is mapped to libncursesw which is the same like
  s/ncurses-libncurses/libncurses

but if you imagine stranger situation with SONAMES which are not
substrings of each other in package names which are substrings, like:
ncurses-libncurses > libncurses
ncurses-libncursesw > libwidecurses

then still
PKG_ncurses-libncurses-dev, libncurses-dev    <= ok
PKG_ncurses-libncursesw-dev, libncursesw-dev  <= bad, because libwidecurses-dev was expected 

and later you won't find ncurses-libncursesw-dev to replace
ncurses-libncursesw with libwidecurses

Regards,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

[PATCH] debian.bbclass: call auto_libname in reverse sorted AUTO_LIBNAME_PKGS

[Martin Jansa]

* see comment for reason why we need this
* more info:
  http://lists.linuxtogo.org/pipermail/openembedded-devel/2011-February/029877.html
---
 classes/debian.bbclass |    8 +++++++-
 1 files changed, 7 insertions(+), 1 deletions(-)

diff --git a/classes/debian.bbclass b/classes/debian.bbclass
index 1a2c7da..7cdb6d2 100644
--- a/classes/debian.bbclass
+++ b/classes/debian.bbclass
@@ -101,7 +101,13 @@ python debian_package_name_hook () {
 					if newpkg != pkg:
 						bb.data.setVar('PKG_' + pkg, newpkg, d)
 
-	for pkg in (bb.data.getVar('AUTO_LIBNAME_PKGS', d, 1) or "").split():
+	# reversed sort is needed when some package is substring of another
+	# ie in ncurses we get without reverse sort: 
+	# DEBUG: LIBNAMES: pkgname libtic5 devname libtic pkg ncurses-libtic orig_pkg ncurses-libtic debian_pn None newpkg libtic5
+	# and later
+	# DEBUG: LIBNAMES: pkgname libtic5 devname libtic pkg ncurses-libticw orig_pkg ncurses-libtic debian_pn None newpkg libticw
+	# so we need to handle ncurses-libticw->libticw5 before ncurses-libtic->libtic5
+	for pkg in sorted((bb.data.getVar('AUTO_LIBNAME_PKGS', d, 1) or "").split(), reverse=True):
 		auto_libname(packages, pkg)
 }
 
-- 
1.7.4.1

Re: debian.bbclass issues when some package is substring of another

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1318 bytes --]

On Tue, Feb 15, 2011 at 10:15:17AM +0100, Martin Jansa wrote:
> I'll try (b), but imho you have same problem there (if you have
> -dev/-dbg for such package names and they are in wrong order, even worse ie:
> PKG_ncurses-libncurses-dev, libncurses-dev    <= ok
> PKG_ncurses-libncursesw-dev, libncursesw-dev  <= works only because
>   ncurses-libncursesw is mapped to libncursesw which is the same like
>   s/ncurses-libncurses/libncurses

for now I've tried just reverse sorted order of calling auto_libname,
see sent patch.

sofar it looks ok and works right for ncurses, only in even stranger
situation t will fail (old unsorted behavior too) ie when
resulting SONAME is substring of some package.

mapping like this:
ncurses-libncursesw > libwidecurses
ncurses-libncursesw-dev > libwidecurses-dev
libwidecurses > libwcurses
libwidecurses-dev > libwcurses-dev

which will probably replace _again_ not only its own package
libwidecurses but also new libwidecurses (originaly ncurses-libncursesw)

but this situation looks really improbable and can be ignored

Maybe we can be more strict and call replace only from begining?
-newpkg = pkg.replace(orig_pkg, devname, 1)
+newpkg = re.sub('^'+orig_pkg, devname, pkg)

Regards,
-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

Re: debian.bbclass issues when some package is substring of another

[Phil Blundell]

On Tue, 2011-02-15 at 10:15 +0100, Martin Jansa wrote:
> On Mon, Feb 14, 2011 at 03:27:23PM +0000, Phil Blundell wrote:
> > b) switch to a two-phase approach where it first renames all the
> > libraries themselves, then uses some heuristic to match the remaining
> > packages against the libraries and apply an appropriate renaming to
> > them.  Probably just matching on longest substring would be good enough
> > here.
> > 
> > I think (b) is probably a better solution although it would be a little
> > harder to implement.
> 
> I'll try (b), but imho you have same problem there (if you have
> -dev/-dbg for such package names and they are in wrong order, even worse ie:
> PKG_ncurses-libncurses-dev, libncurses-dev    <= ok
> PKG_ncurses-libncursesw-dev, libncursesw-dev  <= works only because
>   ncurses-libncursesw is mapped to libncursesw which is the same like
>   s/ncurses-libncurses/libncurses
> 
> but if you imagine stranger situation with SONAMES which are not
> substrings of each other in package names which are substrings, like:
> ncurses-libncurses > libncurses
> ncurses-libncursesw > libwidecurses
> 
> then still
> PKG_ncurses-libncurses-dev, libncurses-dev    <= ok
> PKG_ncurses-libncursesw-dev, libncursesw-dev  <= bad, because libwidecurses-dev was expected 
> 
> and later you won't find ncurses-libncursesw-dev to replace
> ncurses-libncursesw with libwidecurses

I'm not quite sure I understand the difficulty here.

If I understand right you're talking about two output packages,
"ncurses" and "ncursesw", which contain sonames "libncurses.so.5" and
"libwidecurses.so.5", right?  In that case, during the first phase you'd
get:

PKG_ncurses = "libncurses5"
PKG_ncursesw = "libwidecurses5"

and a dictionary of renames performed which would look a bit like:

ncurses: { pkgname: libncurses5, devname: libncurses }
ncursesw: { pkgname: libwidecurses5, devname: libwidecurses }

which is all expected.  Then, during the second phase, you'd consider
the other packages like ncurses-dev and ncursesw-dev.  In this case, the
longest prefix match for ncurses-dev in the dictionary is "ncurses", so
it'd get renamed to libncurses-dev.  The longest matching prefix for
ncursesw-dev in the dictionary is "ncursesw", so it'd get renamed to
libwidecurses-dev.  That all seems like it should work out fine.

p.

Re: debian.bbclass issues when some package is substring of another

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3230 bytes --]

On Tue, Feb 15, 2011 at 10:19:31AM +0000, Phil Blundell wrote:
> On Tue, 2011-02-15 at 10:15 +0100, Martin Jansa wrote:
> > On Mon, Feb 14, 2011 at 03:27:23PM +0000, Phil Blundell wrote:
> > > b) switch to a two-phase approach where it first renames all the
> > > libraries themselves, then uses some heuristic to match the remaining
> > > packages against the libraries and apply an appropriate renaming to
> > > them.  Probably just matching on longest substring would be good enough
> > > here.
> > > 
> > > I think (b) is probably a better solution although it would be a little
> > > harder to implement.
> > 
> > I'll try (b), but imho you have same problem there (if you have
> > -dev/-dbg for such package names and they are in wrong order, even worse ie:
> > PKG_ncurses-libncurses-dev, libncurses-dev    <= ok
> > PKG_ncurses-libncursesw-dev, libncursesw-dev  <= works only because
> >   ncurses-libncursesw is mapped to libncursesw which is the same like
> >   s/ncurses-libncurses/libncurses
> > 
> > but if you imagine stranger situation with SONAMES which are not
> > substrings of each other in package names which are substrings, like:
> > ncurses-libncurses > libncurses
> > ncurses-libncursesw > libwidecurses
> > 
> > then still
> > PKG_ncurses-libncurses-dev, libncurses-dev    <= ok
> > PKG_ncurses-libncursesw-dev, libncursesw-dev  <= bad, because libwidecurses-dev was expected 
> > 
> > and later you won't find ncurses-libncursesw-dev to replace
> > ncurses-libncursesw with libwidecurses
> 
> I'm not quite sure I understand the difficulty here.

Ah sorry, I didn't read your (b) properly, now I see last sentence about
"longest substring".

But that's mostly what reverse sorted patch does now, isn't it? Longer
patterns are replaced before their shorter substrings (which then
matches only to their proper packages, because longer were already
replaced while handling longer pattern and "normally" doesn't match
anymore).

Regards,

> If I understand right you're talking about two output packages,
> "ncurses" and "ncursesw", which contain sonames "libncurses.so.5" and
> "libwidecurses.so.5", right?  In that case, during the first phase you'd
> get:
> 
> PKG_ncurses = "libncurses5"
> PKG_ncursesw = "libwidecurses5"
> 
> and a dictionary of renames performed which would look a bit like:
> 
> ncurses: { pkgname: libncurses5, devname: libncurses }
> ncursesw: { pkgname: libwidecurses5, devname: libwidecurses }
> 
> which is all expected.  Then, during the second phase, you'd consider
> the other packages like ncurses-dev and ncursesw-dev.  In this case, the
> longest prefix match for ncurses-dev in the dictionary is "ncurses", so
> it'd get renamed to libncurses-dev.  The longest matching prefix for
> ncursesw-dev in the dictionary is "ncursesw", so it'd get renamed to
> libwidecurses-dev.  That all seems like it should work out fine.
> 
> p.
> 
> 
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

Re: debian.bbclass issues when some package is substring of another

[Phil Blundell]

On Tue, 2011-02-15 at 11:36 +0100, Martin Jansa wrote:
> Ah sorry, I didn't read your (b) properly, now I see last sentence about
> "longest substring".
> 
> But that's mostly what reverse sorted patch does now, isn't it? Longer
> patterns are replaced before their shorter substrings (which then
> matches only to their proper packages, because longer were already
> replaced while handling longer pattern and "normally" doesn't match
> anymore).

Yeah, I think that's probably true.  I haven't managed to convince
myself entirely that the reverse sorting patch is safe in all
situations, but neither have I been able to think of a case that my
algorithm would get right and yours would get wrong.  So I think you
might as well go ahead and check in your patch, and we can revisit it in
the future if further problems do pop up.

p.