[RFC] Version sorting in BitBake

[RFC] Version sorting in BitBake

[Denys Dmytriyenko]

Hi,

While debugging the problem with the wrong kernel recipe being picked up by 
BitBake, based on a complex PV value (there were several instances, once of 
them recently mentioned at [1]), I found a small flaw in BitBake's version 
comparison algorithm, as it relates to -preX and -rcX versions.

For the above cases we use this format: PV = "2.6.29+2.6.30-rc5"

At this point both BitBake and opkg (I haven't tried other package managers) 
sort those PVs properly, as EOL sorts lower than any other character:

2.6.29 < 2.6.29+2.6.30-rc5 < 2.6.30

Since the introduction of MACHINE_KERNEL_PR, we moved modifiers such as 
"+gitr${SRCREV}" (or any distro/local suffixes) from PR to PV, but still 
placing them after PR for proper sorting:

PV1 = "2.6.29-${PR}+gitr${SRCREV}"
PV2 = "2.6.29+2.6.30-rc5-${PR}+gitr${SRCREV}"
PV3 = "2.6.30-${PR}+gitr${SRCREV}"

That still works with opkg, as ipkg-compare-versions sorts above PVs as 
expected - PV1 < PV2 < PV3

But BitBake now has this flaw and sorts like this: PV2 < PV1 < PV3

Looks like ipkg-compare-versions/opkg algorithm has a special handling for '-' 
and '.' characters, making them sort lower than '+', thus producing expected 
results.

I understand that this behavior can be controlled by PREFERRED_VERSION and 
DEFAULT_PREFERENCE variables. But would it make sense to also fix it in 
BitBake? I do have a patch, which I'll send next. Please provide comments.

[1] http://thread.gmane.org/gmane.comp.handhelds.openembedded/26622/focus=26634

-- 
Denys

[PATCH] utils.py: add special handling for version delimiters

[Denys Dmytriyenko]

Make version comparison work properly for pre-releases and release-candidates,
when there is an extra suffix in the field, such as:
PV = "2.6.29+2.6.30-rc5-${PR}+gitr${SRCREV}"

More details: http://thread.gmane.org/gmane.comp.handhelds.openembedded/26691

Signed-off-by: Denys Dmytriyenko <denis@denix.org>
---
 lib/bb/utils.py |   16 +++++++++++++++-
 1 files changed, 15 insertions(+), 1 deletions(-)

diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index 2469bd7..e6664e2 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -21,8 +21,9 @@ BitBake Utility Functions
 
 digits = "0123456789"
 ascii_letters = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
+separators = ".-"
 
-import re, fcntl, os
+import re, fcntl, os, types
 
 def explode_version(s):
     r = []
@@ -39,12 +40,15 @@ def explode_version(s):
             r.append(m.group(1))
             s = m.group(2)
             continue
+        r.append(s[0])
         s = s[1:]
     return r
 
 def vercmp_part(a, b):
     va = explode_version(a)
     vb = explode_version(b)
+    sa = False
+    sb = False
     while True:
         if va == []:
             ca = None
@@ -56,6 +60,16 @@ def vercmp_part(a, b):
             cb = vb.pop(0)
         if ca == None and cb == None:
             return 0
+
+        if type(ca) is types.StringType:
+            sa = ca in separators
+        if type(cb) is types.StringType:
+            sb = cb in separators
+        if sa and not sb:
+            return -1
+        if not sa and sb:
+            return 1
+
         if ca > cb:
             return 1
         if ca < cb:
-- 
1.6.3.3

Re: [RFC] Version sorting in BitBake

[Otavio Salvador]

Hello Denys,

On Tue, Oct 6, 2009 at 12:46 AM, Denys Dmytriyenko <denis@denix.org> wrote:
[...]
> PV1 = "2.6.29-${PR}+gitr${SRCREV}"
> PV2 = "2.6.29+2.6.30-rc5-${PR}+gitr${SRCREV}"
> PV3 = "2.6.30-${PR}+gitr${SRCREV}"
>
> That still works with opkg, as ipkg-compare-versions sorts above PVs as
> expected - PV1 < PV2 < PV3
>
> But BitBake now has this flaw and sorts like this: PV2 < PV1 < PV3
[...]

Please send the patch for people to review and would be nice to have
it commited since many of us could have been using non-required
overrides.

Thanks by founding and _fixing_ it :-D

-- 
Otavio Salvador                  O.S. Systems
E-mail: otavio@ossystems.com.br  http://www.ossystems.com.br
Mobile: +55 53 9981-7854         http://projetos.ossystems.com.br

Re: [RFC] Version sorting in BitBake

[Denys Dmytriyenko]

On Tue, Oct 06, 2009 at 12:23:18PM -0300, Otavio Salvador wrote:
> Hello Denys,
> 
> On Tue, Oct 6, 2009 at 12:46 AM, Denys Dmytriyenko <denis@denix.org> wrote:
> [...]
> > PV1 = "2.6.29-${PR}+gitr${SRCREV}"
> > PV2 = "2.6.29+2.6.30-rc5-${PR}+gitr${SRCREV}"
> > PV3 = "2.6.30-${PR}+gitr${SRCREV}"
> >
> > That still works with opkg, as ipkg-compare-versions sorts above PVs as
> > expected - PV1 < PV2 < PV3
> >
> > But BitBake now has this flaw and sorts like this: PV2 < PV1 < PV3
> [...]
> 
> Please send the patch for people to review and would be nice to have
> it commited since many of us could have been using non-required
> overrides.

The patch was sent right after the first email:

http://thread.gmane.org/gmane.comp.handhelds.openembedded/26691/focus=26692

-- 
Denys

Re: [RFC] Version sorting in BitBake

[Richard Purdie]

On Tue, 2009-10-06 at 12:23 -0300, Otavio Salvador wrote:
> Hello Denys,
> 
> On Tue, Oct 6, 2009 at 12:46 AM, Denys Dmytriyenko <denis@denix.org> wrote:
> [...]
> > PV1 = "2.6.29-${PR}+gitr${SRCREV}"
> > PV2 = "2.6.29+2.6.30-rc5-${PR}+gitr${SRCREV}"
> > PV3 = "2.6.30-${PR}+gitr${SRCREV}"
> >
> > That still works with opkg, as ipkg-compare-versions sorts above PVs as
> > expected - PV1 < PV2 < PV3
> >
> > But BitBake now has this flaw and sorts like this: PV2 < PV1 < PV3
> [...]
> 
> Please send the patch for people to review and would be nice to have
> it commited since many of us could have been using non-required
> overrides.
> 
> Thanks by founding and _fixing_ it :-D

This worries me quite a bit as it means bitbake's version comparison
function is broken. Is there a python version comparison function
somewhere else we can compare bitbake's with to make sure there aren't
any other glitches we have?

Bitbake's and opkg's version handling is meant to model Debian for
reference.

Cheers,

Richard

Re: [RFC] Version sorting in BitBake

[Denys Dmytriyenko]

On Fri, Oct 09, 2009 at 10:03:29AM +0100, Richard Purdie wrote:
> On Tue, 2009-10-06 at 12:23 -0300, Otavio Salvador wrote:
> > Hello Denys,
> > 
> > On Tue, Oct 6, 2009 at 12:46 AM, Denys Dmytriyenko <denis@denix.org> wrote:
> > [...]
> > > PV1 = "2.6.29-${PR}+gitr${SRCREV}"
> > > PV2 = "2.6.29+2.6.30-rc5-${PR}+gitr${SRCREV}"
> > > PV3 = "2.6.30-${PR}+gitr${SRCREV}"
> > >
> > > That still works with opkg, as ipkg-compare-versions sorts above PVs as
> > > expected - PV1 < PV2 < PV3
> > >
> > > But BitBake now has this flaw and sorts like this: PV2 < PV1 < PV3
> > [...]
> > 
> > Please send the patch for people to review and would be nice to have
> > it commited since many of us could have been using non-required
> > overrides.
> > 
> > Thanks by founding and _fixing_ it :-D
> 
> This worries me quite a bit as it means bitbake's version comparison
> function is broken. Is there a python version comparison function
> somewhere else we can compare bitbake's with to make sure there aren't
> any other glitches we have?

I was consulting with the ipkg-utils ipkg-compare-versions.c

> Bitbake's and opkg's version handling is meant to model Debian for
> reference.

We discussed this matter on irc with Phil/pb couple days ago. The agreement 
was that fixing BitBake to adhere to dpkg/opkg (i.e. Debian) version sorting 
rules is a good thing. The remaining question was with rpm, which, according 
to sorting algorithm docs on the Net, does not take into account any 
separators at all.

Phil was suggesting that the sorting algorithm can be adjusted runtime 
depending on the distro or whether debian.bbclass is inherited...
Any preferences of what to do with rpm?

-- 
Denys