bb.utils.vercmp_string_op() vs Python 3.12

bb.utils.vercmp_string_op() vs Python 3.12

[Böszörményi Zoltán]

Hi,

Just a FYI.

After the bitbake-server getting stuck after bitbake exited
with Python 3.12 issue was fixed, here's another one.

This time it's semi-reliably triggered by the jack recipe
in meta-openembedded:

ERROR: jack-1.19.22-r0 do_configure: Error executing a python function in 
exec_func_python() autogenerated:

The stack trace of python calls that resulted in this exception/failure was:
File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
      0001:
  *** 0002:waf_preconfigure(d)
      0003:
File: 
'/home/zozo/test-yocto-4.3-gh/conf/../layers/openembedded-core/meta/classes-recipe/waf.bbclass', 
lineno: 58, function: waf_preconfigure
      0054:    wafbin = os.path.join(subsrcdir, 'waf')
      0055:    try:
      0056:        result = subprocess.check_output([python, wafbin, '--version'], 
cwd=subsrcdir, stderr=subprocess.STDOUT)
      0057:        version = result.decode('utf-8').split()[1]
  *** 0058:        if bb.utils.vercmp_string_op(version, "1.8.7", ">="):
      0059:            d.setVar("WAF_EXTRA_CONF", "--bindir=${bindir} --libdir=${libdir}")
      0060:    except subprocess.CalledProcessError as e:
      0061:        bb.warn("Unable to execute waf --version, exit code %d. Assuming waf 
version without bindir/libdir support." % e.returncode)
      0062:    except FileNotFoundError:
File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py', lineno: 150, function: 
vercmp_string_op
      0146:    Compare two versions and check if the specified comparison operator matches 
the result of the comparison.
      0147:    This function is fairly liberal about what operators it will accept since 
there are a variety of styles
      0148:    depending on the context.
      0149:    """
  *** 0150:    res = vercmp_string(a, b)
      0151:    if op in ('=', '=='):
      0152:        return res == 0
      0153:    elif op == '<=':
      0154:        return res <= 0
File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py', lineno: 140, function: 
vercmp_string
      0136:    return r
      0137:
      0138:def vercmp_string(a, b):
      0139:    """ Split version strings and compare them """
  *** 0140:    ta = split_version(a)
      0141:    tb = split_version(b)
      0142:    return vercmp(ta, tb)
      0143:
      0144:def vercmp_string_op(a, b, op):
File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py', lineno: 91, function: 
split_version
      0087:    """Split a version string into its constituent parts (PE, PV, PR)"""
      0088:    s = s.strip(" <>=")
      0089:    e = 0
      0090:    if s.count(':'):
  *** 0091:        e = int(s.split(":")[0])
      0092:        s = s.split(":")[1]
      0093:    r = ""
      0094:    if s.count('-'):
      0095:        r = s.rsplit("-", 1)[1]
Exception: ValueError: invalid literal for int() with base 10: 'SyntaxWarning'

ERROR: Logfile of failure stored in: 
/home/zozo/test-yocto-4.3-gh/tmp-sicom-glibc/work/corei7-64-oe-linux/jack/1.19.22/temp/log.do_configure.1709417

"Semi-reliably", i.e. about 50-50 or 60-40 percent failure vs success rate.
Usually the second run succeeds after a failure.

I copied vercmp_string_op() and all others called by it
from bitbake/lib/bb/utils.py into a new python script as a
minimum test case. Running this script inside a locally
cloned jack git repository has 100% success rate.

It looks like an internal Python "flake" issue.

Best regards,
Zoltán Böszörményi

Re: [OE-core] bb.utils.vercmp_string_op() vs Python 3.12

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 4635 bytes --]

> I copied vercmp_string_op() and all others called by it from
bitbake/lib/bb/utils.py into a new python script.

It's not an issue in vercmp_string_op itself, but the version value it gets
from waf --version isn't just the version itself but also SyntaxWarning
from waf.

So you need to update waf in jack not to trigger SyntaxWarning when waf
--version is called.

On Fri, Jan 26, 2024 at 10:17 AM Zoltan Boszormenyi <zboszor@gmail.com>
wrote:

> Hi,
>
> Just a FYI.
>
> After the bitbake-server getting stuck after bitbake exited
> with Python 3.12 issue was fixed, here's another one.
>
> This time it's semi-reliably triggered by the jack recipe
> in meta-openembedded:
>
> ERROR: jack-1.19.22-r0 do_configure: Error executing a python function in
> exec_func_python() autogenerated:
>
> The stack trace of python calls that resulted in this exception/failure
> was:
> File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
>       0001:
>   *** 0002:waf_preconfigure(d)
>       0003:
> File:
> '/home/zozo/test-yocto-4.3-gh/conf/../layers/openembedded-core/meta/classes-recipe/waf.bbclass',
>
> lineno: 58, function: waf_preconfigure
>       0054:    wafbin = os.path.join(subsrcdir, 'waf')
>       0055:    try:
>       0056:        result = subprocess.check_output([python, wafbin,
> '--version'],
> cwd=subsrcdir, stderr=subprocess.STDOUT)
>       0057:        version = result.decode('utf-8').split()[1]
>   *** 0058:        if bb.utils.vercmp_string_op(version, "1.8.7", ">="):
>       0059:            d.setVar("WAF_EXTRA_CONF", "--bindir=${bindir}
> --libdir=${libdir}")
>       0060:    except subprocess.CalledProcessError as e:
>       0061:        bb.warn("Unable to execute waf --version, exit code %d.
> Assuming waf
> version without bindir/libdir support." % e.returncode)
>       0062:    except FileNotFoundError:
> File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py', lineno: 150,
> function:
> vercmp_string_op
>       0146:    Compare two versions and check if the specified comparison
> operator matches
> the result of the comparison.
>       0147:    This function is fairly liberal about what operators it
> will accept since
> there are a variety of styles
>       0148:    depending on the context.
>       0149:    """
>   *** 0150:    res = vercmp_string(a, b)
>       0151:    if op in ('=', '=='):
>       0152:        return res == 0
>       0153:    elif op == '<=':
>       0154:        return res <= 0
> File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py', lineno: 140,
> function:
> vercmp_string
>       0136:    return r
>       0137:
>       0138:def vercmp_string(a, b):
>       0139:    """ Split version strings and compare them """
>   *** 0140:    ta = split_version(a)
>       0141:    tb = split_version(b)
>       0142:    return vercmp(ta, tb)
>       0143:
>       0144:def vercmp_string_op(a, b, op):
> File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py', lineno: 91,
> function:
> split_version
>       0087:    """Split a version string into its constituent parts (PE,
> PV, PR)"""
>       0088:    s = s.strip(" <>=")
>       0089:    e = 0
>       0090:    if s.count(':'):
>   *** 0091:        e = int(s.split(":")[0])
>       0092:        s = s.split(":")[1]
>       0093:    r = ""
>       0094:    if s.count('-'):
>       0095:        r = s.rsplit("-", 1)[1]
> Exception: ValueError: invalid literal for int() with base 10:
> 'SyntaxWarning'
>
> ERROR: Logfile of failure stored in:
>
> /home/zozo/test-yocto-4.3-gh/tmp-sicom-glibc/work/corei7-64-oe-linux/jack/1.19.22/temp/log.do_configure.1709417
>
> "Semi-reliably", i.e. about 50-50 or 60-40 percent failure vs success rate.
> Usually the second run succeeds after a failure.
>
> I copied vercmp_string_op() and all others called by it
> from bitbake/lib/bb/utils.py into a new python script as a
> minimum test case. Running this script inside a locally
> cloned jack git repository has 100% success rate.
>
> It looks like an internal Python "flake" issue.
>
> Best regards,
> Zoltán Böszörményi
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#194348):
> https://lists.openembedded.org/g/openembedded-core/message/194348
> Mute This Topic: https://lists.openembedded.org/mt/103973063/3617156
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> martin.jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 6241 bytes --]

Re: [OE-core] bb.utils.vercmp_string_op() vs Python 3.12

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 4974 bytes --]

See
https://git.openembedded.org/meta-openembedded/commit/?h=kirkstone&id=8a042b540db5421785edcf21d5114be6358246fd

On Fri, Jan 26, 2024 at 11:53 AM Martin Jansa <martin.jansa@gmail.com>
wrote:

> > I copied vercmp_string_op() and all others called by it from
> bitbake/lib/bb/utils.py into a new python script.
>
> It's not an issue in vercmp_string_op itself, but the version value it
> gets from waf --version isn't just the version itself but also
> SyntaxWarning from waf.
>
> So you need to update waf in jack not to trigger SyntaxWarning when waf
> --version is called.
>
> On Fri, Jan 26, 2024 at 10:17 AM Zoltan Boszormenyi <zboszor@gmail.com>
> wrote:
>
>> Hi,
>>
>> Just a FYI.
>>
>> After the bitbake-server getting stuck after bitbake exited
>> with Python 3.12 issue was fixed, here's another one.
>>
>> This time it's semi-reliably triggered by the jack recipe
>> in meta-openembedded:
>>
>> ERROR: jack-1.19.22-r0 do_configure: Error executing a python function in
>> exec_func_python() autogenerated:
>>
>> The stack trace of python calls that resulted in this exception/failure
>> was:
>> File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
>>       0001:
>>   *** 0002:waf_preconfigure(d)
>>       0003:
>> File:
>> '/home/zozo/test-yocto-4.3-gh/conf/../layers/openembedded-core/meta/classes-recipe/waf.bbclass',
>>
>> lineno: 58, function: waf_preconfigure
>>       0054:    wafbin = os.path.join(subsrcdir, 'waf')
>>       0055:    try:
>>       0056:        result = subprocess.check_output([python, wafbin,
>> '--version'],
>> cwd=subsrcdir, stderr=subprocess.STDOUT)
>>       0057:        version = result.decode('utf-8').split()[1]
>>   *** 0058:        if bb.utils.vercmp_string_op(version, "1.8.7", ">="):
>>       0059:            d.setVar("WAF_EXTRA_CONF", "--bindir=${bindir}
>> --libdir=${libdir}")
>>       0060:    except subprocess.CalledProcessError as e:
>>       0061:        bb.warn("Unable to execute waf --version, exit code
>> %d. Assuming waf
>> version without bindir/libdir support." % e.returncode)
>>       0062:    except FileNotFoundError:
>> File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py', lineno:
>> 150, function:
>> vercmp_string_op
>>       0146:    Compare two versions and check if the specified comparison
>> operator matches
>> the result of the comparison.
>>       0147:    This function is fairly liberal about what operators it
>> will accept since
>> there are a variety of styles
>>       0148:    depending on the context.
>>       0149:    """
>>   *** 0150:    res = vercmp_string(a, b)
>>       0151:    if op in ('=', '=='):
>>       0152:        return res == 0
>>       0153:    elif op == '<=':
>>       0154:        return res <= 0
>> File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py', lineno:
>> 140, function:
>> vercmp_string
>>       0136:    return r
>>       0137:
>>       0138:def vercmp_string(a, b):
>>       0139:    """ Split version strings and compare them """
>>   *** 0140:    ta = split_version(a)
>>       0141:    tb = split_version(b)
>>       0142:    return vercmp(ta, tb)
>>       0143:
>>       0144:def vercmp_string_op(a, b, op):
>> File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py', lineno: 91,
>> function:
>> split_version
>>       0087:    """Split a version string into its constituent parts (PE,
>> PV, PR)"""
>>       0088:    s = s.strip(" <>=")
>>       0089:    e = 0
>>       0090:    if s.count(':'):
>>   *** 0091:        e = int(s.split(":")[0])
>>       0092:        s = s.split(":")[1]
>>       0093:    r = ""
>>       0094:    if s.count('-'):
>>       0095:        r = s.rsplit("-", 1)[1]
>> Exception: ValueError: invalid literal for int() with base 10:
>> 'SyntaxWarning'
>>
>> ERROR: Logfile of failure stored in:
>>
>> /home/zozo/test-yocto-4.3-gh/tmp-sicom-glibc/work/corei7-64-oe-linux/jack/1.19.22/temp/log.do_configure.1709417
>>
>> "Semi-reliably", i.e. about 50-50 or 60-40 percent failure vs success
>> rate.
>> Usually the second run succeeds after a failure.
>>
>> I copied vercmp_string_op() and all others called by it
>> from bitbake/lib/bb/utils.py into a new python script as a
>> minimum test case. Running this script inside a locally
>> cloned jack git repository has 100% success rate.
>>
>> It looks like an internal Python "flake" issue.
>>
>> Best regards,
>> Zoltán Böszörményi
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#194348):
>> https://lists.openembedded.org/g/openembedded-core/message/194348
>> Mute This Topic: https://lists.openembedded.org/mt/103973063/3617156
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
>> martin.jansa@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
>>

[-- Attachment #2: Type: text/html, Size: 6883 bytes --]

Re: [OE-core] bb.utils.vercmp_string_op() vs Python 3.12

[Böszörményi Zoltán]

2024. 01. 26. 11:53 keltezéssel, Martin Jansa írta:
> > I copied vercmp_string_op() and all others called by it from bitbake/lib/bb/utils.py 
> into a new python script.
>
> It's not an issue in vercmp_string_op itself,

I already know this.
Apparently you haven't read my mail fully for the conclusion.

> but the version value it gets from waf --version isn't just the version itself but also 
> SyntaxWarning from waf.

That is not a problem in itself as the preceding call only collects the stdout:

result = subprocess.check_output([python, wafbin, '--version'], cwd=subsrcdir, 
stderr=subprocess.STDOUT)

SyntaxWarning goes (or should go) to stderr.

When this part is moved into a minimal test case, there is no error at all.

The error only happens inside bitbake and there was no such problem
with Python 3.11.x or earlier.

Ergo, the problem must be a Python 3.12 issue.
The actual version here is python3-3.12.1-2.fc39.x86_64 for me.

> So you need to update waf in jack not to trigger SyntaxWarning when waf --version is called.
>
> On Fri, Jan 26, 2024 at 10:17 AM Zoltan Boszormenyi <zboszor@gmail.com> wrote:
>
>     Hi,
>
>     Just a FYI.
>
>     After the bitbake-server getting stuck after bitbake exited
>     with Python 3.12 issue was fixed, here's another one.
>
>     This time it's semi-reliably triggered by the jack recipe
>     in meta-openembedded:
>
>     ERROR: jack-1.19.22-r0 do_configure: Error executing a python function in
>     exec_func_python() autogenerated:
>
>     The stack trace of python calls that resulted in this exception/failure was:
>     File: 'exec_func_python() autogenerated', lineno: 2, function: <module>
>           0001:
>       *** 0002:waf_preconfigure(d)
>           0003:
>     File:
>     '/home/zozo/test-yocto-4.3-gh/conf/../layers/openembedded-core/meta/classes-recipe/waf.bbclass',
>
>     lineno: 58, function: waf_preconfigure
>           0054:    wafbin = os.path.join(subsrcdir, 'waf')
>           0055:    try:
>           0056:        result = subprocess.check_output([python, wafbin, '--version'],
>     cwd=subsrcdir, stderr=subprocess.STDOUT)
>           0057:        version = result.decode('utf-8').split()[1]
>       *** 0058:        if bb.utils.vercmp_string_op(version, "1.8.7", ">="):
>           0059:            d.setVar("WAF_EXTRA_CONF", "--bindir=${bindir}
>     --libdir=${libdir}")
>           0060:    except subprocess.CalledProcessError as e:
>           0061:        bb.warn("Unable to execute waf --version, exit code %d. Assuming waf
>     version without bindir/libdir support." % e.returncode)
>           0062:    except FileNotFoundError:
>     File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py', lineno: 150, function:
>     vercmp_string_op
>           0146:    Compare two versions and check if the specified comparison operator
>     matches
>     the result of the comparison.
>           0147:    This function is fairly liberal about what operators it will accept
>     since
>     there are a variety of styles
>           0148:    depending on the context.
>           0149:    """
>       *** 0150:    res = vercmp_string(a, b)
>           0151:    if op in ('=', '=='):
>           0152:        return res == 0
>           0153:    elif op == '<=':
>           0154:        return res <= 0
>     File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py', lineno: 140, function:
>     vercmp_string
>           0136:    return r
>           0137:
>           0138:def vercmp_string(a, b):
>           0139:    """ Split version strings and compare them """
>       *** 0140:    ta = split_version(a)
>           0141:    tb = split_version(b)
>           0142:    return vercmp(ta, tb)
>           0143:
>           0144:def vercmp_string_op(a, b, op):
>     File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py', lineno: 91, function:
>     split_version
>           0087:    """Split a version string into its constituent parts (PE, PV, PR)"""
>           0088:    s = s.strip(" <>=")
>           0089:    e = 0
>           0090:    if s.count(':'):
>       *** 0091:        e = int(s.split(":")[0])
>           0092:        s = s.split(":")[1]
>           0093:    r = ""
>           0094:    if s.count('-'):
>           0095:        r = s.rsplit("-", 1)[1]
>     Exception: ValueError: invalid literal for int() with base 10: 'SyntaxWarning'
>
>     ERROR: Logfile of failure stored in:
>     /home/zozo/test-yocto-4.3-gh/tmp-sicom-glibc/work/corei7-64-oe-linux/jack/1.19.22/temp/log.do_configure.1709417
>
>     "Semi-reliably", i.e. about 50-50 or 60-40 percent failure vs success rate.
>     Usually the second run succeeds after a failure.
>
>     I copied vercmp_string_op() and all others called by it
>     from bitbake/lib/bb/utils.py into a new python script as a
>     minimum test case. Running this script inside a locally
>     cloned jack git repository has 100% success rate.
>
>     It looks like an internal Python "flake" issue.
>
>     Best regards,
>     Zoltán Böszörményi
>
>
>     -=-=-=-=-=-=-=-=-=-=-=-
>     Links: You receive all messages sent to this group.
>     View/Reply Online (#194348):
>     https://lists.openembedded.org/g/openembedded-core/message/194348
>     Mute This Topic: https://lists.openembedded.org/mt/103973063/3617156
>     Group Owner: openembedded-core+owner@lists.openembedded.org
>     <mailto:openembedded-core%2Bowner@lists.openembedded.org>
>     Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
>     [martin.jansa@gmail.com]
>     -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] bb.utils.vercmp_string_op() vs Python 3.12

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 9165 bytes --]

On Fri, Jan 26, 2024 at 12:14 PM Böszörményi Zoltán <zboszor@gmail.com>
wrote:

> 2024. 01. 26. 11:53 keltezéssel, Martin Jansa írta:
> > > I copied vercmp_string_op() and all others called by it from
> bitbake/lib/bb/utils.py
> > into a new python script.
> >
> > It's not an issue in vercmp_string_op itself,
>
> I already know this.
> Apparently you haven't read my mail fully for the conclusion.
>

I did and you said that you were testing vercmp_string_op in jack, instead
of looking at waf --version.

> but the version value it gets from waf --version isn't just the version
> itself but also
> > SyntaxWarning from waf.
>
> That is not a problem in itself as the preceding call only collects the
> stdout:
>
> result = subprocess.check_output([python, wafbin, '--version'],
> cwd=subsrcdir,
> stderr=subprocess.STDOUT)
>
> SyntaxWarning goes (or should go) to stderr.
>

Which goes to stdout as configured in this call.

After applying
https://gitlab.com/ita1024/waf/-/commit/d2060dfd8af4edb5824153ff24e207b39ecd67a2.patch
to fix:
| DEBUG: Executing shell function do_configure
| Traceback (most recent call last):
|   File
"/OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/jack/1.9.22/git/./waf",
line 166, in <module>
|     from waflib import Scripting
|   File
"/OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/jack/1.9.22/git/waflib/Scripting.py",
line 10, in <module>
|     from waflib import Utils, Configure, Logs, Options, ConfigSet,
Context, Errors, Build, Node
|   File
"/OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/jack/1.9.22/git/waflib/Configure.py",
line 16, in <module>
|     from waflib import ConfigSet, Utils, Options, Logs, Context, Build,
Errors
|   File
"/OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/jack/1.9.22/git/waflib/Options.py",
line 14, in <module>
|     from waflib import Logs, Utils, Context, Errors
|   File
"/OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/jack/1.9.22/git/waflib/Context.py",
line 9, in <module>
|     import os, re, imp, sys
| ModuleNotFoundError: No module named 'imp'

martin@jama
/OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/jack/1.9.22/git $ git
diff
diff --git a/waflib/Context.py b/waflib/Context.py
index 761b521f..38ab03f1 100644
--- a/waflib/Context.py
+++ b/waflib/Context.py
@@ -6,10 +6,17 @@
 Classes and functions enabling the command system
 """

-import os, re, imp, sys
+import os, re, sys
 from waflib import Utils, Errors, Logs
 import waflib.Node

+if sys.hexversion > 0x3040000:
+ import types
+ class imp(object):
+         new_module = lambda x: types.ModuleType(x)
+else:
+ import imp
+
 # the following 3 constants are updated on each new release (do not touch)
 HEXVERSION=0x2000c00
 """Constant updated on new releases"""

The SyntaxWarning is easily reproducible:

martin@jama
/OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/jack/1.9.22/git $
python3 waf --version
/OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/jack/1.9.22/git/waflib/Context.py:617:
SyntaxWarning: invalid escape sequence '\_'
  """
/OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/jack/1.9.22/git/waflib/Build.py:107:
SyntaxWarning: invalid escape sequence '\*'
  """List of targets to build (default: \*)"""
/OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/jack/1.9.22/git/waflib/Task.py:1047:
SyntaxWarning: invalid escape sequence '\w'
  re_cond = re.compile('(?P<var>\w+)|(?P<or>\|)|(?P<and>&)')
/OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/jack/1.9.22/git/waflib/TaskGen.py:730:
SyntaxWarning: invalid escape sequence '\w'
  re_m4 = re.compile('@(\w+)@', re.M)
waf 2.0.12 (54841218840ffa34fddf834680a5a17db69caa12)

martin@jama
/OE/build/oe-core/tmp-glibc/work/core2-64-oe-linux/jack/1.9.22/git $
python3 --version
Python 3.12.1

When this part is moved into a minimal test case, there is no error at all.
>
> The error only happens inside bitbake and there was no such problem
> with Python 3.11.x or earlier.
>
> Ergo, the problem must be a Python 3.12 issue.
> The actual version here is python3-3.12.1-2.fc39.x86_64 for me.
>
> > So you need to update waf in jack not to trigger SyntaxWarning when waf
> --version is called.
> >
> > On Fri, Jan 26, 2024 at 10:17 AM Zoltan Boszormenyi <zboszor@gmail.com>
> wrote:
> >
> >     Hi,
> >
> >     Just a FYI.
> >
> >     After the bitbake-server getting stuck after bitbake exited
> >     with Python 3.12 issue was fixed, here's another one.
> >
> >     This time it's semi-reliably triggered by the jack recipe
> >     in meta-openembedded:
> >
> >     ERROR: jack-1.19.22-r0 do_configure: Error executing a python
> function in
> >     exec_func_python() autogenerated:
> >
> >     The stack trace of python calls that resulted in this
> exception/failure was:
> >     File: 'exec_func_python() autogenerated', lineno: 2, function:
> <module>
> >           0001:
> >       *** 0002:waf_preconfigure(d)
> >           0003:
> >     File:
> >
>  '/home/zozo/test-yocto-4.3-gh/conf/../layers/openembedded-core/meta/classes-recipe/waf.bbclass',
> >
> >     lineno: 58, function: waf_preconfigure
> >           0054:    wafbin = os.path.join(subsrcdir, 'waf')
> >           0055:    try:
> >           0056:        result = subprocess.check_output([python, wafbin,
> '--version'],
> >     cwd=subsrcdir, stderr=subprocess.STDOUT)
> >           0057:        version = result.decode('utf-8').split()[1]
> >       *** 0058:        if bb.utils.vercmp_string_op(version, "1.8.7",
> ">="):
> >           0059:            d.setVar("WAF_EXTRA_CONF", "--bindir=${bindir}
> >     --libdir=${libdir}")
> >           0060:    except subprocess.CalledProcessError as e:
> >           0061:        bb.warn("Unable to execute waf --version, exit
> code %d. Assuming waf
> >     version without bindir/libdir support." % e.returncode)
> >           0062:    except FileNotFoundError:
> >     File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py',
> lineno: 150, function:
> >     vercmp_string_op
> >           0146:    Compare two versions and check if the specified
> comparison operator
> >     matches
> >     the result of the comparison.
> >           0147:    This function is fairly liberal about what operators
> it will accept
> >     since
> >     there are a variety of styles
> >           0148:    depending on the context.
> >           0149:    """
> >       *** 0150:    res = vercmp_string(a, b)
> >           0151:    if op in ('=', '=='):
> >           0152:        return res == 0
> >           0153:    elif op == '<=':
> >           0154:        return res <= 0
> >     File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py',
> lineno: 140, function:
> >     vercmp_string
> >           0136:    return r
> >           0137:
> >           0138:def vercmp_string(a, b):
> >           0139:    """ Split version strings and compare them """
> >       *** 0140:    ta = split_version(a)
> >           0141:    tb = split_version(b)
> >           0142:    return vercmp(ta, tb)
> >           0143:
> >           0144:def vercmp_string_op(a, b, op):
> >     File: '/home/zozo/test-yocto-4.3-gh/bitbake/lib/bb/utils.py',
> lineno: 91, function:
> >     split_version
> >           0087:    """Split a version string into its constituent parts
> (PE, PV, PR)"""
> >           0088:    s = s.strip(" <>=")
> >           0089:    e = 0
> >           0090:    if s.count(':'):
> >       *** 0091:        e = int(s.split(":")[0])
> >           0092:        s = s.split(":")[1]
> >           0093:    r = ""
> >           0094:    if s.count('-'):
> >           0095:        r = s.rsplit("-", 1)[1]
> >     Exception: ValueError: invalid literal for int() with base 10:
> 'SyntaxWarning'
> >
> >     ERROR: Logfile of failure stored in:
> >
>  /home/zozo/test-yocto-4.3-gh/tmp-sicom-glibc/work/corei7-64-oe-linux/jack/1.19.22/temp/log.do_configure.1709417
> >
> >     "Semi-reliably", i.e. about 50-50 or 60-40 percent failure vs
> success rate.
> >     Usually the second run succeeds after a failure.
> >
> >     I copied vercmp_string_op() and all others called by it
> >     from bitbake/lib/bb/utils.py into a new python script as a
> >     minimum test case. Running this script inside a locally
> >     cloned jack git repository has 100% success rate.
> >
> >     It looks like an internal Python "flake" issue.
> >
> >     Best regards,
> >     Zoltán Böszörményi
> >
> >
> >     -=-=-=-=-=-=-=-=-=-=-=-
> >     Links: You receive all messages sent to this group.
> >     View/Reply Online (#194348):
> >     https://lists.openembedded.org/g/openembedded-core/message/194348
> >     Mute This Topic: https://lists.openembedded.org/mt/103973063/3617156
> >     Group Owner: openembedded-core+owner@lists.openembedded.org
> >     <mailto:openembedded-core%2Bowner@lists.openembedded.org>
> >     Unsubscribe:
> https://lists.openembedded.org/g/openembedded-core/unsub
> >     [martin.jansa@gmail.com]
> >     -=-=-=-=-=-=-=-=-=-=-=-
> >
>
>

[-- Attachment #2: Type: text/html, Size: 12473 bytes --]

Re: [OE-core] bb.utils.vercmp_string_op() vs Python 3.12

[Böszörményi Zoltán]

2024. 01. 26. 12:14 keltezéssel, Zoltan Boszormenyi via lists.openembedded.org írta:
> 2024. 01. 26. 11:53 keltezéssel, Martin Jansa írta:
>> > I copied vercmp_string_op() and all others called by it from bitbake/lib/bb/utils.py 
>> into a new python script.
>>
>> It's not an issue in vercmp_string_op itself,

Here's what the internal waf in jack 1.9.22 says:

zozo@localhost:~/jack2$ python3 waf --version
waf 2.0.12 (54841218840ffa34fddf834680a5a17db69caa12)
zozo@localhost:~/jack2$ python3 --version
Python 3.12.1

No SyntaxWarning emitted, no matter how many times I try.
This is why I think it's an internal python 3.12.x issue.