[PATCH 1/3] data_smart: Add missing regexp markup

[PATCH 1/3] data_smart: Add missing regexp markup

[Richard Purdie]

Fix some further python3 warnings about unescaped regexs.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data_smart.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 67af38050e..297a2f45b4 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -39,10 +39,10 @@ from bb.COW  import COWDictBase
 logger = logging.getLogger("BitBake.Data")
 
 __setvar_keyword__ = ["_append", "_prepend", "_remove"]
-__setvar_regexp__ = re.compile('(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
+__setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
 __expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
 __expand_python_regexp__ = re.compile(r"\${@.+?}")
-__whitespace_split__ = re.compile('(\s)')
+__whitespace_split__ = re.compile(r'(\s)')
 
 def infer_caller_details(loginfo, parent = False, varval = True):
     """Save the caller the trouble of specifying everything."""
-- 
2.19.1

[PATCH 2/3] data_smart: Allow numeric characters in overrides

[Richard Purdie]

We're seeing problems due to the way x86-64 is handled (or not handled)
as an override. Relax the containts on overrides from being lowercase
to being lowercase or numeric. This fixes problem where MACHINE=qemux86
would work but MACHINE=qemux86-64 would fail the same tests.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/data_smart.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 297a2f45b4..c342adaa0a 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -43,6 +43,7 @@ __setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>_append|_prepend|_remo
 __expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
 __expand_python_regexp__ = re.compile(r"\${@.+?}")
 __whitespace_split__ = re.compile(r'(\s)')
+__override_regexp__ = re.compile(r'[a-z0-9]+')
 
 def infer_caller_details(loginfo, parent = False, varval = True):
     """Save the caller the trouble of specifying everything."""
@@ -597,7 +598,7 @@ class DataSmart(MutableMapping):
         # aka pay the cookie monster
         override = var[var.rfind('_')+1:]
         shortvar = var[:var.rfind('_')]
-        while override and override.islower():
+        while override and __override_regexp__.match(override):
             if shortvar not in self.overridedata:
                 self.overridedata[shortvar] = []
             if [var, override] not in self.overridedata[shortvar]:
-- 
2.19.1

[PATCH 3/3] utils: Add aarch64 support to ioprio_set

[Richard Purdie]

With aarch64 hosts coming into use, set the syscall number to
avoid ioprio warnings on that platform.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/utils.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index f687ee4127..9cb702dbb7 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -1497,6 +1497,8 @@ def ioprio_set(who, cls, value):
       NR_ioprio_set = 251
     elif _unamearch[0] == "i" and _unamearch[2:3] == "86":
       NR_ioprio_set = 289
+    elif _unamearch == "aarch64":
+      NR_ioprio_set = 30
 
     if NR_ioprio_set:
         ioprio = value | (cls << IOPRIO_CLASS_SHIFT)
-- 
2.19.1

Re: [PATCH 2/3] data_smart: Allow numeric characters in overrides

[Peter Kjellerstedt]

> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org <bitbake-devel-
> bounces@lists.openembedded.org> On Behalf Of Richard Purdie
> Sent: den 14 december 2018 15:02
> To: bitbake-devel@lists.openembedded.org
> Subject: [bitbake-devel] [PATCH 2/3] data_smart: Allow numeric
> characters in overrides
> 
> We're seeing problems due to the way x86-64 is handled (or not handled)
> as an override. Relax the containts on overrides from being lowercase
> to being lowercase or numeric. This fixes problem where MACHINE=qemux86
> would work but MACHINE=qemux86-64 would fail the same tests.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  lib/bb/data_smart.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
> index 297a2f45b4..c342adaa0a 100644
> --- a/lib/bb/data_smart.py
> +++ b/lib/bb/data_smart.py
> @@ -43,6 +43,7 @@ __setvar_regexp__ =
> re.compile(r'(?P<base>.*?)(?P<keyword>_append|_prepend|_remo
>  __expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
>  __expand_python_regexp__ = re.compile(r"\${@.+?}")
>  __whitespace_split__ = re.compile(r'(\s)')
> +__override_regexp__ = re.compile(r'[a-z0-9]+')
> 
>  def infer_caller_details(loginfo, parent = False, varval = True):
>      """Save the caller the trouble of specifying everything."""
> @@ -597,7 +598,7 @@ class DataSmart(MutableMapping):
>          # aka pay the cookie monster
>          override = var[var.rfind('_')+1:]
>          shortvar = var[:var.rfind('_')]
> -        while override and override.islower():
> +        while override and __override_regexp__.match(override):
>              if shortvar not in self.overridedata:
>                  self.overridedata[shortvar] = []
>              if [var, override] not in self.overridedata[shortvar]:
> --
> 2.19.1

I do not understand this commit. The commit explanation and the code change 
don't match up. The motivation in the commit message is that there was 
problems with overrides such as x86-64 containing digits and that the 
code should be changed to allow overrides to be lower case and numeric.
However, the modified code changes the test from using islower(), which as 
far as I can tell only validates the alpha-characters in the string and 
ignores all else, to using a regular expression r'[a-z0-9]+', which will 
match as long as the first character is a lower case character or a numeric 
character (note that there is no $ at the end of the regular expression).

Here are tests with islower(), which correctly validates "x86" and "x86-64" 
as lower case overrides, but not "fooBar":

>>> "x86".islower()
True
>>> "x86-64".islower()
True
>>> "fooBar".islower()
False

Here are the corresponding tests using the r'[a-z0-9]+' regular expression.
Note that it only matches "x86" from "x86-64" and "foo" from "fooBar", and 
will incorrectly allow "fooBar" as an override:

>>> re.match(r'[a-z0-9]+', "x86")
<_sre.SRE_Match object; span=(0, 3), match='x86'>
>>> re.match(r'[a-z0-9]+', "x86-64")
<_sre.SRE_Match object; span=(0, 3), match='x86'>
>>> re.match(r'[a-z0-9]+', "fooBar")
<_sre.SRE_Match object; span=(0, 3), match='foo'>

So why change from islower(), which as far as I can tell did the right thing 
from the beginning?

//Peter

Re: [PATCH 2/3] data_smart: Allow numeric characters in overrides

[richard.purdie]

On Tue, 2018-12-18 at 21:05 +0000, Peter Kjellerstedt wrote:
> I do not understand this commit. The commit explanation and the code change 
> don't match up. The motivation in the commit message is that there was 
> problems with overrides such as x86-64 containing digits and that the 
> code should be changed to allow overrides to be lower case and numeric.
> However, the modified code changes the test from using islower(), which as 
> far as I can tell only validates the alpha-characters in the string and 
> ignores all else, to using a regular expression r'[a-z0-9]+', which will 
> match as long as the first character is a lower case character or a numeric 
> character (note that there is no $ at the end of the regular expression).
> 
> Here are tests with islower(), which correctly validates "x86" and
> "x86-64" 
> as lower case overrides, but not "fooBar":
> 
> > > > "x86".islower()
> True
> > > > "x86-64".islower()
> True
> > > > "fooBar".islower()
> False
> 
> Here are the corresponding tests using the r'[a-z0-9]+' regular
> expression.
> Note that it only matches "x86" from "x86-64" and "foo" from
> "fooBar", and 
> will incorrectly allow "fooBar" as an override:
> 
> > > > re.match(r'[a-z0-9]+', "x86")
> <_sre.SRE_Match object; span=(0, 3), match='x86'>
> > > > re.match(r'[a-z0-9]+', "x86-64")
> <_sre.SRE_Match object; span=(0, 3), match='x86'>
> > > > re.match(r'[a-z0-9]+', "fooBar")
> <_sre.SRE_Match object; span=(0, 3), match='foo'>
> 
> So why change from islower(), which as far as I can tell did the
> right thing from the beginning?

The commit is clearly wrong. The intent of the commit message is
correct, it doesn't match what that regex actually does :(

Cheers,

Richard

Re: [PATCH 2/3] data_smart: Allow numeric characters in overrides

[Peter Kjellerstedt]

> -----Original Message-----
> From: richard.purdie@linuxfoundation.org
> <richard.purdie@linuxfoundation.org>
> Sent: den 18 december 2018 22:17
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; bitbake-
> devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [PATCH 2/3] data_smart: Allow numeric
> characters in overrides
> 
> On Tue, 2018-12-18 at 21:05 +0000, Peter Kjellerstedt wrote:
> > I do not understand this commit. The commit explanation and the code
> change
> > don't match up. The motivation in the commit message is that there
> was
> > problems with overrides such as x86-64 containing digits and that the
> > code should be changed to allow overrides to be lower case and
> numeric.
> > However, the modified code changes the test from using islower(),
> which as
> > far as I can tell only validates the alpha-characters in the string
> and
> > ignores all else, to using a regular expression r'[a-z0-9]+', which
> will
> > match as long as the first character is a lower case character or a
> numeric
> > character (note that there is no $ at the end of the regular
> expression).
> >
> > Here are tests with islower(), which correctly validates "x86" and
> > "x86-64"
> > as lower case overrides, but not "fooBar":
> >
> > > > > "x86".islower()
> > True
> > > > > "x86-64".islower()
> > True
> > > > > "fooBar".islower()
> > False
> >
> > Here are the corresponding tests using the r'[a-z0-9]+' regular
> > expression.
> > Note that it only matches "x86" from "x86-64" and "foo" from
> > "fooBar", and
> > will incorrectly allow "fooBar" as an override:
> >
> > > > > re.match(r'[a-z0-9]+', "x86")
> > <_sre.SRE_Match object; span=(0, 3), match='x86'>
> > > > > re.match(r'[a-z0-9]+', "x86-64")
> > <_sre.SRE_Match object; span=(0, 3), match='x86'>
> > > > > re.match(r'[a-z0-9]+', "fooBar")
> > <_sre.SRE_Match object; span=(0, 3), match='foo'>
> >
> > So why change from islower(), which as far as I can tell did the
> > right thing from the beginning?
> 
> The commit is clearly wrong. The intent of the commit message is
> correct, it doesn't match what that regex actually does :(
> 
> Cheers,
> 
> Richard

I don't understand the commit message either. It says that there are 
problems with x86-64 because it contains digits, but x86 (which the 
commit message says works) contains digits too, so that cannot really 
be the case. If it is actually the dash in x86-64 that is supposedly 
a cause of problems, then islower() ignores both digits and non-alpha 
characters like the dash so that cannot really be a reason to change 
the code either. All in all I'm confused by this commit and what it 
is trying to solve...

//Peter

Re: [PATCH 2/3] data_smart: Allow numeric characters in overrides

[richard.purdie]

On Wed, 2018-12-19 at 10:55 +0000, Peter Kjellerstedt wrote:
> I don't understand the commit message either. It says that there are 
> problems with x86-64 because it contains digits, but x86 (which the 
> commit message says works) contains digits too, so that cannot
> really 
> be the case. If it is actually the dash in x86-64 that is supposedly 
> a cause of problems, then islower() ignores both digits and non-
> alpha 
> characters like the dash so that cannot really be a reason to change 
> the code either. All in all I'm confused by this commit and what it 
> is trying to solve...

Consider this output from python:

>>> "64".islower()
False
>>> "x64".islower()
True
>>> "x64A".islower()
False
>>> 

The problem is that "64".islower() returns False when we need True
(overrides get split by '_' so x86_64 turns into "x86" and "64" in the
code in question).

Cheers,

Richard

Re: [PATCH 2/3] data_smart: Allow numeric characters in overrides

[Peter Kjellerstedt]

> -----Original Message-----
> From: richard.purdie@linuxfoundation.org
> <richard.purdie@linuxfoundation.org>
> Sent: den 19 december 2018 12:42
> To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; bitbake-
> devel@lists.openembedded.org
> Subject: Re: [bitbake-devel] [PATCH 2/3] data_smart: Allow numeric
> characters in overrides
> 
> On Wed, 2018-12-19 at 10:55 +0000, Peter Kjellerstedt wrote:
> > I don't understand the commit message either. It says that there are
> > problems with x86-64 because it contains digits, but x86 (which the
> > commit message says works) contains digits too, so that cannot
> > really
> > be the case. If it is actually the dash in x86-64 that is supposedly
> > a cause of problems, then islower() ignores both digits and non-
> > alpha
> > characters like the dash so that cannot really be a reason to change
> > the code either. All in all I'm confused by this commit and what it
> > is trying to solve...
> 
> Consider this output from python:
> 
> >>> "64".islower()
> False
> >>> "x64".islower()
> True
> >>> "x64A".islower()
> False
> >>>
> 
> The problem is that "64".islower() returns False when we need True
> (overrides get split by '_' so x86_64 turns into "x86" and "64" in the
> code in question).
> 
> Cheers,
> 
> Richard

Ok, that makes more sense, except for one thing: where/when is _x86_64 
used as override(s)? The only related overrides I can find in OE-Core 
all use _x86-64, which matches ${TRANSLATED_TARGET_ARCH} for, e.g., 
qemux86-64.

//Peter

Re: [PATCH 2/3] data_smart: Allow numeric characters in overrides

[richard.purdie]

On Wed, 2018-12-19 at 16:12 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: richard.purdie@linuxfoundation.org
> > <richard.purdie@linuxfoundation.org>
> > Sent: den 19 december 2018 12:42
> > To: Peter Kjellerstedt <peter.kjellerstedt@axis.com>; bitbake-
> > devel@lists.openembedded.org
> > Subject: Re: [bitbake-devel] [PATCH 2/3] data_smart: Allow numeric
> > characters in overrides
> > 
> > On Wed, 2018-12-19 at 10:55 +0000, Peter Kjellerstedt wrote:
> > > I don't understand the commit message either. It says that there
> > > are
> > > problems with x86-64 because it contains digits, but x86 (which
> > > the
> > > commit message says works) contains digits too, so that cannot
> > > really
> > > be the case. If it is actually the dash in x86-64 that is
> > > supposedly
> > > a cause of problems, then islower() ignores both digits and non-
> > > alpha
> > > characters like the dash so that cannot really be a reason to
> > > change
> > > the code either. All in all I'm confused by this commit and what
> > > it
> > > is trying to solve...
> > 
> > Consider this output from python:
> > 
> > > > > "64".islower()
> > False
> > > > > "x64".islower()
> > True
> > > > > "x64A".islower()
> > False
> > 
> > The problem is that "64".islower() returns False when we need True
> > (overrides get split by '_' so x86_64 turns into "x86" and "64" in
> > the
> > code in question).
> > 
> > Cheers,
> > 
> > Richard
> 
> Ok, that makes more sense, except for one thing: where/when is _x86_64 
> used as override(s)? The only related overrides I can find in OE-Core 
> all use _x86-64, which matches ${TRANSLATED_TARGET_ARCH} for, e.g., 
> qemux86-64.

Its not obvious as its indirect:

conf/distro/include/maintainers.inc:RECIPE_MAINTAINER_pn-gcc-cross-${TARGET_ARCH} = "Khem Raj <raj.khem@gmail.com>"

That worked for MACHINE=qemux86 but not for MACHINE=qemux86-64.

Arguably you could rename that to use TRANSLATED_TARGET_ARCH but I seem
to remember there were others which would have been harder to rename
out of.

We should add some testcases to bitbake-selftest for this.

Cheers,

Richard