[PATCH 0/2] Only try to expand refs to valid variable names

[PATCH 0/2] Only try to expand refs to valid variable names

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This aligns the behavior of expansion with the recipe parser, only
attempting to expand references to valid variable names. This avoids
adding references for things like `${foo#${TOPDIR}}` to our vardeps
without imposing much additional processing overhead beyond the change
to the expansion regexp.

YOCTO #12987

The following changes since commit 610dbee5634677f5055e2b36a3043cd197fb8c51:

  gitsm.py: Refactor the functions and simplify the class (2019-01-15 22:22:29 +0000)

are available in the Git repository at:

  git@github.com:kergoth/bitbake yocto-12987-new

for you to fetch changes up to 9cb64e9781c98f1aaa42571af54655eaf6816583:

  bb.tests.codeparser: add parameter expansion modifiers test (2019-01-16 19:46:10 +0000)

----------------------------------------------------------------
Christopher Larson (2):
      bb.data_smart: only try to expand refs to valid variable names
      bb.tests.codeparser: add parameter expansion modifiers test

 lib/bb/data_smart.py       | 2 +-
 lib/bb/tests/codeparser.py | 7 +++++++
 2 files changed, 8 insertions(+), 1 deletion(-)

-- 
2.17.1

[PATCH 1/2] bb.data_smart: only try to expand refs to valid variable names

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

This aligns the behavior of expansion with the recipe parser, only
attempting to expand references to valid variable names. This avoids
adding references for things like `${foo#${TOPDIR}}` to our vardeps
without imposing much additional processing overhead beyond the change
to the expansion regexp.

YOCTO #12987

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 lib/bb/data_smart.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 1f45cd97..07db7be9 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -40,7 +40,7 @@ logger = logging.getLogger("BitBake.Data")
 
 __setvar_keyword__ = ["_append", "_prepend", "_remove"]
 __setvar_regexp__ = re.compile(r'(?P<base>.*?)(?P<keyword>_append|_prepend|_remove)(_(?P<add>[^A-Z]*))?$')
-__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
+__expand_var_regexp__ = re.compile(r"\${[a-zA-Z0-9\-_+./~]+?}")
 __expand_python_regexp__ = re.compile(r"\${@.+?}")
 __whitespace_split__ = re.compile(r'(\s)')
 __override_regexp__ = re.compile(r'[a-z0-9]+')
-- 
2.17.1

[PATCH 2/2] bb.tests.codeparser: add parameter expansion modifiers test

[Christopher Larson]

From: Christopher Larson <chris_larson@mentor.com>

We don't want references including shell parameter expansion modifiers
(i.e. `:-`, `#`, `%%`, etc) to be added to our vardeps, so add a test to
ensure this.

YOCTO #12987

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
---
 lib/bb/tests/codeparser.py | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lib/bb/tests/codeparser.py b/lib/bb/tests/codeparser.py
index e30e78c1..3fd76a8f 100644
--- a/lib/bb/tests/codeparser.py
+++ b/lib/bb/tests/codeparser.py
@@ -123,6 +123,13 @@ ${D}${libdir}/pkgconfig/*.pc
         self.parseExpression("sed -i -e 's:IP{:I${:g' $pc")
         self.assertExecs(set(["sed"]))
 
+    def test_parameter_expansion_modifiers(self):
+        # - and + are also valid modifiers for parameter expansion, but are
+        # valid characters in bitbake variable names, so are not included here
+        for i in ('=', ':-', ':=', '?', ':?', ':+', '#', '%', '##', '%%'):
+            name = "foo%sbar" % i
+            self.parseExpression("${%s}" % name)
+            self.assertNotIn(name, self.references)
 
     def test_until(self):
         self.parseExpression("until false; do echo true; done")
-- 
2.17.1

Re: [PATCH 0/2] Only try to expand refs to valid variable names

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 771 bytes --]

On Fri, Jan 18, 2019 at 9:46 AM Christopher Larson <kergoth@gmail.com>
wrote:

> From: Christopher Larson <chris_larson@mentor.com>
>
> This aligns the behavior of expansion with the recipe parser, only
> attempting to expand references to valid variable names. This avoids
> adding references for things like `${foo#${TOPDIR}}` to our vardeps
> without imposing much additional processing overhead beyond the change
> to the expansion regexp.
>
> YOCTO #12987
>

There’s no statistically significant performance impact to bitbake -p,
bitbake-selftest, or my test image builds, only typical run-to-run
variation.
-- 
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1315 bytes --]