[PATCH 0/1] data_smart: fix key expansion in conditional overrides

[PATCH 0/1] data_smart: fix key expansion in conditional overrides

[Nguyen Minh Tien]

An :append whose override name still contains a variable is silently
dropped. From the report:

    RDEPENDS:${PN}:append:pn-packagegroup-cross-canadian-${MACHINE} = " ..."

Still reproduces on master (c251833d2). The mechanism is in the patch;
briefly, renameVar() rebuilds dependent override keys with a string
replace, so renaming RDEPENDS:${PN} leaves the ${MACHINE} behind and the
append lands on a key no override can ever match.

The part worth flagging is the '${' not in newkey guard. Expanding
unconditionally looks like the obvious fix but regresses parsing:
native.bbclass renames to a key that is itself still unexpanded, before
expandKeys() runs, and expanding the dependent keys there turns the later
rename into a no-op. oe-core then parses with three warnings master does
not have, in dbus and python3-psutil. The second of the two added tests
covers this and fails without the guard.

Tested against bitbake c251833d2 and oe-core 9d89b3b802:

 - bitbake-selftest: 790 tests, no failures. Six bb.tests.fetch errors
   remain, host issues that occur identically on master
 - all 952 oe-core recipes parse with no warnings, same as master
 - bitbake -e for dbus-native and python3-psutil-native, the two recipes
   that hit this path, identical to master
 - same parse with multilib enabled, exercising multilib_global.bbclass
 - core-image-minimal builds for qemux86-64, 5275 tasks

Nguyen Minh Tien (1):
  data_smart: fix operations lost when an override name contains a
    variable

 lib/bb/data_smart.py | 11 +++++++++--
 lib/bb/tests/data.py | 25 +++++++++++++++++++++++++
 2 files changed, 34 insertions(+), 2 deletions(-)

-- 
2.34.1

[PATCH 1/1] data_smart: fix operations lost when an override name contains a variable

[Nguyen Minh Tien]

An operation whose override name needs key expansion is silently dropped:

    RDEPENDS:${PN}:append:pn-foo-${MACHINE} = " bar"

renameVar() rebuilds the dependent override keys with a plain string
replace, so renaming RDEPENDS:${PN} leaves the append attached to
RDEPENDS:foo:append:pn-foo-${MACHINE}, which never matches an active
override. expandKeys() cannot fix that up afterwards either, as it works
from a list of keys collected before any renaming happened.

Expand the derived name before renaming it. Only do so once newkey is
itself expanded, otherwise expandKeys() has still to rename newkey and
handles the dependent keys along with it.

Add regression tests for both cases.

Fixes [YOCTO #14867]

Signed-off-by: Nguyen Minh Tien <zizuzacker@gmail.com>
---
 lib/bb/data_smart.py | 11 +++++++++--
 lib/bb/tests/data.py | 25 +++++++++++++++++++++++++
 2 files changed, 34 insertions(+), 2 deletions(-)

diff --git a/lib/bb/data_smart.py b/lib/bb/data_smart.py
index 9961269a3..5738aff95 100644
--- a/lib/bb/data_smart.py
+++ b/lib/bb/data_smart.py
@@ -697,8 +697,15 @@ class DataSmart(MutableMapping):
             found = True
             self.overridedata[newkey] = []
             for (v, o) in self.overridedata[key]:
-                self.overridedata[newkey].append([v.replace(key, newkey), o])
-                self.renameVar(v, v.replace(key, newkey))
+                newv = v.replace(key, newkey)
+                # The derived name may still hold a variable reference which
+                # expandKeys() will never revisit, so expand it here. Only once
+                # newkey is expanded though, otherwise expandKeys() has still to
+                # rename newkey and handles the dependent keys along with it.
+                if '${' in newv and '${' not in newkey:
+                    newv = self.expand(newv)
+                self.overridedata[newkey].append([newv, o])
+                self.renameVar(v, newv)
 
         if not found:
             # No variable to rename so not worth the work in writing extra
diff --git a/lib/bb/tests/data.py b/lib/bb/tests/data.py
index fd690a9e2..5ef159eae 100644
--- a/lib/bb/tests/data.py
+++ b/lib/bb/tests/data.py
@@ -406,6 +406,31 @@ class TestOverrides(unittest.TestCase):
         bb.data.expandKeys(self.d)
         self.assertEqual(self.d.getVar("VERSION"), "2")
 
+    # Test an :append whose override name is only resolved by key expansion
+    def test_append_in_expanded_override(self):
+        self.d.setVar("MACHINE", "qemux86")
+        self.d.setVar("PN", "gizmo")
+        self.d.setVar("OVERRIDES", "gizmo:pn-gizmo-qemux86")
+        self.d.setVar("TEST:${PN}", "base")
+        self.d.setVar("TEST:${PN}:append:pn-gizmo-${MACHINE}", " appended")
+        bb.data.expandKeys(self.d)
+        self.assertEqual(self.d.getVar("TEST"), "base appended")
+
+    # Test renaming to a key which is itself not expanded yet, as native.bbclass
+    # does. The dependent override keys must be left for expandKeys() to rename.
+    def test_rename_to_unexpanded_key_with_override(self):
+        self.d.setVar("BPN", "gizmo")
+        self.d.setVar("PN", "gizmo-native")
+        self.d.setVar("OVERRIDES", "class-target")
+        self.d.setVar("TEST:${PN}-lib", "base")
+        self.d.setVar("TEST:${PN}-lib:class-target", "target")
+        with LogRecord() as logs:
+            self.d.renameVar("TEST:${PN}-lib", "TEST:${BPN}-lib-native")
+            bb.data.expandKeys(self.d)
+            self.assertFalse(logContains("renameVar with equivalent keys", logs))
+            self.assertFalse(logContains("replaces original key", logs))
+        self.assertEqual(self.d.getVar("TEST:gizmo-lib-native"), "target")
+
     def test_remove_with_override(self):
         self.d.setVar("TEST:bar", "testvalue2")
         self.d.setVar("TEST:some_val", "testvalue3 testvalue5")
-- 
2.34.1