[OE-core][PATCH 0/2] Fix: allow specification of dtb directory

[OE-core][PATCH 0/2] Fix: allow specification of dtb directory

[rs]

From: Randolph Sapp <rs@ti.com>

Fix a few of the oddities introduced by the previous patch that allowed
specification of the DTB install directory and specification of vendor
directories in that directory.

Add a recursive search for dtb and dtbo files to make sure
upstream dtb directory standards don't break our packages.

Makes the kernel-devicetree class posix compliant again by using BB
variables more efficiently.

Martin Jansa (1):
  kernel-devicetree: make shell scripts posix compliant

Randolph Sapp (1):
  kernel-devicetree: recursively search for dtbs

 meta/classes-recipe/kernel-devicetree.bbclass | 26 +++++++++++++------
 meta/classes-recipe/kernel.bbclass            |  2 +-
 2 files changed, 19 insertions(+), 9 deletions(-)

-- 
2.40.1

[OE-core][PATCH 1/2] kernel-devicetree: make shell scripts posix compliant

[rs]

From: Martin Jansa <Martin.Jansa@gmail.com>

Use the notation suggested by Martin Jansa to avoid a bashism. Also
switch KERNEL_DTBVENDORED to the more common 0/1 notation instead of
true/false.

Fixes: 04ab57d200 (kernel-devicetree: allow specification of dtb
directory, 2023-05-02)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Randolph Sapp <rs@ti.com>
---
 meta/classes-recipe/kernel-devicetree.bbclass | 4 ++--
 meta/classes-recipe/kernel.bbclass            | 2 +-
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/meta/classes-recipe/kernel-devicetree.bbclass b/meta/classes-recipe/kernel-devicetree.bbclass
index a6c6c5f227..831fdd1527 100644
--- a/meta/classes-recipe/kernel-devicetree.bbclass
+++ b/meta/classes-recipe/kernel-devicetree.bbclass
@@ -82,7 +82,7 @@ do_install:append() {
 	for dtbf in ${KERNEL_DEVICETREE}; do
 		dtb=`normalize_dtb "$dtbf"`
 		dtb_path=`get_real_dtb_path_in_kernel "$dtb"`
-		if [ ${KERNEL_DTBVENDORED} == "false" ]; then
+		if "${@'false' if oe.types.boolean(d.getVar('KERNEL_DTBVENDORED')) else 'true'}"; then
 			dtb_ext=${dtb##*.}
 			dtb_base_name=`basename $dtb .$dtb_ext`
 			dtb=$dtb_base_name.$dtb_ext
@@ -97,7 +97,7 @@ do_deploy:append() {
 		dtb_ext=${dtb##*.}
 		dtb_base_name=`basename $dtb .$dtb_ext`
 		install -d $deployDir
-		if [ ${KERNEL_DTBVENDORED} == "false" ]; then
+		if "${@'false' if oe.types.boolean(d.getVar('KERNEL_DTBVENDORED')) else 'true'}"; then
 			dtb=$dtb_base_name.$dtb_ext
 		fi
 		install -m 0644 ${D}/${KERNEL_DTBDEST}/$dtb $deployDir/$dtb_base_name-${KERNEL_DTB_NAME}.$dtb_ext
diff --git a/meta/classes-recipe/kernel.bbclass b/meta/classes-recipe/kernel.bbclass
index 8f022b234d..4bce64cf6a 100644
--- a/meta/classes-recipe/kernel.bbclass
+++ b/meta/classes-recipe/kernel.bbclass
@@ -216,7 +216,7 @@ KERNEL_RELEASE ?= "${KERNEL_VERSION}"
 KERNEL_OUTPUT_DIR ?= "arch/${ARCH}/boot"
 KERNEL_IMAGEDEST ?= "boot"
 KERNEL_DTBDEST ?= "${KERNEL_IMAGEDEST}"
-KERNEL_DTBVENDORED ?= "false"
+KERNEL_DTBVENDORED ?= "0"
 
 #
 # configuration
-- 
2.40.1

[OE-core][PATCH 2/2] kernel-devicetree: recursively search for dtbs

[rs]

From: Randolph Sapp <rs@ti.com>

Upstream's dtb directory structure has no real standard. They just tend
to idle around the 2/3 directory depth. Recursively search for the
dtb/dtbo files instead of assuming anything.

Fixes: 04ab57d200 (kernel-devicetree: allow specification of dtb
directory, 2023-05-02)

Signed-off-by: Randolph Sapp <rs@ti.com>
---
 meta/classes-recipe/kernel-devicetree.bbclass | 22 ++++++++++++++-----
 1 file changed, 16 insertions(+), 6 deletions(-)

diff --git a/meta/classes-recipe/kernel-devicetree.bbclass b/meta/classes-recipe/kernel-devicetree.bbclass
index 831fdd1527..1beaedfb44 100644
--- a/meta/classes-recipe/kernel-devicetree.bbclass
+++ b/meta/classes-recipe/kernel-devicetree.bbclass
@@ -12,12 +12,22 @@ python () {
             d.appendVar("PACKAGES", " ${KERNEL_PACKAGE_NAME}-image-zimage-bundle")
 }
 
-FILES:${KERNEL_PACKAGE_NAME}-devicetree = " \
-    /${KERNEL_DTBDEST}/*.dtb \
-    /${KERNEL_DTBDEST}/*.dtbo \
-    /${KERNEL_DTBDEST}/*/*.dtb \
-    /${KERNEL_DTBDEST}/*/*.dtbo \
-"
+# recursivly search for devicetree files
+python () {
+    import glob
+    import os
+
+    file_paths = []
+    dest_dir = d.getVar('D')
+    full_dtb_dir = dest_dir + '/' + d.getVar('KERNEL_DTBDEST')
+
+    for file in glob.glob(full_dtb_dir + '/**/*.dtb*', recursive=True):
+        file_paths.append('/' + os.path.relpath(file, dest_dir))
+
+    d.appendVar("FILES:" + d.getVar("KERNEL_PACKAGE_NAME") + '-devicetree',
+                ' '+' '.join(file_paths))
+}
+
 FILES:${KERNEL_PACKAGE_NAME}-image-zimage-bundle = "/${KERNEL_IMAGEDEST}/zImage-*.dtb.bin"
 
 # Generate kernel+devicetree bundle
-- 
2.40.1

Re: [OE-core][PATCH 0/2] Fix: allow specification of dtb directory

[Alexandre Belloni]

Hello,

This caused the following failures on the AB:

https://autobuilder.yoctoproject.org/typhoon/#/builders/113/builds/3986/steps/13/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5176/steps/15/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5167/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/7160/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/106/builds/6067/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/1406/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5126/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5205/steps/14/logs/stdio


2023-05-12 09:01:46,475 - oe-selftest - INFO - fitimage.FitImageTests.test_initramfs_bundle (subunit.RemotedTestCase)
2023-05-12 09:01:46,476 - oe-selftest - INFO -  ... FAIL

and

ERROR: When reparsing /home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/recipes-kernel/linux/linux-yocto_6.1.bb:do_package, the basehash value changed from 856d8373327180fa74e88cff2bbc28b82eb24fb635dc7d3ac16d7a2b7d89ff2e to d2b77c38850e4954edc1422022fb0dbd307742cf2eafac6d8ae699d03d2ce13b. The metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake linux-yocto -cdo_package -Snone
ERROR: Then:
ERROR: $ bitbake linux-yocto -cdo_package -Sprintdiff

On 11/05/2023 16:36:12-0500, Randolph Sapp via lists.openembedded.org wrote:
> From: Randolph Sapp <rs@ti.com>
> 
> Fix a few of the oddities introduced by the previous patch that allowed
> specification of the DTB install directory and specification of vendor
> directories in that directory.
> 
> Add a recursive search for dtb and dtbo files to make sure
> upstream dtb directory standards don't break our packages.
> 
> Makes the kernel-devicetree class posix compliant again by using BB
> variables more efficiently.
> 
> Martin Jansa (1):
>   kernel-devicetree: make shell scripts posix compliant
> 
> Randolph Sapp (1):
>   kernel-devicetree: recursively search for dtbs
> 
>  meta/classes-recipe/kernel-devicetree.bbclass | 26 +++++++++++++------
>  meta/classes-recipe/kernel.bbclass            |  2 +-
>  2 files changed, 19 insertions(+), 9 deletions(-)
> 
> -- 
> 2.40.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#181157): https://lists.openembedded.org/g/openembedded-core/message/181157
> Mute This Topic: https://lists.openembedded.org/mt/98837534/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [EXTERNAL] Re: [OE-core][PATCH 0/2] Fix: allow specification of dtb directory

[Randolph Sapp]

On 5/12/23 10:06, Alexandre Belloni wrote:
> Hello,
> 
> This caused the following failures on the AB:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/113/builds/3986/steps/13/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5176/steps/15/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5167/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/65/builds/7160/steps/11/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/106/builds/6067/steps/12/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/1406/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5126/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5205/steps/14/logs/stdio
> 
> 
> 2023-05-12 09:01:46,475 - oe-selftest - INFO - fitimage.FitImageTests.test_initramfs_bundle (subunit.RemotedTestCase)
> 2023-05-12 09:01:46,476 - oe-selftest - INFO -  ... FAIL
> 
> and
> 
> ERROR: When reparsing /home/pokybuild/yocto-worker/oe-selftest-centos/build/meta/recipes-kernel/linux/linux-yocto_6.1.bb:do_package, the basehash value changed from 856d8373327180fa74e88cff2bbc28b82eb24fb635dc7d3ac16d7a2b7d89ff2e to d2b77c38850e4954edc1422022fb0dbd307742cf2eafac6d8ae699d03d2ce13b. The metadata is not deterministic and this needs to be fixed.
> ERROR: The following commands may help:
> ERROR: $ bitbake linux-yocto -cdo_package -Snone
> ERROR: Then:
> ERROR: $ bitbake linux-yocto -cdo_package -Sprintdiff
> 

Right you are. Expect a v2 using a specialized function to avoid the 
metadata change in a minute.

> On 11/05/2023 16:36:12-0500, Randolph Sapp via lists.openembedded.org wrote:
>> From: Randolph Sapp <rs@ti.com>
>>
>> Fix a few of the oddities introduced by the previous patch that allowed
>> specification of the DTB install directory and specification of vendor
>> directories in that directory.
>>
>> Add a recursive search for dtb and dtbo files to make sure
>> upstream dtb directory standards don't break our packages.
>>
>> Makes the kernel-devicetree class posix compliant again by using BB
>> variables more efficiently.
>>
>> Martin Jansa (1):
>>    kernel-devicetree: make shell scripts posix compliant
>>
>> Randolph Sapp (1):
>>    kernel-devicetree: recursively search for dtbs
>>
>>   meta/classes-recipe/kernel-devicetree.bbclass | 26 +++++++++++++------
>>   meta/classes-recipe/kernel.bbclass            |  2 +-
>>   2 files changed, 19 insertions(+), 9 deletions(-)
>>
>> -- 
>> 2.40.1
>>
> 
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#181157): https://lists.openembedded.org/g/openembedded-core/message/181157
>> Mute This Topic: https://lists.openembedded.org/mt/98837534/3617179
>> Group Owner: openembedded-core+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>
> 
>