[PATCH 1/2] bitbake.conf: Add layer-<layername> override support

[PATCH 1/2] bitbake.conf: Add layer-<layername> override support

[Richard Purdie]

Add a new layer specific override based upon the FILE_LAYERNAME support now present
in bitbake. In particular this allows layer specific QA warnings and errors to
be made more easily.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/conf/bitbake.conf | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 702881144ef..9625a6fef4c 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -806,7 +806,8 @@ DISTRO_NAME ??= "OpenEmbedded"
 # And finally '<foo>:forcevariable' overrides any standard variable, with the highest priority.
 # This works for functions as well, they are really just variables.
 #
-OVERRIDES = "${TARGET_OS}:${TRANSLATED_TARGET_ARCH}:pn-${PN}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${CLASSOVERRIDE}${LIBCOVERRIDE}:forcevariable"
+OVERRIDES = "${TARGET_OS}:${TRANSLATED_TARGET_ARCH}:pn-${PN}:layer-${FILE_LAYERNAME}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${CLASSOVERRIDE}${LIBCOVERRIDE}:forcevariable"
+FILE_LAYERNAME ??= "config"
 LIBCOVERRIDE ?= ""
 CLASSOVERRIDE ?= "class-target"
 DISTROOVERRIDES ?= "${@d.getVar('DISTRO') or ''}"
-- 
2.39.2

[PATCH 2/2] insane: Improve patch-status layer filtering

[Richard Purdie]

Now that we have layer overrides, we can easily enable patch-status in
ERROR_QA without the hardcoded code making it easier for other layers
to opt into the checks.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 meta/classes-global/insane.bbclass | 16 +++-------------
 1 file changed, 3 insertions(+), 13 deletions(-)

diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass
index f391fa80538..114781c7803 100644
--- a/meta/classes-global/insane.bbclass
+++ b/meta/classes-global/insane.bbclass
@@ -45,10 +45,11 @@ ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
             already-stripped installed-vs-shipped ldflags compile-host-path \
             install-host-path pn-overrides unknown-configure-option \
             useless-rpaths rpaths staticdev empty-dirs \
-            patch-fuzz patch-status-core\
+            patch-fuzz \
             "
 # Add usrmerge QA check based on distro feature
 ERROR_QA:append = "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)}"
+ERROR_QA:append:layer-core = " patch-status"
 
 FAKEROOT_QA = "host-user-contaminated"
 FAKEROOT_QA[doc] = "QA tests which need to run under fakeroot. If any \
@@ -1340,24 +1341,13 @@ python do_qa_patch() {
     import re
     from oe import patch
 
-    allpatches = False
-    if bb.utils.filter('ERROR_QA', 'patch-status-noncore', d) or bb.utils.filter('WARN_QA', 'patch-status-noncore', d):
-        allpatches = True
-
     coremeta_path = os.path.join(d.getVar('COREBASE'), 'meta', '')
     for url in patch.src_patches(d):
         (_, _, fullpath, _, _, _) = bb.fetch.decodeurl(url)
 
-        # skip patches not in oe-core
-        patchtype = "patch-status-core"
-        if not os.path.abspath(fullpath).startswith(coremeta_path):
-            patchtype = "patch-status-noncore"
-            if not allpatches:
-                continue
-
         msg = oe.qa.check_upstream_status(fullpath)
         if msg:
-            oe.qa.handle_error(patchtype, msg, d)
+            oe.qa.handle_error("patch-status", msg, d)
 
     oe.qa.exit_if_errors(d)
 }
-- 
2.39.2

Re: [OE-core] [PATCH 2/2] insane: Improve patch-status layer filtering

[Ross Burton]

😀

> On 19 Jun 2023, at 15:06, Richard Purdie via lists.openembedded.org <richard.purdie=linuxfoundation.org@lists.openembedded.org> wrote:
> 
> Now that we have layer overrides, we can easily enable patch-status in
> ERROR_QA without the hardcoded code making it easier for other layers
> to opt into the checks.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> meta/classes-global/insane.bbclass | 16 +++-------------
> 1 file changed, 3 insertions(+), 13 deletions(-)
> 
> diff --git a/meta/classes-global/insane.bbclass b/meta/classes-global/insane.bbclass
> index f391fa80538..114781c7803 100644
> --- a/meta/classes-global/insane.bbclass
> +++ b/meta/classes-global/insane.bbclass
> @@ -45,10 +45,11 @@ ERROR_QA ?= "dev-so debug-deps dev-deps debug-files arch pkgconfig la \
>             already-stripped installed-vs-shipped ldflags compile-host-path \
>             install-host-path pn-overrides unknown-configure-option \
>             useless-rpaths rpaths staticdev empty-dirs \
> -            patch-fuzz patch-status-core\
> +            patch-fuzz \
>             "
> # Add usrmerge QA check based on distro feature
> ERROR_QA:append = "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', ' usrmerge', '', d)}"
> +ERROR_QA:append:layer-core = " patch-status"
> 
> FAKEROOT_QA = "host-user-contaminated"
> FAKEROOT_QA[doc] = "QA tests which need to run under fakeroot. If any \
> @@ -1340,24 +1341,13 @@ python do_qa_patch() {
>     import re
>     from oe import patch
> 
> -    allpatches = False
> -    if bb.utils.filter('ERROR_QA', 'patch-status-noncore', d) or bb.utils.filter('WARN_QA', 'patch-status-noncore', d):
> -        allpatches = True
> -
>     coremeta_path = os.path.join(d.getVar('COREBASE'), 'meta', '')
>     for url in patch.src_patches(d):
>         (_, _, fullpath, _, _, _) = bb.fetch.decodeurl(url)
> 
> -        # skip patches not in oe-core
> -        patchtype = "patch-status-core"
> -        if not os.path.abspath(fullpath).startswith(coremeta_path):
> -            patchtype = "patch-status-noncore"
> -            if not allpatches:
> -                continue
> -
>         msg = oe.qa.check_upstream_status(fullpath)
>         if msg:
> -            oe.qa.handle_error(patchtype, msg, d)
> +            oe.qa.handle_error("patch-status", msg, d)
> 
>     oe.qa.exit_if_errors(d)
> }
> -- 
> 2.39.2
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#183115): https://lists.openembedded.org/g/openembedded-core/message/183115
> Mute This Topic: https://lists.openembedded.org/mt/99623855/6875888
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ross.burton@arm.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [OE-core] [PATCH 2/2] insane: Improve patch-status layer filtering

[Luca Ceresoli]

Hi Richard,

On Mon, 19 Jun 2023 15:06:02 +0100
"Richard Purdie" <richard.purdie@linuxfoundation.org> wrote:

> Now that we have layer overrides, we can easily enable patch-status in
> ERROR_QA without the hardcoded code making it easier for other layers
> to opt into the checks.
> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>

I love the cleanup this patch introduces, however I'm afraid it seems
to be causing some failures on selftest:

AssertionError: Command 'bitbake  dos2unix -c patch' returned non-zero exit status 1:
...
ERROR: dos2unix-7.5.0-r0 do_patch: QA Issue: Missing Upstream-Status in patch

https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5367/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5402/steps/15/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5325/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5375/steps/15/logs/stdio

Luca

PS: I noticed you seems to already have a fix under test on
origin/master-next!

-- 
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [OE-core] [PATCH 2/2] insane: Improve patch-status layer filtering

[Richard Purdie]

On Tue, 2023-06-20 at 22:16 +0200, Luca Ceresoli wrote:
> Hi Richard,
> 
> On Mon, 19 Jun 2023 15:06:02 +0100
> "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote:
> 
> > Now that we have layer overrides, we can easily enable patch-status in
> > ERROR_QA without the hardcoded code making it easier for other layers
> > to opt into the checks.
> > 
> > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> 
> I love the cleanup this patch introduces, however I'm afraid it seems
> to be causing some failures on selftest:
> 
> AssertionError: Command 'bitbake  dos2unix -c patch' returned non-zero exit status 1:
> ...
> ERROR: dos2unix-7.5.0-r0 do_patch: QA Issue: Missing Upstream-Status in patch
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5367/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5402/steps/15/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5325/steps/14/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5375/steps/15/logs/stdio
> 
> Luca
> 
> PS: I noticed you seems to already have a fix under test on
> origin/master-next!

Thanks! As you noticed, I did have a fix I was testing and I did run a
test build of that just of oe-selftest. I've squashed a couple of
selftest fixes into the commit and merged it.

Cheers,

Richard

Re: [OE-core] [PATCH 2/2] insane: Improve patch-status layer filtering

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3628 bytes --]

There is maybe unexpected change in behavior caused by this.

Before it was triggering error only for .patch files in oe-core, while now
it seems to apply for all recipes from oe-core layer (independently from
where the .patch file was added), so e.g. meta-raspberrypi is now causing
bluez5 do_patch to fail:

bluez5-5.66: Missing Upstream-Status in patch
meta-raspberrypi/recipes-connectivity/bluez5/bluez5/0001-bcm43xx-Add-bcm43xx-3wire-variant.patch
Please add according to
https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines#Patch_Header_Recommendations:_Upstream-Status
. [patch-status]
bluez5-5.66: Missing Upstream-Status in patch
meta-raspberrypi/recipes-connectivity/bluez5/bluez5/0002-bcm43xx-The-UART-speed-must-be-reset-after-the-firmw.patch
Please add according to
https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines#Patch_Header_Recommendations:_Upstream-Status
. [patch-status]
bluez5-5.66: Missing Upstream-Status in patch
meta-raspberrypi/recipes-connectivity/bluez5/bluez5/0003-Increase-firmware-load-timeout-to-30s.patch
Please add according to
https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines#Patch_Header_Recommendations:_Upstream-Status
. [patch-status]
bluez5-5.66: Missing Upstream-Status in patch
meta-raspberrypi/recipes-connectivity/bluez5/bluez5/0004-Move-the-hciattach-firmware-into-lib-firmware.patch
Please add according to
https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines#Patch_Header_Recommendations:_Upstream-Status
. [patch-status]

On Wed, Jun 21, 2023 at 12:27 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Tue, 2023-06-20 at 22:16 +0200, Luca Ceresoli wrote:
> > Hi Richard,
> >
> > On Mon, 19 Jun 2023 15:06:02 +0100
> > "Richard Purdie" <richard.purdie@linuxfoundation.org> wrote:
> >
> > > Now that we have layer overrides, we can easily enable patch-status in
> > > ERROR_QA without the hardcoded code making it easier for other layers
> > > to opt into the checks.
> > >
> > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> >
> > I love the cleanup this patch introduces, however I'm afraid it seems
> > to be causing some failures on selftest:
> >
> > AssertionError: Command 'bitbake  dos2unix -c patch' returned non-zero
> exit status 1:
> > ...
> > ERROR: dos2unix-7.5.0-r0 do_patch: QA Issue: Missing Upstream-Status in
> patch
> >
> >
> https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/5367/steps/14/logs/stdio
> >
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/5402/steps/15/logs/stdio
> >
> https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5325/steps/14/logs/stdio
> >
> https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/5375/steps/15/logs/stdio
> >
> > Luca
> >
> > PS: I noticed you seems to already have a fix under test on
> > origin/master-next!
>
> Thanks! As you noticed, I did have a fix I was testing and I did run a
> test build of that just of oe-selftest. I've squashed a couple of
> selftest fixes into the commit and merged it.
>
> Cheers,
>
> Richard
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#183161):
> https://lists.openembedded.org/g/openembedded-core/message/183161
> Mute This Topic: https://lists.openembedded.org/mt/99623855/3617156
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> Martin.Jansa@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

[-- Attachment #2: Type: text/html, Size: 5845 bytes --]

Re: [OE-core] [PATCH 2/2] insane: Improve patch-status layer filtering

[Richard Purdie]

On Wed, 2023-06-21 at 10:58 +0200, Martin Jansa wrote:
> There is maybe unexpected change in behavior caused by this.
> 
> Before it was triggering error only for .patch files in oe-core,
> while now it seems to apply for all recipes from oe-core layer
> (independently from where the .patch file was added), so e.g. meta-
> raspberrypi is now causing bluez5 do_patch to fail:
> 
> bluez5-5.66: Missing Upstream-Status in patch
> meta-raspberrypi/recipes-connectivity/bluez5/bluez5/0001-bcm43xx-Add-bcm43xx-3wire-variant.patch
> Please add according to https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines#Patch_Header_Recommendations:_Upstream-Status . [patch-status]
> bluez5-5.66: Missing Upstream-Status in patch
> meta-raspberrypi/recipes-connectivity/bluez5/bluez5/0002-bcm43xx-The-UART-speed-must-be-reset-after-the-firmw.patch
> Please add according to https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines#Patch_Header_Recommendations:_Upstream-Status . [patch-status]
> bluez5-5.66: Missing Upstream-Status in patch
> meta-raspberrypi/recipes-connectivity/bluez5/bluez5/0003-Increase-firmware-load-timeout-to-30s.patch
> Please add according to https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines#Patch_Header_Recommendations:_Upstream-Status . [patch-status]
> bluez5-5.66: Missing Upstream-Status in patch
> meta-raspberrypi/recipes-connectivity/bluez5/bluez5/0004-Move-the-hciattach-firmware-into-lib-firmware.patch
> Please add according to https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines#Patch_Header_Recommendations:_Upstream-Status . [patch-status]

I did mention this in the docs update patch to the migration guide I
sent yesterday. It wasn't entirely expected but is a side effect from
the way the layer override works as it is based on the recipe itself,
not where the patches may reside. We noticed it on the autobuilder for
meta-agl for example and it also broke from of the selftests.

Overall, I don't think this should be a huge issue once we update the
hopefully small number of patches to core recipes in layers and the
layer mechanism is an overall win in the end?

You could work around this with things like:

ERROR_QA:remove:pn-bluez = "patch-status"

to, not that I'd recommend it.

Cheers,

Richard