[PATCH 1/3] BBHandler: Allow earlier exit for classes not found

[PATCH 1/3] BBHandler: Allow earlier exit for classes not found

[Richard Purdie]

Rather than relying on later code to error if the class isn't found,
exit earlier and more clearly from a code perspective.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/parse/parse_py/BBHandler.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/lib/bb/parse/parse_py/BBHandler.py b/lib/bb/parse/parse_py/BBHandler.py
index 68415735fd..532a4e08f1 100644
--- a/lib/bb/parse/parse_py/BBHandler.py
+++ b/lib/bb/parse/parse_py/BBHandler.py
@@ -56,6 +56,9 @@ def inherit(files, fn, lineno, d):
             if abs_fn:
                 file = abs_fn
 
+        if not os.path.exists(file):
+            raise ParseError("Could not inherit file %s" % (file), fn, lineno)
+
         if not file in __inherit_cache:
             logger.debug("Inheriting %s (from %s:%d)" % (file, fn, lineno))
             __inherit_cache.append( file )
-- 
2.34.1

[PATCH 2/3] BBHandler: Make inherit calls more directly

[Richard Purdie]

Rather than recursing into the conf handler code, simply call into
the parse code directly when inheriting files as we've already resolved
the paths and don't need anything the other codepath brings. This
makes the codepath clearer at the expense of some slight duplication.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/parse/parse_py/BBHandler.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/lib/bb/parse/parse_py/BBHandler.py b/lib/bb/parse/parse_py/BBHandler.py
index 532a4e08f1..1189114341 100644
--- a/lib/bb/parse/parse_py/BBHandler.py
+++ b/lib/bb/parse/parse_py/BBHandler.py
@@ -63,7 +63,10 @@ def inherit(files, fn, lineno, d):
             logger.debug("Inheriting %s (from %s:%d)" % (file, fn, lineno))
             __inherit_cache.append( file )
             d.setVar('__inherit_cache', __inherit_cache)
-            include(fn, file, lineno, d, "inherit")
+            try:
+                bb.parse.handle(file, d, True)
+            except (IOError, OSError) as exc:
+                raise ParseError("Could not inherit file %s: %s" % (fn, exc.strerror), fn, lineno)
             __inherit_cache = d.getVar('__inherit_cache', False) or []
 
 def get_statements(filename, absolute_filename, base_name):
-- 
2.34.1

[PATCH 3/3] BBHandler/cooker: Implement recipe and global classes

[Richard Purdie]

We have some confusion for users since some classes are meant to work
in the configuration space (or "globally") and some are meant to be
selected by recipes individually.

The cleanest way I could find to clarify this is to create "classes-global"
and "classes-recipe" directories which contain the approproate classes and
have bitbake switch scope between them at the appropriate point during
parsing. The existing "classes" directory is always searched as a fallback.

Once a class is moved to a specific directory, it will no longer be found
in the incorrect context. A good example from OE is that

INHERIT += "testimage"

will no longer work but

IMAGE_CLASSES += "testimage"

will, which makes the global scope cleaner by only including it where it
is useful and intended to be used (images).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
 lib/bb/cooker.py                   |  1 +
 lib/bb/cookerdata.py               |  2 ++
 lib/bb/parse/parse_py/BBHandler.py | 29 ++++++++++++++++++-----------
 3 files changed, 21 insertions(+), 11 deletions(-)

diff --git a/lib/bb/cooker.py b/lib/bb/cooker.py
index 2adf4d297d..1b6ee3032c 100644
--- a/lib/bb/cooker.py
+++ b/lib/bb/cooker.py
@@ -402,6 +402,7 @@ class BBCooker:
         for mc in self.databuilder.mcdata.values():
             mc.renameVar("__depends", "__base_depends")
             self.add_filewatch(mc.getVar("__base_depends", False), self.configwatcher)
+            mc.setVar("__bbclasstype", "recipe")
 
         self.baseconfig_valid = True
         self.parsecache_valid = False
diff --git a/lib/bb/cookerdata.py b/lib/bb/cookerdata.py
index d54ac932e5..b3a171509b 100644
--- a/lib/bb/cookerdata.py
+++ b/lib/bb/cookerdata.py
@@ -254,6 +254,7 @@ class CookerDataBuilder(object):
         filtered_keys = bb.utils.approved_variables()
         bb.data.inheritFromOS(self.basedata, self.savedenv, filtered_keys)
         self.basedata.setVar("BB_ORIGENV", self.savedenv)
+        self.basedata.setVar("__bbclasstype", "global")
 
         if worker:
             self.basedata.setVar("BB_WORKERCONTEXT", "1")
@@ -448,6 +449,7 @@ class CookerDataBuilder(object):
 
         # Handle any INHERITs and inherit the base class
         bbclasses  = ["base"] + (data.getVar('INHERIT') or "").split()
+        bb.warn(str(bbclasses))
         for bbclass in bbclasses:
             data = _inherit(bbclass, data)
 
diff --git a/lib/bb/parse/parse_py/BBHandler.py b/lib/bb/parse/parse_py/BBHandler.py
index 1189114341..18e6868387 100644
--- a/lib/bb/parse/parse_py/BBHandler.py
+++ b/lib/bb/parse/parse_py/BBHandler.py
@@ -44,17 +44,24 @@ def inherit(files, fn, lineno, d):
     __inherit_cache = d.getVar('__inherit_cache', False) or []
     files = d.expand(files).split()
     for file in files:
-        if not os.path.isabs(file) and not file.endswith(".bbclass"):
-            file = os.path.join('classes', '%s.bbclass' % file)
-
-        if not os.path.isabs(file):
-            bbpath = d.getVar("BBPATH")
-            abs_fn, attempts = bb.utils.which(bbpath, file, history=True)
-            for af in attempts:
-                if af != abs_fn:
-                    bb.parse.mark_dependency(d, af)
-            if abs_fn:
-                file = abs_fn
+        classtype = d.getVar("__bbclasstype", False)
+        origfile = file
+        for t in ["classes-" + classtype, "classes"]:
+            file = origfile
+            if not os.path.isabs(file) and not file.endswith(".bbclass"):
+                file = os.path.join(t, '%s.bbclass' % file)
+
+            if not os.path.isabs(file):
+                bbpath = d.getVar("BBPATH")
+                abs_fn, attempts = bb.utils.which(bbpath, file, history=True)
+                for af in attempts:
+                    if af != abs_fn:
+                        bb.parse.mark_dependency(d, af)
+                if abs_fn:
+                    file = abs_fn
+
+            if os.path.exists(file):
+                break
 
         if not os.path.exists(file):
             raise ParseError("Could not inherit file %s" % (file), fn, lineno)
-- 
2.34.1

Re: [bitbake-devel] [PATCH 1/3] BBHandler: Allow earlier exit for classes not found

[Khem Raj]

I seeing

% bitbake binutils
WARNING: ['base', 'sanity-meta-odroid-extras', 'clang',
'report-error', 'image-buildinfo', 'package_ipk', 'buildhistory',
'buildstats', 'buildstats-summary', 'debian', 'devshell', 'sstate',
'license', 'remove-libtool', 'sanity']
ERROR: ParseError at
/mnt/b/yoe/master/sources/openembedded-core/meta/classes/buildhistory.bbclass:10:
Could not inherit file classes/image-artifact-names.bbclass

On Wed, Aug 10, 2022 at 6:43 AM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> Rather than relying on later code to error if the class isn't found,
> exit earlier and more clearly from a code perspective.
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
>  lib/bb/parse/parse_py/BBHandler.py | 3 +++
>  1 file changed, 3 insertions(+)
>
> diff --git a/lib/bb/parse/parse_py/BBHandler.py b/lib/bb/parse/parse_py/BBHandler.py
> index 68415735fd..532a4e08f1 100644
> --- a/lib/bb/parse/parse_py/BBHandler.py
> +++ b/lib/bb/parse/parse_py/BBHandler.py
> @@ -56,6 +56,9 @@ def inherit(files, fn, lineno, d):
>              if abs_fn:
>                  file = abs_fn
>
> +        if not os.path.exists(file):
> +            raise ParseError("Could not inherit file %s" % (file), fn, lineno)
> +
>          if not file in __inherit_cache:
>              logger.debug("Inheriting %s (from %s:%d)" % (file, fn, lineno))
>              __inherit_cache.append( file )
> --
> 2.34.1
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#13880): https://lists.openembedded.org/g/bitbake-devel/message/13880
> Mute This Topic: https://lists.openembedded.org/mt/92936457/1997914
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [raj.khem@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [bitbake-devel] [PATCH 1/3] BBHandler: Allow earlier exit for classes not found

[Richard Purdie]

On Wed, 2022-08-10 at 08:55 -0700, Khem Raj wrote:
> I seeing
> 
> % bitbake binutils
> WARNING: ['base', 'sanity-meta-odroid-extras', 'clang',
> 'report-error', 'image-buildinfo', 'package_ipk', 'buildhistory',
> 'buildstats', 'buildstats-summary', 'debian', 'devshell', 'sstate',
> 'license', 'remove-libtool', 'sanity']
> ERROR: ParseError at
> /mnt/b/yoe/master/sources/openembedded-core/meta/classes/buildhistory.bbclass:10:
> Could not inherit file classes/image-artifact-names.bbclass

I think we need something like:

diff --git a/meta/classes/buildhistory.bbclass b/meta/classes/buildhistory.bbclass
index 4345ffc6939..203310ba975 100644
--- a/meta/classes/buildhistory.bbclass
+++ b/meta/classes/buildhistory.bbclass
@@ -7,7 +7,7 @@
 # Copyright (C) 2007-2011 Koen Kooi <koen@openembedded.org>
 #
 
-inherit image-artifact-names
+IMAGE_CLASSES += "image-artifact-names"
 
 BUILDHISTORY_FEATURES ?= "image package sdk"
 BUILDHISTORY_DIR ?= "${TOPDIR}/buildhistory"

Does that help?

I need to remove that warning too! Although it does help debugging right now.

Cheers,

Richard

Re: [bitbake-devel] [PATCH 1/3] BBHandler: Allow earlier exit for classes not found

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1943 bytes --]

Can we not make the inherit command adapt to current inherit context,
global or recipe?

On Wed, Aug 10, 2022 at 9:19 AM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Wed, 2022-08-10 at 08:55 -0700, Khem Raj wrote:
> > I seeing
> >
> > % bitbake binutils
> > WARNING: ['base', 'sanity-meta-odroid-extras', 'clang',
> > 'report-error', 'image-buildinfo', 'package_ipk', 'buildhistory',
> > 'buildstats', 'buildstats-summary', 'debian', 'devshell', 'sstate',
> > 'license', 'remove-libtool', 'sanity']
> > ERROR: ParseError at
> >
> /mnt/b/yoe/master/sources/openembedded-core/meta/classes/buildhistory.bbclass:10:
> > Could not inherit file classes/image-artifact-names.bbclass
>
> I think we need something like:
>
> diff --git a/meta/classes/buildhistory.bbclass
> b/meta/classes/buildhistory.bbclass
> index 4345ffc6939..203310ba975 100644
> --- a/meta/classes/buildhistory.bbclass
> +++ b/meta/classes/buildhistory.bbclass
> @@ -7,7 +7,7 @@
>  # Copyright (C) 2007-2011 Koen Kooi <koen@openembedded.org>
>  #
>
> -inherit image-artifact-names
> +IMAGE_CLASSES += "image-artifact-names"
>
>  BUILDHISTORY_FEATURES ?= "image package sdk"
>  BUILDHISTORY_DIR ?= "${TOPDIR}/buildhistory"
>
> Does that help?
>
> I need to remove that warning too! Although it does help debugging right
> now.
>
> Cheers,
>
> Richard
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#13888):
> https://lists.openembedded.org/g/bitbake-devel/message/13888
> Mute This Topic: https://lists.openembedded.org/mt/92936457/3617123
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [
> kergoth@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

-- 
Christopher Larson
chris_larson@mentor.com, chris.larson@siemens.com, kergoth@gmail.com
Principal Software Engineer, Embedded Linux Solutions, Siemens Digital
Industries Software

[-- Attachment #2: Type: text/html, Size: 3241 bytes --]

Re: [bitbake-devel] [PATCH 1/3] BBHandler: Allow earlier exit for classes not found

[Richard Purdie]

On Wed, 2022-08-10 at 09:27 -0700, Christopher Larson wrote:
> Can we not make the inherit command adapt to current inherit context,
> global or recipe?

The issue is that I'd made image-artifact-names a recipes class in OE-
Core and it was trying to be pulled into the global space via
buildhistory.

Either we accept it as a global class and change some of the users, or
we make it image specific as I'm proposing in the patch.

Cheers,

Richard

> On Wed, Aug 10, 2022 at 9:19 AM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Wed, 2022-08-10 at 08:55 -0700, Khem Raj wrote:
> > > I seeing
> > > 
> > > % bitbake binutils
> > > WARNING: ['base', 'sanity-meta-odroid-extras', 'clang',
> > > 'report-error', 'image-buildinfo', 'package_ipk', 'buildhistory',
> > > 'buildstats', 'buildstats-summary', 'debian', 'devshell',
> > > 'sstate',
> > > 'license', 'remove-libtool', 'sanity']
> > > ERROR: ParseError at
> > > /mnt/b/yoe/master/sources/openembedded-
> > > core/meta/classes/buildhistory.bbclass:10:
> > > Could not inherit file classes/image-artifact-names.bbclass
> > 
> > I think we need something like:
> > 
> > diff --git a/meta/classes/buildhistory.bbclass
> > b/meta/classes/buildhistory.bbclass
> > index 4345ffc6939..203310ba975 100644
> > --- a/meta/classes/buildhistory.bbclass
> > +++ b/meta/classes/buildhistory.bbclass
> > @@ -7,7 +7,7 @@
> >  # Copyright (C) 2007-2011 Koen Kooi <koen@openembedded.org>
> >  #
> > 
> > -inherit image-artifact-names
> > +IMAGE_CLASSES += "image-artifact-names"
> > 
> >  BUILDHISTORY_FEATURES ?= "image package sdk"
> >  BUILDHISTORY_DIR ?= "${TOPDIR}/buildhistory"
> > 
> > Does that help?
> > 
> > I need to remove that warning too! Although it does help debugging
> > right now.
> > 
> > Cheers,
> > 
> > Richard
> > 
> > -=-=-=-=-=-=-=-=-=-=-=-
> > Links: You receive all messages sent to this group.
> > View/Reply Online (#13888):
> > https://lists.openembedded.org/g/bitbake-devel/message/13888
> > Mute This Topic: https://lists.openembedded.org/mt/92936457/3617123
> > Group Owner: bitbake-devel+owner@lists.openembedded.org
> > Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub
> > [kergoth@gmail.com]
> > -=-=-=-=-=-=-=-=-=-=-=-
> > 
> 
>