[PATCH 0/2] more flexible include and require statements

[PATCH 0/2] more flexible include and require statements

[Patrick Ohly]

As discussed in the "[Openembedded-architecture] Yocto Compatible 2.0
+ signature changes" mail thread, some changes in a .bbappend cannot
be made conditional via overrides, like for example manipulating
varflags.

By allowing "require" and "include" without parameter, one can include
those changes depending on some condition with ${@ } which expands to
empty when the condition is not met.

However, this is still not particularly developer-friendly, in
particular when the change is needed under different conditions (like
one of several DISTRO_FEATURES) or more than one include file is
involved.

When allowing also to include more than one file in a single
statement, some helper function could be written which takes a
declaration of the relationship between features and include files and
then a single line covers all cases.

The example in the second commit uses this helper function which
should go into OE-core because of the DISTRO_FEATURES default value:

def optional_includes(d, mapping, key_var="DISTRO_FEATURES"):
    """
    This can be used to generate a list of files to include depending on
    the distro features that are selected. key_var contains the features
    that are set, mapping_var a space-separated set of <feature(s)>:<file(s)>
    entries. Features and files are separated by comma. Each file on the
    right-hand side is included in the result once if any of the features one
    the left-hand side is set.

    Example:
       require ${@ oe.utils.optional_includes(d, "foo,bar:foo-or-bar.inc xyz:x.inc,y.inc,z.inc")}

    For DISTRO_FEATURES = "foo xyz" that will include four .inc files in the
    order in which they are listed.
    """
    key = set((d.getVar(key_var) or "").split())
    mapping = mapping.split()
    includes = []
    for entry in mapping:
        parts = entry.split(":", 1)
        if len(parts) != 2:
            bb.fatal("%s must contain entries of the form <feature(s)>:<file(s)>, not %s" % (mapping_var, entry))
        features, files = parts
        for feature in features.split(","):
            if feature in key:
                for file in files.split(","):
                    if file not in includes:
                        includes.append(file)
    return " ".join(includes)

Patrick Ohly (2):
  ConfHandler.py: allow inherit or include without parameter
  ConfHandler.py: allow inherit or include with multiple parameters

 lib/bb/parse/parse_py/ConfHandler.py | 16 ++++++++++++----
 1 file changed, 12 insertions(+), 4 deletions(-)

base-commit: cf2bf9a0b0d9380025f61c7e7a39e6e19b46a7a1
-- 
git-series 0.9.1

[PATCH 1/2] ConfHandler.py: allow inherit or include without parameter

[Patrick Ohly]

Writing .bbappends that only have an effect when some configuration
variable like DISTRO_FEATURES is changed becomes easier when allowing
"inherit" or "require" without a parameter. The same was already
allowed for "inherit".

Then one can write in a .bbappend:

  require ${@bb.utils.contains('DISTRO_FEATURES', 'foo', 'bar.inc', '', d)}

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 lib/bb/parse/parse_py/ConfHandler.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/lib/bb/parse/parse_py/ConfHandler.py b/lib/bb/parse/parse_py/ConfHandler.py
index bf7e985..b006d06 100644
--- a/lib/bb/parse/parse_py/ConfHandler.py
+++ b/lib/bb/parse/parse_py/ConfHandler.py
@@ -81,6 +81,10 @@ def include(parentfn, fn, lineno, data, error_out):
     fn = data.expand(fn)
     parentfn = data.expand(parentfn)
 
+    if not fn:
+        # "include" or "require" without parameter is fine, just return.
+        return
+
     if not os.path.isabs(fn):
         dname = os.path.dirname(parentfn)
         bbpath = "%s:%s" % (dname, data.getVar("BBPATH"))
-- 
git-series 0.9.1

Re: [PATCH 1/2] ConfHandler.py: allow inherit or include without parameter

[Peter Kjellerstedt]

> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> devel-bounces@lists.openembedded.org] On Behalf Of Patrick Ohly
> Sent: den 7 juni 2017 15:56
> To: bitbake-devel@lists.openembedded.org
> Subject: [bitbake-devel] [PATCH 1/2] ConfHandler.py: allow inherit or
> include without parameter

Change "inherit or include" to "include or require".

> Writing .bbappends that only have an effect when some configuration
> variable like DISTRO_FEATURES is changed becomes easier when allowing
> "inherit" or "require" without a parameter. The same was already

Change "inherit" to "include".

> allowed for "inherit".
> 
> Then one can write in a .bbappend:
> 
>   require ${@bb.utils.contains('DISTRO_FEATURES', 'foo', 'bar.inc', '', d)}
> 
> Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
> ---
>  lib/bb/parse/parse_py/ConfHandler.py | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/lib/bb/parse/parse_py/ConfHandler.py b/lib/bb/parse/parse_py/ConfHandler.py
> index bf7e985..b006d06 100644
> --- a/lib/bb/parse/parse_py/ConfHandler.py
> +++ b/lib/bb/parse/parse_py/ConfHandler.py
> @@ -81,6 +81,10 @@ def include(parentfn, fn, lineno, data, error_out):
>      fn = data.expand(fn)
>      parentfn = data.expand(parentfn)
> 
> +    if not fn:
> +        # "include" or "require" without parameter is fine, just return.
> +        return
> +
>      if not os.path.isabs(fn):
>          dname = os.path.dirname(parentfn)
>          bbpath = "%s:%s" % (dname, data.getVar("BBPATH"))
> --
> git-series 0.9.1

//Peter

Re: [PATCH 1/2] ConfHandler.py: allow inherit or include without parameter

[Patrick Ohly]

On Wed, 2017-06-07 at 14:47 +0000, Peter Kjellerstedt wrote:
> > -----Original Message-----
> > From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> > devel-bounces@lists.openembedded.org] On Behalf Of Patrick Ohly
> > Sent: den 7 juni 2017 15:56
> > To: bitbake-devel@lists.openembedded.org
> > Subject: [bitbake-devel] [PATCH 1/2] ConfHandler.py: allow inherit or
> > include without parameter
> 
> Change "inherit or include" to "include or require".
> 
> > Writing .bbappends that only have an effect when some configuration
> > variable like DISTRO_FEATURES is changed becomes easier when allowing
> > "inherit" or "require" without a parameter. The same was already
> 
> Change "inherit" to "include".

You are right of course. At least I was consistent in my mistake ;-} I
guess I'm still struggling with the oddly named "require" vs. "include"
distinction.

I can send a V2 if there are no other objections, or perhaps it can be
fixed up as part of merging?

-- 
Best Regards, Patrick Ohly

The content of this message is my personal opinion only and although
I am an employee of Intel, the statements I make here in no way
represent Intel's position on the issue, nor am I authorized to speak
on behalf of Intel on this matter.

Re: [PATCH 1/2] ConfHandler.py: allow inherit or include without parameter

[Mark Hatle]

On 6/8/17 1:24 AM, Patrick Ohly wrote:
> On Wed, 2017-06-07 at 14:47 +0000, Peter Kjellerstedt wrote:
>>> -----Original Message-----
>>> From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
>>> devel-bounces@lists.openembedded.org] On Behalf Of Patrick Ohly
>>> Sent: den 7 juni 2017 15:56
>>> To: bitbake-devel@lists.openembedded.org
>>> Subject: [bitbake-devel] [PATCH 1/2] ConfHandler.py: allow inherit or
>>> include without parameter
>>
>> Change "inherit or include" to "include or require".
>>
>>> Writing .bbappends that only have an effect when some configuration
>>> variable like DISTRO_FEATURES is changed becomes easier when allowing
>>> "inherit" or "require" without a parameter. The same was already
>>
>> Change "inherit" to "include".
> 
> You are right of course. At least I was consistent in my mistake ;-} I
> guess I'm still struggling with the oddly named "require" vs. "include"
> distinction.

require -- it's an error if it doesn't exist
include -- it's NOT an error if it doesn't exist

Yes, this has messed up others, but too late to do anything about it without a
massive change.  (These can load 'any' files...)

(while inherit is only class files)

--Mark

> I can send a V2 if there are no other objections, or perhaps it can be
> fixed up as part of merging?
> 

[PATCH 2/2] ConfHandler.py: allow inherit or include with multiple parameters

[Patrick Ohly]

"inherit" already allows inheriting more than one class in a single
statement. The same also makes sense for "include" and "require",
because then one can generate a list of files to be included
dynamically also for the case that more than one file needs to be
included.

For example, with a new utility method, include files could be
included depending on the current distro features like this:

   require ${@ oe.utils.optional_includes(d, \
               "foo,bar:foo-or-bar.inc xyz:x.inc,y.inc,z.inc", \
               "DISTRO_FEATURES")}

Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
---
 lib/bb/parse/parse_py/ConfHandler.py | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/lib/bb/parse/parse_py/ConfHandler.py b/lib/bb/parse/parse_py/ConfHandler.py
index b006d06..97aa130 100644
--- a/lib/bb/parse/parse_py/ConfHandler.py
+++ b/lib/bb/parse/parse_py/ConfHandler.py
@@ -69,21 +69,25 @@ def init(data):
 def supports(fn, d):
     return fn[-5:] == ".conf"
 
-def include(parentfn, fn, lineno, data, error_out):
+def include(parentfn, fns, lineno, data, error_out):
     """
     error_out: A string indicating the verb (e.g. "include", "inherit") to be
     used in a ParseError that will be raised if the file to be included could
     not be included. Specify False to avoid raising an error in this case.
     """
-    if parentfn == fn: # prevent infinite recursion
-        return None
-
-    fn = data.expand(fn)
+    fns = data.expand(fns)
     parentfn = data.expand(parentfn)
 
-    if not fn:
-        # "include" or "require" without parameter is fine, just return.
-        return
+    # "include" or "require" accept zero to n space-separated file names to include.
+    for fn in fns.split():
+        include_single_file(parentfn, fn, lineno, data, error_out)
+
+def include_single_file(parentfn, fn, lineno, data, error_out):
+    """
+    Helper function for include() which does not expand or split its parameters.
+    """
+    if parentfn == fn: # prevent infinite recursion
+        return None
 
     if not os.path.isabs(fn):
         dname = os.path.dirname(parentfn)
-- 
git-series 0.9.1

Re: [PATCH 2/2] ConfHandler.py: allow inherit or include with multiple parameters

[Peter Kjellerstedt]

> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> devel-bounces@lists.openembedded.org] On Behalf Of Patrick Ohly
> Sent: den 7 juni 2017 15:56
> To: bitbake-devel@lists.openembedded.org
> Subject: [bitbake-devel] [PATCH 2/2] ConfHandler.py: allow inherit or
> include with multiple parameters

Change "inherit or include" to "include or require".

> "inherit" already allows inheriting more than one class in a single
> statement. The same also makes sense for "include" and "require",
> because then one can generate a list of files to be included
> dynamically also for the case that more than one file needs to be
> included.
> 
> For example, with a new utility method, include files could be
> included depending on the current distro features like this:
> 
>    require ${@ oe.utils.optional_includes(d, \
>                "foo,bar:foo-or-bar.inc xyz:x.inc,y.inc,z.inc", \
>                "DISTRO_FEATURES")}
> 
> Signed-off-by: Patrick Ohly <patrick.ohly@intel.com>
> ---
>  lib/bb/parse/parse_py/ConfHandler.py | 20 ++++++++++++--------
>  1 file changed, 12 insertions(+), 8 deletions(-)
> 
> diff --git a/lib/bb/parse/parse_py/ConfHandler.py
> b/lib/bb/parse/parse_py/ConfHandler.py
> index b006d06..97aa130 100644
> --- a/lib/bb/parse/parse_py/ConfHandler.py
> +++ b/lib/bb/parse/parse_py/ConfHandler.py
> @@ -69,21 +69,25 @@ def init(data):
>  def supports(fn, d):
>      return fn[-5:] == ".conf"
> 
> -def include(parentfn, fn, lineno, data, error_out):
> +def include(parentfn, fns, lineno, data, error_out):
>      """
>      error_out: A string indicating the verb (e.g. "include", "inherit") to be
>      used in a ParseError that will be raised if the file to be included could
>      not be included. Specify False to avoid raising an error in this case.
>      """
> -    if parentfn == fn: # prevent infinite recursion
> -        return None
> -
> -    fn = data.expand(fn)
> +    fns = data.expand(fns)
>      parentfn = data.expand(parentfn)
> 
> -    if not fn:
> -        # "include" or "require" without parameter is fine, just return.
> -        return
> +    # "include" or "require" accept zero to n space-separated file names to include.
> +    for fn in fns.split():
> +        include_single_file(parentfn, fn, lineno, data, error_out)
> +
> +def include_single_file(parentfn, fn, lineno, data, error_out):
> +    """
> +    Helper function for include() which does not expand or split its parameters.
> +    """
> +    if parentfn == fn: # prevent infinite recursion
> +        return None
> 
>      if not os.path.isabs(fn):
>          dname = os.path.dirname(parentfn)
> --
> git-series 0.9.1

//Peter