[PATCH 0/5] Make BBMASK support multiple regular expressions

[PATCH 0/5] Make BBMASK support multiple regular expressions

[Peter Kjellerstedt]

[ This is a bitbake change, but since the documentation is all over
  Poky I am cross-posting to the Poky list as well. ]

This series of patches changes BBMASK so that it can contain multiple,
whitespace separated regular expressions. It should be fully backwards
compatible with any existing values in BBMASK.

The first two patches are for bitbake, the third is for yocto-docs,
the fourth is for OE-Core, and the fifth is for meta-yocto...

//Peter

The following changes since commit 3d2c0f5902cacf9d8544bf263b51ef0dd1a7218c:

  cmake: update to 3.4.2 (2016-01-26 22:49:40 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib pkj/bbmask_multiple_re
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=pkj/bbmask_multiple_re

Peter Kjellerstedt (5):
  cooker: Allow BBMASK to contain multiple regular expressions
  bitbake-user-manual-ref-variables: Update the help for BBMASK
  ref-manual: Update the help for BBMASK
  documentation.conf: Update the help for BBMASK
  local.conf.sample.extended: Update the info about BBMASK

 .../bitbake-user-manual-ref-variables.xml          | 20 ++++++++---------
 bitbake/lib/bb/cooker.py                           | 17 ++++++++++++--
 documentation/ref-manual/ref-variables.xml         | 26 +++++++++-------------
 meta-yocto/conf/local.conf.sample.extended         |  4 ++--
 meta/conf/documentation.conf                       |  2 +-
 5 files changed, 37 insertions(+), 32 deletions(-)

-- 
2.1.0

[PATCH 1/5] cooker: Allow BBMASK to contain multiple regular expressions

[Peter Kjellerstedt]

Before, BBMASK was only permitted to contain one regular expression.
This made it hard to add to the BBMASK in multiple places as one was
supposed to separate the different regular expressions with a "|"
rather than with whitespace as is customary in BitBake variables.

Now one can specify any number of regular expressions in BBMASK. This
makes it possible to, e.g., mask out recipes in another layer from the
layer.conf file.

This also properly ignores any regular expressions that do not compile
(before an invalid regular expression would cause a ParseError in the
first bbappend file found stating that it was not a BitBake file...)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 bitbake/lib/bb/cooker.py | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/cooker.py b/bitbake/lib/bb/cooker.py
index 9c58d95..a02d143 100644
--- a/bitbake/lib/bb/cooker.py
+++ b/bitbake/lib/bb/cooker.py
@@ -1759,11 +1759,24 @@ class CookerCollectFiles(object):
         bbmask = config.getVar('BBMASK', True)
 
         if bbmask:
+            # First validate the individual regular expressions and ignore any
+            # that do not compile
+            bbmasks = []
+            for mask in bbmask.split():
+                try:
+                    re.compile(mask)
+                    bbmasks.append(mask)
+                except sre_constants.error:
+                    collectlog.critical("BBMASK contains an invalid regular expression, ignoring: %s" % mask)
+
+            # Then validate the combined regular expressions. This should never
+            # fail, but better safe than sorry...
+            bbmask = "|".join(bbmasks)
             try:
                 bbmask_compiled = re.compile(bbmask)
             except sre_constants.error:
-                collectlog.critical("BBMASK is not a valid regular expression, ignoring.")
-                return list(newfiles), 0
+                collectlog.critical("BBMASK is not a valid regular expression, ignoring: %s" % bbmask)
+                bbmask = None
 
         bbfiles = []
         bbappend = []
-- 
2.1.0

[PATCH 2/5] bitbake-user-manual-ref-variables: Update the help for BBMASK

[Peter Kjellerstedt]

Update the help for BBMASK to reflect that it is now allowed to
contain multiple regular expressions. Also changed the examples for
BBMASK to be a bit more diverse.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 .../bitbake-user-manual-ref-variables.xml            | 20 +++++++++-----------
 1 file changed, 9 insertions(+), 11 deletions(-)

diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
index e5aeffc..ae7e4ce 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-ref-variables.xml
@@ -1142,14 +1142,14 @@
                     to "hide" these <filename>.bb</filename> and
                     <filename>.bbappend</filename> files.
                     BitBake ignores any recipe or recipe append files that
-                    match the expression.
+                    match any of the expressions.
                     It is as if BitBake does not see them at all.
                     Consequently, matching files are not parsed or otherwise
                     used by BitBake.</para>
                 <para>
-                    The value you provide is passed to Python's regular
+                    The values you provide are passed to Python's regular
                     expression compiler.
-                    The expression is compared against the full paths to
+                    The expressions are compared against the full paths to
                     the files.
                     For complete syntax information, see Python's
                     documentation at
@@ -1165,18 +1165,16 @@
      BBMASK = "meta-ti/recipes-misc/"
                     </literallayout>
                     If you want to mask out multiple directories or recipes,
-                    use the vertical bar to separate the regular expression
-                    fragments.
+                    you can specify multiple regular expression fragments.
                     This next example masks out multiple directories and
                     individual recipes:
                     <literallayout class='monospaced'>
-     BBMASK = "meta-ti/recipes-misc/|meta-ti/recipes-ti/packagegroup/"
-     BBMASK .= "|.*meta-oe/recipes-support/"
-     BBMASK .= "|.*openldap"
-     BBMASK .= "|.*opencv"
-     BBMASK .= "|.*lzma"
+     BBMASK += "/meta-ti/recipes-misc/ meta-ti/recipes-ti/packagegroup/"
+     BBMASK += "/meta-oe/recipes-support/"
+     BBMASK += "/meta-foo/.*/openldap"
+     BBMASK += "opencv.*\.bbappend"
+     BBMASK += "lzma"
                     </literallayout>
-                    Notice how the vertical bar is used to append the fragments.
                     <note>
                         When specifying a directory name, use the trailing
                         slash character to ensure you match just that directory
-- 
2.1.0

[PATCH 3/5] ref-manual: Update the help for BBMASK

[Peter Kjellerstedt]

Update the help for BBMASK to reflect that it is now allowed to
contain multiple regular expressions. Also changed the examples for
BBMASK to be a bit more diverse.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 documentation/ref-manual/ref-variables.xml | 26 ++++++++++----------------
 1 file changed, 10 insertions(+), 16 deletions(-)

diff --git a/documentation/ref-manual/ref-variables.xml b/documentation/ref-manual/ref-variables.xml
index 673e431..49aea7a 100644
--- a/documentation/ref-manual/ref-variables.xml
+++ b/documentation/ref-manual/ref-variables.xml
@@ -1200,17 +1200,13 @@
 
         <glossentry id='var-BBMASK'><glossterm>BBMASK</glossterm>
             <info>
-                BBMASK[doc] = "Prevents BitBake from processing specific recipes or recipe append files. Use the BBMASK variable from within conf/local.conf."
+                BBMASK[doc] = "Prevents BitBake from processing specific recipes or recipe append files."
             </info>
             <glossdef>
                 <para role="glossdeffirst">
 <!--                <para role="glossdeffirst"><imagedata fileref="figures/define-generic.png" /> -->
                     Prevents BitBake from processing recipes and recipe
                     append files.
-                    Use the <filename>BBMASK</filename> variable from within the
-                    <filename>conf/local.conf</filename> file found
-                    in the
-                    <ulink url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
                 </para>
 
                 <para>
@@ -1218,14 +1214,14 @@
                     to "hide" these <filename>.bb</filename> and
                     <filename>.bbappend</filename> files.
                     BitBake ignores any recipe or recipe append files that
-                    match the expression.
+                    match any of the expressions.
                     It is as if BitBake does not see them at all.
                     Consequently, matching files are not parsed or otherwise
                     used by BitBake.</para>
                 <para>
-                    The value you provide is passed to Python's regular
+                    The values you provide are passed to Python's regular
                     expression compiler.
-                    The expression is compared against the full paths to
+                    The expressions are compared against the full paths to
                     the files.
                     For complete syntax information, see Python's
                     documentation at
@@ -1241,18 +1237,16 @@
      BBMASK = "meta-ti/recipes-misc/"
                     </literallayout>
                     If you want to mask out multiple directories or recipes,
-                    use the vertical bar to separate the regular expression
-                    fragments.
+                    you can specify multiple regular expression fragments.
                     This next example masks out multiple directories and
                     individual recipes:
                     <literallayout class='monospaced'>
-     BBMASK = "meta-ti/recipes-misc/|meta-ti/recipes-ti/packagegroup/"
-     BBMASK .= "|.*meta-oe/recipes-support/"
-     BBMASK .= "|.*openldap"
-     BBMASK .= "|.*opencv"
-     BBMASK .= "|.*lzma"
+     BBMASK += "/meta-ti/recipes-misc/ meta-ti/recipes-ti/packagegroup/"
+     BBMASK += "/meta-oe/recipes-support/"
+     BBMASK += "/meta-foo/.*/openldap"
+     BBMASK += "opencv.*\.bbappend"
+     BBMASK += "lzma"
                     </literallayout>
-                    Notice how the vertical bar is used to append the fragments.
                     <note>
                         When specifying a directory name, use the trailing
                         slash character to ensure you match just that directory
-- 
2.1.0

Re: [poky] [PATCH 3/5] ref-manual: Update the help for BBMASK

[Scott Rifenbark]

[-- Attachment #1: Type: text/plain, Size: 4509 bytes --]

This has been applied.  see
http://www.yoctoproject.org/docs/2.1/ref-manual/ref-manual.html#var-BBMASK.


Thanks

On Wed, Jan 27, 2016 at 9:23 AM, Peter Kjellerstedt <
peter.kjellerstedt@axis.com> wrote:

> Update the help for BBMASK to reflect that it is now allowed to
> contain multiple regular expressions. Also changed the examples for
> BBMASK to be a bit more diverse.
>
> Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> ---
>  documentation/ref-manual/ref-variables.xml | 26 ++++++++++----------------
>  1 file changed, 10 insertions(+), 16 deletions(-)
>
> diff --git a/documentation/ref-manual/ref-variables.xml
> b/documentation/ref-manual/ref-variables.xml
> index 673e431..49aea7a 100644
> --- a/documentation/ref-manual/ref-variables.xml
> +++ b/documentation/ref-manual/ref-variables.xml
> @@ -1200,17 +1200,13 @@
>
>          <glossentry id='var-BBMASK'><glossterm>BBMASK</glossterm>
>              <info>
> -                BBMASK[doc] = "Prevents BitBake from processing specific
> recipes or recipe append files. Use the BBMASK variable from within
> conf/local.conf."
> +                BBMASK[doc] = "Prevents BitBake from processing specific
> recipes or recipe append files."
>              </info>
>              <glossdef>
>                  <para role="glossdeffirst">
>  <!--                <para role="glossdeffirst"><imagedata
> fileref="figures/define-generic.png" /> -->
>                      Prevents BitBake from processing recipes and recipe
>                      append files.
> -                    Use the <filename>BBMASK</filename> variable from
> within the
> -                    <filename>conf/local.conf</filename> file found
> -                    in the
> -                    <ulink
> url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
>                  </para>
>
>                  <para>
> @@ -1218,14 +1214,14 @@
>                      to "hide" these <filename>.bb</filename> and
>                      <filename>.bbappend</filename> files.
>                      BitBake ignores any recipe or recipe append files that
> -                    match the expression.
> +                    match any of the expressions.
>                      It is as if BitBake does not see them at all.
>                      Consequently, matching files are not parsed or
> otherwise
>                      used by BitBake.</para>
>                  <para>
> -                    The value you provide is passed to Python's regular
> +                    The values you provide are passed to Python's regular
>                      expression compiler.
> -                    The expression is compared against the full paths to
> +                    The expressions are compared against the full paths to
>                      the files.
>                      For complete syntax information, see Python's
>                      documentation at
> @@ -1241,18 +1237,16 @@
>       BBMASK = "meta-ti/recipes-misc/"
>                      </literallayout>
>                      If you want to mask out multiple directories or
> recipes,
> -                    use the vertical bar to separate the regular
> expression
> -                    fragments.
> +                    you can specify multiple regular expression fragments.
>                      This next example masks out multiple directories and
>                      individual recipes:
>                      <literallayout class='monospaced'>
> -     BBMASK = "meta-ti/recipes-misc/|meta-ti/recipes-ti/packagegroup/"
> -     BBMASK .= "|.*meta-oe/recipes-support/"
> -     BBMASK .= "|.*openldap"
> -     BBMASK .= "|.*opencv"
> -     BBMASK .= "|.*lzma"
> +     BBMASK += "/meta-ti/recipes-misc/ meta-ti/recipes-ti/packagegroup/"
> +     BBMASK += "/meta-oe/recipes-support/"
> +     BBMASK += "/meta-foo/.*/openldap"
> +     BBMASK += "opencv.*\.bbappend"
> +     BBMASK += "lzma"
>                      </literallayout>
> -                    Notice how the vertical bar is used to append the
> fragments.
>                      <note>
>                          When specifying a directory name, use the trailing
>                          slash character to ensure you match just that
> directory
> --
> 2.1.0
>
> --
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky
>

[-- Attachment #2: Type: text/html, Size: 6168 bytes --]

Re: [PATCH 3/5] ref-manual: Update the help for BBMASK

[Scott Rifenbark]

[-- Attachment #1: Type: text/plain, Size: 4509 bytes --]

This has been applied.  see
http://www.yoctoproject.org/docs/2.1/ref-manual/ref-manual.html#var-BBMASK.


Thanks

On Wed, Jan 27, 2016 at 9:23 AM, Peter Kjellerstedt <
peter.kjellerstedt@axis.com> wrote:

> Update the help for BBMASK to reflect that it is now allowed to
> contain multiple regular expressions. Also changed the examples for
> BBMASK to be a bit more diverse.
>
> Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
> ---
>  documentation/ref-manual/ref-variables.xml | 26 ++++++++++----------------
>  1 file changed, 10 insertions(+), 16 deletions(-)
>
> diff --git a/documentation/ref-manual/ref-variables.xml
> b/documentation/ref-manual/ref-variables.xml
> index 673e431..49aea7a 100644
> --- a/documentation/ref-manual/ref-variables.xml
> +++ b/documentation/ref-manual/ref-variables.xml
> @@ -1200,17 +1200,13 @@
>
>          <glossentry id='var-BBMASK'><glossterm>BBMASK</glossterm>
>              <info>
> -                BBMASK[doc] = "Prevents BitBake from processing specific
> recipes or recipe append files. Use the BBMASK variable from within
> conf/local.conf."
> +                BBMASK[doc] = "Prevents BitBake from processing specific
> recipes or recipe append files."
>              </info>
>              <glossdef>
>                  <para role="glossdeffirst">
>  <!--                <para role="glossdeffirst"><imagedata
> fileref="figures/define-generic.png" /> -->
>                      Prevents BitBake from processing recipes and recipe
>                      append files.
> -                    Use the <filename>BBMASK</filename> variable from
> within the
> -                    <filename>conf/local.conf</filename> file found
> -                    in the
> -                    <ulink
> url='&YOCTO_DOCS_DEV_URL;#build-directory'>Build Directory</ulink>.
>                  </para>
>
>                  <para>
> @@ -1218,14 +1214,14 @@
>                      to "hide" these <filename>.bb</filename> and
>                      <filename>.bbappend</filename> files.
>                      BitBake ignores any recipe or recipe append files that
> -                    match the expression.
> +                    match any of the expressions.
>                      It is as if BitBake does not see them at all.
>                      Consequently, matching files are not parsed or
> otherwise
>                      used by BitBake.</para>
>                  <para>
> -                    The value you provide is passed to Python's regular
> +                    The values you provide are passed to Python's regular
>                      expression compiler.
> -                    The expression is compared against the full paths to
> +                    The expressions are compared against the full paths to
>                      the files.
>                      For complete syntax information, see Python's
>                      documentation at
> @@ -1241,18 +1237,16 @@
>       BBMASK = "meta-ti/recipes-misc/"
>                      </literallayout>
>                      If you want to mask out multiple directories or
> recipes,
> -                    use the vertical bar to separate the regular
> expression
> -                    fragments.
> +                    you can specify multiple regular expression fragments.
>                      This next example masks out multiple directories and
>                      individual recipes:
>                      <literallayout class='monospaced'>
> -     BBMASK = "meta-ti/recipes-misc/|meta-ti/recipes-ti/packagegroup/"
> -     BBMASK .= "|.*meta-oe/recipes-support/"
> -     BBMASK .= "|.*openldap"
> -     BBMASK .= "|.*opencv"
> -     BBMASK .= "|.*lzma"
> +     BBMASK += "/meta-ti/recipes-misc/ meta-ti/recipes-ti/packagegroup/"
> +     BBMASK += "/meta-oe/recipes-support/"
> +     BBMASK += "/meta-foo/.*/openldap"
> +     BBMASK += "opencv.*\.bbappend"
> +     BBMASK += "lzma"
>                      </literallayout>
> -                    Notice how the vertical bar is used to append the
> fragments.
>                      <note>
>                          When specifying a directory name, use the trailing
>                          slash character to ensure you match just that
> directory
> --
> 2.1.0
>
> --
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky
>

[-- Attachment #2: Type: text/html, Size: 6168 bytes --]

[PATCH 4/5] documentation.conf: Update the help for BBMASK

[Peter Kjellerstedt]

Since it is now possible to concatenate multiple regular expressions
into BBMASK, there is no longer any real reason to limit it to be
specified only in local.conf.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta/conf/documentation.conf | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/conf/documentation.conf b/meta/conf/documentation.conf
index e09f7d8..a74da77 100644
--- a/meta/conf/documentation.conf
+++ b/meta/conf/documentation.conf
@@ -91,7 +91,7 @@ BBINCLUDELOGS[doc] = "Variable that controls how BitBake displays logs on build
 BBINCLUDELOGS_LINES[doc] = "Amount of log lines printed on failure."
 BBLAYERS[doc] = "Lists the layers to enable during the build. This variable is defined in the bblayers.conf configuration file."
 BBLAYERS_NON_REMOVABLE[doc] = "Lists core layers that cannot be removed from the bblayers.conf file."
-BBMASK[doc] = "Prevents BitBake from processing specific recipes or recipe append files. Use the BBMASK variable from within conf/local.conf."
+BBMASK[doc] = "Prevents BitBake from processing specific recipes or recipe append files."
 BBPATH[doc] = "Used by BitBake to locate .bbclass and configuration files. This variable is analogous to the PATH variable."
 BBSERVER[doc] = "Points to the server that runs memory-resident BitBake."
 BINCONFIG_GLOB[doc] = "When inheriting binconfig.bbclass from a recipe, this variable specifies a wildcard for configuration scripts that need editing."
-- 
2.1.0

[PATCH 5/5] local.conf.sample.extended: Update the info about BBMASK

[Peter Kjellerstedt]

Update the info about BBMASK to reflect that it can now contain
multiple regular expressions.

Also correct a misspelled PREFERRED_VERSION in a comment.

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
---
 meta-yocto/conf/local.conf.sample.extended | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta-yocto/conf/local.conf.sample.extended b/meta-yocto/conf/local.conf.sample.extended
index 925270b..521cdcb 100644
--- a/meta-yocto/conf/local.conf.sample.extended
+++ b/meta-yocto/conf/local.conf.sample.extended
@@ -1,4 +1,4 @@
-# BBMASK is a regular expression that can be used to tell BitBake to ignore
+# BBMASK contains regular expressions that can be used to tell BitBake to ignore
 # certain recipes.
 #BBMASK = ""
 
@@ -375,7 +375,7 @@
 #
 # Currently the rootfs_rpm code has a hard depends on rpmresolve:do_populate_sysroot,
 # when using rpm4 the rpmresolve code will not compile due to a missing header file.
-# That dependency needs to be removed when using RPM4, also the PREFRRED_VERSION needs
+# That dependency needs to be removed when using RPM4, also the PREFERRED_VERSION needs
 # to be set. This example shows how to enable rpm4
 # PREFERRED_VERSION_rpm = "4.11.2"
 # PREFERRED_VERSION_rpm-native = "4.11.2"
-- 
2.1.0