[PATCH 0/1] Add a syntax to clear variables and flags from a conf file

[PATCH 0/1] Add a syntax to clear variables and flags from a conf file

[Jérémy Rosen]

From: Jérémy Rosen <jeremy.rosen@openwide.fr>

While working on a Yocto project I had a need to reactivate a task that was
disabled (do_fetch[noexec]="1") Currently, bitbake does not check the value
of the noexec flag but only if the flag is set. The only way to unset a flag
is to use an inline python syntax like the one below

python () {
   d.delVarFlag("do_fetch","noexec")
}

Using inline python to do something as simple as clearing a flag sounded too
complicated to me so I added a new keyword "unset" to bitbake with the
following syntax :

unset VAR

will call d.delVar("VAR")

unset VAR[flag]

will call d.delVarFlag("VAR","flag")

Documentation has been updated accordingly


Jérémy Rosen (1):
  add a syntax to clear variable

 .../bitbake-user-manual-metadata.xml               | 17 +++++++++++
 bitbake/lib/bb/parse/ast.py                        | 33 ++++++++++++++++++++++
 bitbake/lib/bb/parse/parse_py/ConfHandler.py       | 12 ++++++++
 3 files changed, 62 insertions(+)

-- 
2.8.1

[PATCH 1/1] add a syntax to clear variable

[Jérémy Rosen]

From: Jérémy Rosen <jeremy.rosen@openwide.fr>

unset VAR
will clear variable VAR
unset VAR[flag]
will clear flag "flag" from var VAR

Signed-off-by: Jérémy Rosen <jeremy.rosen@openwide.fr>
---
 .../bitbake-user-manual-metadata.xml               | 17 +++++++++++
 bitbake/lib/bb/parse/ast.py                        | 33 ++++++++++++++++++++++
 bitbake/lib/bb/parse/parse_py/ConfHandler.py       | 12 ++++++++
 3 files changed, 62 insertions(+)

diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
index 6329cd6..c42907e 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
@@ -333,6 +333,23 @@
             </para>
         </section>
 
+        <section id='unseting-variables'>
+            <title>Unseting variables</title>
+
+            <para>
+                It is possible to completely remove a variable or a variable flag 
+                from BitBake's internal data dictionary using the "unset" keyword
+                Here is an example:
+                <literallayout class='monospaced'>
+        unset DATE
+        unset do_fetch[noexec]
+                </literallayout>
+                This example results in the <filename>DATE</filename> being unset and the flag
+                <filename>do_fetch[noexec]</filename> to be cleared
+            </para>
+
+        </section>
+
         <section id='providing-pathnames'>
             <title>Providing Pathnames</title>
 
diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py
index 8b9baa7..b407b09 100644
--- a/bitbake/lib/bb/parse/ast.py
+++ b/bitbake/lib/bb/parse/ast.py
@@ -69,6 +69,33 @@ class ExportNode(AstNode):
     def eval(self, data):
         data.setVarFlag(self.var, "export", 1, op = 'exported')
 
+class UnsetNode(AstNode):
+    def __init__(self, filename, lineno, var):
+        AstNode.__init__(self, filename, lineno)
+        self.var = var
+
+    def eval(self, data):
+        loginfo = {
+            'variable': self.var,
+            'file': self.filename,
+            'line': self.lineno,
+        }
+        data.delVar(self.var,**loginfo)
+
+class UnsetFlagNode(AstNode):
+    def __init__(self, filename, lineno, var, flag):
+        AstNode.__init__(self, filename, lineno)
+        self.var = var
+        self.flag = flag
+
+    def eval(self, data):
+        loginfo = {
+            'variable': self.var,
+            'file': self.filename,
+            'line': self.lineno,
+        }
+        data.delVarFlag(self.var, self.flag, **loginfo)
+
 class DataNode(AstNode):
     """
     Various data related updates. For the sake of sanity
@@ -270,6 +297,12 @@ def handleInclude(statements, filename, lineno, m, force):
 def handleExport(statements, filename, lineno, m):
     statements.append(ExportNode(filename, lineno, m.group(1)))
 
+def handleUnset(statements, filename, lineno, m):
+    statements.append(UnsetNode(filename, lineno, m.group(1)))
+
+def handleUnsetFlag(statements, filename, lineno, m):
+    statements.append(UnsetFlagNode(filename, lineno, m.group(1), m.group(2)))
+
 def handleData(statements, filename, lineno, groupd):
     statements.append(DataNode(filename, lineno, groupd))
 
diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
index b971292..875250d 100644
--- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
+++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
@@ -57,6 +57,8 @@ __config_regexp__  = re.compile( r"""
 __include_regexp__ = re.compile( r"include\s+(.+)" )
 __require_regexp__ = re.compile( r"require\s+(.+)" )
 __export_regexp__ = re.compile( r"export\s+([a-zA-Z0-9\-_+.${}/]+)$" )
+__unset_regexp__ = re.compile( r"unset\s+([a-zA-Z0-9\-_+.${}/]+)$" )
+__unset_flag_regexp__ = re.compile( r"unset\s+([a-zA-Z0-9\-_+.${}/]+)\[([a-zA-Z0-9\-_+.${}/]+)\]$" )
 
 def init(data):
     topdir = data.getVar('TOPDIR', False)
@@ -185,6 +187,16 @@ def feeder(lineno, s, fn, statements):
         ast.handleExport(statements, fn, lineno, m)
         return
 
+    m = __unset_regexp__.match(s)
+    if m:
+        ast.handleUnset(statements, fn, lineno, m)
+        return
+
+    m = __unset_flag_regexp__.match(s)
+    if m:
+        ast.handleUnsetFlag(statements, fn, lineno, m)
+        return
+
     raise ParseError("unparsed line: '%s'" % s, fn, lineno);
 
 # Add us to the handlers list
-- 
2.8.1

Re: [PATCH 1/1] add a syntax to clear variable

[Peter Kjellerstedt]

How will this interact with ?=, ??=, _append and _remove?

Shouldn't there be unit tests for this? It seems like it can easily 
break and/or be broken by other changes.

//Peter

> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> devel-bounces@lists.openembedded.org] On Behalf Of Jérémy Rosen
> Sent: den 21 juli 2016 12:56
> To: bitbake-devel@lists.openembedded.org
> Cc: Jérémy Rosen
> Subject: [bitbake-devel] [PATCH 1/1] add a syntax to clear variable
> 
> From: Jérémy Rosen <jeremy.rosen@openwide.fr>
> 
> unset VAR
> will clear variable VAR
> unset VAR[flag]
> will clear flag "flag" from var VAR
> 
> Signed-off-by: Jérémy Rosen <jeremy.rosen@openwide.fr>
> ---
>  .../bitbake-user-manual-metadata.xml               | 17 +++++++++++
>  bitbake/lib/bb/parse/ast.py                        | 33
> ++++++++++++++++++++++
>  bitbake/lib/bb/parse/parse_py/ConfHandler.py       | 12 ++++++++
>  3 files changed, 62 insertions(+)
> 
> diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-
> metadata.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-
> metadata.xml
> index 6329cd6..c42907e 100644
> --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
> +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
> @@ -333,6 +333,23 @@
>              </para>
>          </section>
> 
> +        <section id='unseting-variables'>
> +            <title>Unseting variables</title>
> +
> +            <para>
> +                It is possible to completely remove a variable or a
> variable flag
> +                from BitBake's internal data dictionary using the
> "unset" keyword
> +                Here is an example:
> +                <literallayout class='monospaced'>
> +        unset DATE
> +        unset do_fetch[noexec]
> +                </literallayout>
> +                This example results in the <filename>DATE</filename>
> being unset and the flag
> +                <filename>do_fetch[noexec]</filename> to be cleared
> +            </para>
> +
> +        </section>
> +
>          <section id='providing-pathnames'>
>              <title>Providing Pathnames</title>
> 
> diff --git a/bitbake/lib/bb/parse/ast.py b/bitbake/lib/bb/parse/ast.py
> index 8b9baa7..b407b09 100644
> --- a/bitbake/lib/bb/parse/ast.py
> +++ b/bitbake/lib/bb/parse/ast.py
> @@ -69,6 +69,33 @@ class ExportNode(AstNode):
>      def eval(self, data):
>          data.setVarFlag(self.var, "export", 1, op = 'exported')
> 
> +class UnsetNode(AstNode):
> +    def __init__(self, filename, lineno, var):
> +        AstNode.__init__(self, filename, lineno)
> +        self.var = var
> +
> +    def eval(self, data):
> +        loginfo = {
> +            'variable': self.var,
> +            'file': self.filename,
> +            'line': self.lineno,
> +        }
> +        data.delVar(self.var,**loginfo)
> +
> +class UnsetFlagNode(AstNode):
> +    def __init__(self, filename, lineno, var, flag):
> +        AstNode.__init__(self, filename, lineno)
> +        self.var = var
> +        self.flag = flag
> +
> +    def eval(self, data):
> +        loginfo = {
> +            'variable': self.var,
> +            'file': self.filename,
> +            'line': self.lineno,
> +        }
> +        data.delVarFlag(self.var, self.flag, **loginfo)
> +
>  class DataNode(AstNode):
>      """
>      Various data related updates. For the sake of sanity
> @@ -270,6 +297,12 @@ def handleInclude(statements, filename, lineno, m,
> force):
>  def handleExport(statements, filename, lineno, m):
>      statements.append(ExportNode(filename, lineno, m.group(1)))
> 
> +def handleUnset(statements, filename, lineno, m):
> +    statements.append(UnsetNode(filename, lineno, m.group(1)))
> +
> +def handleUnsetFlag(statements, filename, lineno, m):
> +    statements.append(UnsetFlagNode(filename, lineno, m.group(1),
> m.group(2)))
> +
>  def handleData(statements, filename, lineno, groupd):
>      statements.append(DataNode(filename, lineno, groupd))
> 
> diff --git a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
> b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
> index b971292..875250d 100644
> --- a/bitbake/lib/bb/parse/parse_py/ConfHandler.py
> +++ b/bitbake/lib/bb/parse/parse_py/ConfHandler.py
> @@ -57,6 +57,8 @@ __config_regexp__  = re.compile( r"""
>  __include_regexp__ = re.compile( r"include\s+(.+)" )
>  __require_regexp__ = re.compile( r"require\s+(.+)" )
>  __export_regexp__ = re.compile( r"export\s+([a-zA-Z0-9\-_+.${}/]+)$" )
> +__unset_regexp__ = re.compile( r"unset\s+([a-zA-Z0-9\-_+.${}/]+)$" )
> +__unset_flag_regexp__ = re.compile( r"unset\s+([a-zA-Z0-9\-
> _+.${}/]+)\[([a-zA-Z0-9\-_+.${}/]+)\]$" )
> 
>  def init(data):
>      topdir = data.getVar('TOPDIR', False)
> @@ -185,6 +187,16 @@ def feeder(lineno, s, fn, statements):
>          ast.handleExport(statements, fn, lineno, m)
>          return
> 
> +    m = __unset_regexp__.match(s)
> +    if m:
> +        ast.handleUnset(statements, fn, lineno, m)
> +        return
> +
> +    m = __unset_flag_regexp__.match(s)
> +    if m:
> +        ast.handleUnsetFlag(statements, fn, lineno, m)
> +        return
> +
>      raise ParseError("unparsed line: '%s'" % s, fn, lineno);
> 
>  # Add us to the handlers list
> --
> 2.8.1
> 
> --
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel

Re: [PATCH 1/1] add a syntax to clear variable

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 870 bytes --]

On Mon, Aug 15, 2016 at 7:44 AM, Peter Kjellerstedt <
peter.kjellerstedt@axis.com> wrote:

> How will this interact with ?=, ??=, _append and _remove?
>
> Shouldn't there be unit tests for this? It seems like it can easily
> break and/or be broken by other changes.
>

It's implemented as an immediate operation, so it'd happen long before ??=,
_append, or _remove are applied. ?= is also immediate, so isn't relevant --
?= before it would do nothing, as the unset would undo it, and ?= after the
unset would apply fine.

I think that's the most sane approach for something like this. If you want
the operation postponed, you can use anonymous python for that. Agreed re:
unit tests, though.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1382 bytes --]

Re: [PATCH 1/1] add a syntax to clear variable

[Jérémy Rosen]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

As Christopher explained, this is an immediate operation.

I'll look into unit test, I've never looked into this so I'm not sure 
how they work...


On 15/08/2016 16:47, Christopher Larson wrote:
>
> On Mon, Aug 15, 2016 at 7:44 AM, Peter Kjellerstedt 
> <peter.kjellerstedt@axis.com <mailto:peter.kjellerstedt@axis.com>> wrote:
>
>     How will this interact with ?=, ??=, _append and _remove?
>
>     Shouldn't there be unit tests for this? It seems like it can easily
>     break and/or be broken by other changes.
>
>
> It's implemented as an immediate operation, so it'd happen long before 
> ??=, _append, or _remove are applied. ?= is also immediate, so isn't 
> relevant -- ?= before it would do nothing, as the unset would undo it, 
> and ?= after the unset would apply fine.
>
> I think that's the most sane approach for something like this. If you 
> want the operation postponed, you can use anonymous python for that. 
> Agreed re: unit tests, though.
> -- 
> Christopher Larson
> clarson at kergoth dot com
> Founder - BitBake, OpenEmbedded, OpenZaurus
> Maintainer - Tslib
> Senior Software Engineer, Mentor Graphics


[-- Attachment #2: Type: text/html, Size: 2493 bytes --]

Re: [PATCH 1/1] add a syntax to clear variable

[Richard Purdie]

On Tue, 2016-08-16 at 12:38 +0200, Jérémy Rosen wrote:
> As Christopher explained, this is an immediate operation.
> I'll look into unit test, I've never looked into this so I'm not sure
> how they work...

If you run "bitbake-selftest" you will see them in action. There is a
way to disable the network tests to make them much faster (see the help
page).

Cheers,

Richard

Re: [PATCH 1/1] add a syntax to clear variable

[Ulf.Magnusson]

Hello,

> -----Original Message-----
> From: bitbake-devel-bounces@lists.openembedded.org [mailto:bitbake-
> devel-bounces@lists.openembedded.org] On Behalf Of Jérémy Rosen
> Sent: den 21 juli 2016 12:56
> To: bitbake-devel@lists.openembedded.org
> Cc: Jérémy Rosen
> Subject: [bitbake-devel] [PATCH 1/1] add a syntax to clear variable
>
> From: Jérémy Rosen <jeremy.rosen@openwide.fr>
>
> unset VAR
> will clear variable VAR
> unset VAR[flag]
> will clear flag "flag" from var VAR
>
> Signed-off-by: Jérémy Rosen <jeremy.rosen@openwide.fr>
> ---
>  .../bitbake-user-manual-metadata.xml               | 17 +++++++++++
>  bitbake/lib/bb/parse/ast.py                        | 33
> ++++++++++++++++++++++
>  bitbake/lib/bb/parse/parse_py/ConfHandler.py       | 12 ++++++++
>  3 files changed, 62 insertions(+)
>
> diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-
> metadata.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-
> metadata.xml
> index 6329cd6..c42907e 100644
> --- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
> +++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-metadata.xml
> @@ -333,6 +333,23 @@
>              </para>
>          </section>
>
> +        <section id='unseting-variables'>

"Unsetting" is spelled with two t's.

> +            <title>Unseting variables</title>
> +
> +            <para>
> +                It is possible to completely remove a variable or a
> variable flag
> +                from BitBake's internal data dictionary using the
> "unset" keyword

There's a missing period at the end of this sentence, and "by using"
sounds a bit better.

> +                Here is an example:
> +                <literallayout class='monospaced'>
> +        unset DATE
> +        unset do_fetch[noexec]
> +                </literallayout>
> +                This example results in the <filename>DATE</filename>
> being unset and the flag
> +                <filename>do_fetch[noexec]</filename> to be cleared

"These two statements remove the DATE variable and the do_fetch[noexec]
variable flag" is better. "...results in... to be cleared" sounds weird.

Cheers,
Ulf

Re: [PATCH 1/1] add a syntax to clear variable

[Jérémy Rosen]

Will fix all those wording in V2.0

thx


On 15/08/2016 16:58, Ulf.Magnusson@bmw.de wrote:
> These two statements remove the DATE variable and the do_fetch[noexec]
> variable flag" is better. "...results in... to be cleared" sounds weird.