[PATCH] wic: allow bitbake variables in kickstarter files

[PATCH] wic: allow bitbake variables in kickstarter files

[Rasmus Villemoes]

image_types_wic.bbclass has a mechanism for doing variable substitution
on .wks files by simply letting the input file be called
.wks.in. However, that doesn't allow using variables in files included
via the include directive.

This adds (somewhat naive) support for variable substitution in all
files parsed by wic. The user should add all required variables to
WICVARS to get them exported appropriately.

Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
---
 scripts/lib/wic/ksparser.py | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index e590b2fe3c..3dc17d7fde 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -28,14 +28,30 @@
 import os
 import shlex
 import logging
+import re
 
 from argparse import ArgumentParser, ArgumentError, ArgumentTypeError
 
 from wic.engine import find_canned
 from wic.partition import Partition
+from wic.misc import get_bitbake_var
 
 logger = logging.getLogger('wic')
 
+__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
+
+def expand_line(line):
+    while True:
+        m = __expand_var_regexp__.search(line)
+        if not m:
+            return line
+        key = m.group()[2:-1]
+        val = get_bitbake_var(key)
+        if val is None:
+            logger.warning("cannot expand variable %s" % key)
+            return line
+        line = line[:m.start()] + val + line[m.end():]
+
 class KickStartError(Exception):
     """Custom exception."""
     pass
@@ -189,6 +205,7 @@ class KickStart():
                 line = line.strip()
                 lineno += 1
                 if line and line[0] != '#':
+                    line = expand_line(line)
                     try:
                         line_args = shlex.split(line)
                         parsed = parser.parse_args(line_args)
-- 
2.15.1

Re: [PATCH] wic: allow bitbake variables in kickstarter files

[Rasmus Villemoes]

On 2018-03-22 14:44, Rasmus Villemoes wrote:
> image_types_wic.bbclass has a mechanism for doing variable substitution
> on .wks files by simply letting the input file be called
> .wks.in. However, that doesn't allow using variables in files included
> via the include directive.
> 
> This adds (somewhat naive) support for variable substitution in all
> files parsed by wic. The user should add all required variables to
> WICVARS to get them exported appropriately.

ping.

Re: [PATCH] wic: allow bitbake variables in kickstarter files

[Rasmus Villemoes]

On 2018-03-22 14:44, Rasmus Villemoes wrote:
> image_types_wic.bbclass has a mechanism for doing variable substitution
> on .wks files by simply letting the input file be called
> .wks.in. However, that doesn't allow using variables in files included
> via the include directive.
> 
> This adds (somewhat naive) support for variable substitution in all
> files parsed by wic. The user should add all required variables to
> WICVARS to get them exported appropriately.
> 
> Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>

ping^2

Re: [PATCH] wic: allow bitbake variables in kickstarter files

[Martin Hundebøll]

On 2018-03-22 14:44, Rasmus Villemoes wrote:
> image_types_wic.bbclass has a mechanism for doing variable substitution
> on .wks files by simply letting the input file be called
> .wks.in. However, that doesn't allow using variables in files included
> via the include directive.
> 
> This adds (somewhat naive) support for variable substitution in all
> files parsed by wic. The user should add all required variables to
> WICVARS to get them exported appropriately.
> 
> Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>

ping³

[PATCH resend] wic: allow bitbake variables in kickstarter files

[Rasmus Villemoes]

image_types_wic.bbclass has a mechanism for doing variable substitution
on .wks files by simply letting the input file be called
.wks.in. However, that doesn't allow using variables in files included
via the include directive. This is unfortunate, because lacking either
the ability to include other files or variable substitution leads to
fragile and error-prone duplication between kickstarter files and
recipes/configuration files used for various boards.

This adds (somewhat naive) support for variable substitution in all
files parsed by wic. The user should add all required variables to
WICVARS to get them exported appropriately.

Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
---
 scripts/lib/wic/ksparser.py | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index e590b2fe3c..3dc17d7fde 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -28,14 +28,30 @@
 import os
 import shlex
 import logging
+import re
 
 from argparse import ArgumentParser, ArgumentError, ArgumentTypeError
 
 from wic.engine import find_canned
 from wic.partition import Partition
+from wic.misc import get_bitbake_var
 
 logger = logging.getLogger('wic')
 
+__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
+
+def expand_line(line):
+    while True:
+        m = __expand_var_regexp__.search(line)
+        if not m:
+            return line
+        key = m.group()[2:-1]
+        val = get_bitbake_var(key)
+        if val is None:
+            logger.warning("cannot expand variable %s" % key)
+            return line
+        line = line[:m.start()] + val + line[m.end():]
+
 class KickStartError(Exception):
     """Custom exception."""
     pass
@@ -189,6 +205,7 @@ class KickStart():
                 line = line.strip()
                 lineno += 1
                 if line and line[0] != '#':
+                    line = expand_line(line)
                     try:
                         line_args = shlex.split(line)
                         parsed = parser.parse_args(line_args)
-- 
2.16.4

Re: [PATCH resend] wic: allow bitbake variables in kickstarter files

[Rasmus Villemoes]

On 2018-07-03 14:54, Rasmus Villemoes wrote:
> image_types_wic.bbclass has a mechanism for doing variable substitution
> on .wks files by simply letting the input file be called
> .wks.in. However, that doesn't allow using variables in files included
> via the include directive. This is unfortunate, because lacking either
> the ability to include other files or variable substitution leads to
> fragile and error-prone duplication between kickstarter files and
> recipes/configuration files used for various boards.
> 
> This adds (somewhat naive) support for variable substitution in all
> files parsed by wic. The user should add all required variables to
> WICVARS to get them exported appropriately.

ping^n

Re: [PATCH resend] wic: allow bitbake variables in kickstarter files

[Sean Nyekjær]

On 2018-07-17 10:14, Rasmus Villemoes wrote:
> On 2018-07-03 14:54, Rasmus Villemoes wrote:
>> image_types_wic.bbclass has a mechanism for doing variable substitution
>> on .wks files by simply letting the input file be called
>> .wks.in. However, that doesn't allow using variables in files included
>> via the include directive. This is unfortunate, because lacking either
>> the ability to include other files or variable substitution leads to
>> fragile and error-prone duplication between kickstarter files and
>> recipes/configuration files used for various boards.
>>
>> This adds (somewhat naive) support for variable substitution in all
>> files parsed by wic. The user should add all required variables to
>> WICVARS to get them exported appropriately.
> 
> ping^n
> 
ping^5

Hi can you please take a look at this :-) Please...

/Sean

[PATCH resend^2] wic: allow bitbake variables in kickstarter files

[Rasmus Villemoes]

image_types_wic.bbclass has a mechanism for doing variable substitution
on .wks files by simply letting the input file be called
.wks.in. However, that doesn't allow using variables in files included
via the include directive. This is unfortunate, because lacking either
the ability to include other files or variable substitution leads to
fragile and error-prone duplication between kickstarter files and
recipes/configuration files used for various boards.

This adds (somewhat naive) support for variable substitution in all
files parsed by wic. The user should add all required variables to
WICVARS to get them exported appropriately.

Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
---
scripts/lib/wic/ksparser.py | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)

diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
index 7e5a9c5092..08baf76123 100644
--- a/scripts/lib/wic/ksparser.py
+++ b/scripts/lib/wic/ksparser.py
@@ -28,14 +28,30 @@
 import os
 import shlex
 import logging
+import re
 
 from argparse import ArgumentParser, ArgumentError, ArgumentTypeError
 
 from wic.engine import find_canned
 from wic.partition import Partition
+from wic.misc import get_bitbake_var
 
 logger = logging.getLogger('wic')
 
+__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
+
+def expand_line(line):
+    while True:
+        m = __expand_var_regexp__.search(line)
+        if not m:
+            return line
+        key = m.group()[2:-1]
+        val = get_bitbake_var(key)
+        if val is None:
+            logger.warning("cannot expand variable %s" % key)
+            return line
+        line = line[:m.start()] + val + line[m.end():]
+
 class KickStartError(Exception):
     """Custom exception."""
     pass
@@ -190,6 +206,7 @@ class KickStart():
                 line = line.strip()
                 lineno += 1
                 if line and line[0] != '#':
+                    line = expand_line(line)
                     try:
                         line_args = shlex.split(line)
                         parsed = parser.parse_args(line_args)
-- 
2.19.1.6.gbde171bbf5

Re: [PATCH resend^2] wic: allow bitbake variables in kickstarter files

[Rasmus Villemoes]

ping

On 29/12/2018 01.06, Rasmus Villemoes wrote:
> image_types_wic.bbclass has a mechanism for doing variable substitution
> on .wks files by simply letting the input file be called
> .wks.in. However, that doesn't allow using variables in files included
> via the include directive. This is unfortunate, because lacking either
> the ability to include other files or variable substitution leads to
> fragile and error-prone duplication between kickstarter files and
> recipes/configuration files used for various boards.
> 
> This adds (somewhat naive) support for variable substitution in all
> files parsed by wic. The user should add all required variables to
> WICVARS to get them exported appropriately.
> 
> Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
> ---
> scripts/lib/wic/ksparser.py | 17 +++++++++++++++++
>  1 file changed, 17 insertions(+)
> 
> diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> index 7e5a9c5092..08baf76123 100644
> --- a/scripts/lib/wic/ksparser.py
> +++ b/scripts/lib/wic/ksparser.py
> @@ -28,14 +28,30 @@
>  import os
>  import shlex
>  import logging
> +import re
>  
>  from argparse import ArgumentParser, ArgumentError, ArgumentTypeError
>  
>  from wic.engine import find_canned
>  from wic.partition import Partition
> +from wic.misc import get_bitbake_var
>  
>  logger = logging.getLogger('wic')
>  
> +__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
> +
> +def expand_line(line):
> +    while True:
> +        m = __expand_var_regexp__.search(line)
> +        if not m:
> +            return line
> +        key = m.group()[2:-1]
> +        val = get_bitbake_var(key)
> +        if val is None:
> +            logger.warning("cannot expand variable %s" % key)
> +            return line
> +        line = line[:m.start()] + val + line[m.end():]
> +
>  class KickStartError(Exception):
>      """Custom exception."""
>      pass
> @@ -190,6 +206,7 @@ class KickStart():
>                  line = line.strip()
>                  lineno += 1
>                  if line and line[0] != '#':
> +                    line = expand_line(line)
>                      try:
>                          line_args = shlex.split(line)
>                          parsed = parser.parse_args(line_args)
> 


-- 
Rasmus Villemoes
Software Developer
Prevas A/S
Hedeager 3
DK-8200 Aarhus N
+45 51210274
rasmus.villemoes@prevas.dk
www.prevas.dk

Re: [PATCH resend^2] wic: allow bitbake variables in kickstarter files

[Rasmus Villemoes]

ping ping

On 29/12/2018 01.06, Rasmus Villemoes wrote:
> image_types_wic.bbclass has a mechanism for doing variable substitution
> on .wks files by simply letting the input file be called
> .wks.in. However, that doesn't allow using variables in files included
> via the include directive. This is unfortunate, because lacking either
> the ability to include other files or variable substitution leads to
> fragile and error-prone duplication between kickstarter files and
> recipes/configuration files used for various boards.
> 
> This adds (somewhat naive) support for variable substitution in all
> files parsed by wic. The user should add all required variables to
> WICVARS to get them exported appropriately.
> 
> Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>
> ---
> scripts/lib/wic/ksparser.py | 17 +++++++++++++++++
>  1 file changed, 17 insertions(+)
> 
> diff --git a/scripts/lib/wic/ksparser.py b/scripts/lib/wic/ksparser.py
> index 7e5a9c5092..08baf76123 100644
> --- a/scripts/lib/wic/ksparser.py
> +++ b/scripts/lib/wic/ksparser.py
> @@ -28,14 +28,30 @@
>  import os
>  import shlex
>  import logging
> +import re
>  
>  from argparse import ArgumentParser, ArgumentError, ArgumentTypeError
>  
>  from wic.engine import find_canned
>  from wic.partition import Partition
> +from wic.misc import get_bitbake_var
>  
>  logger = logging.getLogger('wic')
>  
> +__expand_var_regexp__ = re.compile(r"\${[^{}@\n\t :]+}")
> +
> +def expand_line(line):
> +    while True:
> +        m = __expand_var_regexp__.search(line)
> +        if not m:
> +            return line
> +        key = m.group()[2:-1]
> +        val = get_bitbake_var(key)
> +        if val is None:
> +            logger.warning("cannot expand variable %s" % key)
> +            return line
> +        line = line[:m.start()] + val + line[m.end():]
> +
>  class KickStartError(Exception):
>      """Custom exception."""
>      pass
> @@ -190,6 +206,7 @@ class KickStart():
>                  line = line.strip()
>                  lineno += 1
>                  if line and line[0] != '#':
> +                    line = expand_line(line)
>                      try:
>                          line_args = shlex.split(line)
>                          parsed = parser.parse_args(line_args)
> 


-- 
Rasmus Villemoes
Software Developer
Prevas A/S
Hedeager 3
DK-8200 Aarhus N
+45 51210274
rasmus.villemoes@prevas.dk
www.prevas.dk

Re: [PATCH] wic: allow bitbake variables in kickstarter files

[Tom Rini]

On Thu, Mar 22, 2018 at 02:44:13PM +0100, Rasmus Villemoes wrote:

> image_types_wic.bbclass has a mechanism for doing variable substitution
> on .wks files by simply letting the input file be called
> .wks.in. However, that doesn't allow using variables in files included
> via the include directive.
> 
> This adds (somewhat naive) support for variable substitution in all
> files parsed by wic. The user should add all required variables to
> WICVARS to get them exported appropriately.
> 
> Signed-off-by: Rasmus Villemoes <rasmus.villemoes@prevas.dk>

This seems quite useful and I like the idea.  I don't know the wic
internals well enough to "Reviewed-by" this myself, sorry.

-- 
Tom