[PATCH] Allow environment variables and ~ in bbconf paths

[PATCH] Allow environment variables and ~ in bbconf paths

[Ed Bartosh]

This should allow to use ~, $HOME and other environment
variables in bbconf paths.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/cookerdata.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py
index 6c11a60..ec5c9ea 100644
--- a/bitbake/lib/bb/cookerdata.py
+++ b/bitbake/lib/bb/cookerdata.py
@@ -271,6 +271,7 @@ class CookerDataBuilder(object):
             data = bb.data.createCopy(data)
             for layer in layers:
                 parselog.debug(2, "Adding layer %s", layer)
+                layer = os.path.expanduser(os.path.expandvars(layer))
                 data.setVar('LAYERDIR', layer)
                 data = parse_config_file(os.path.join(layer, "conf", "layer.conf"), data)
                 data.expandVarref('LAYERDIR')
-- 
2.1.4

Re: [PATCH] Allow environment variables and ~ in bbconf paths

[Rifenbark, Scott M]

Ed, 

This worked for me.... Thanks.

Scott

>-----Original Message-----
>From: Ed Bartosh [mailto:ed.bartosh@linux.intel.com]
>Sent: Tuesday, April 14, 2015 1:46 PM
>To: bitbake-devel@lists.openembedded.org
>Cc: Rifenbark, Scott M; Ed Bartosh
>Subject: [PATCH] Allow environment variables and ~ in bbconf paths
>
>This should allow to use ~, $HOME and other environment variables in bbconf
>paths.
>
>Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
>---
> bitbake/lib/bb/cookerdata.py | 1 +
> 1 file changed, 1 insertion(+)
>
>diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py
>index 6c11a60..ec5c9ea 100644
>--- a/bitbake/lib/bb/cookerdata.py
>+++ b/bitbake/lib/bb/cookerdata.py
>@@ -271,6 +271,7 @@ class CookerDataBuilder(object):
>             data = bb.data.createCopy(data)
>             for layer in layers:
>                 parselog.debug(2, "Adding layer %s", layer)
>+                layer = os.path.expanduser(os.path.expandvars(layer))
>                 data.setVar('LAYERDIR', layer)
>                 data = parse_config_file(os.path.join(layer, "conf", "layer.conf"),
>data)
>                 data.expandVarref('LAYERDIR')
>--
>2.1.4

Re: [PATCH] Allow environment variables and ~ in bbconf paths

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 939 bytes --]

On Tue, Apr 14, 2015 at 2:19 PM, Rifenbark, Scott M <
scott.m.rifenbark@intel.com> wrote:

> Ed,
>
> This worked for me.... Thanks.
>
> Scott
>
> >-----Original Message-----
> >From: Ed Bartosh [mailto:ed.bartosh@linux.intel.com]
> >Sent: Tuesday, April 14, 2015 1:46 PM
> >To: bitbake-devel@lists.openembedded.org
> >Cc: Rifenbark, Scott M; Ed Bartosh
> >Subject: [PATCH] Allow environment variables and ~ in bbconf paths
> >
> >This should allow to use ~, $HOME and other environment variables in
> bbconf
> >paths.
> >
> >Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
>

We already have a mechanism by which environment variables flow into the
metadata. Is there something wrong with using BB_ENV_EXTRAWHITE to use
${HOME} in this context?
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1623 bytes --]

Re: [PATCH] Allow environment variables and ~ in bbconf paths

[Ed Bartosh]

On Tue, Apr 14, 2015 at 02:33:41PM -0700, Christopher Larson wrote:
> On Tue, Apr 14, 2015 at 2:19 PM, Rifenbark, Scott M <
> scott.m.rifenbark@intel.com> wrote:
> 
> > Ed,
> >
> > This worked for me.... Thanks.
> >
> > Scott
> >
> > >-----Original Message-----
> > >From: Ed Bartosh [mailto:ed.bartosh@linux.intel.com]
> > >Sent: Tuesday, April 14, 2015 1:46 PM
> > >To: bitbake-devel@lists.openembedded.org
> > >Cc: Rifenbark, Scott M; Ed Bartosh
> > >Subject: [PATCH] Allow environment variables and ~ in bbconf paths
> > >
> > >This should allow to use ~, $HOME and other environment variables in
> > bbconf
> > >paths.
> > >
> > >Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> >
> 
> We already have a mechanism by which environment variables flow into the
> metadata. Is there something wrong with using BB_ENV_EXTRAWHITE to use
> ${HOME} in this context?

Thank you for pointing out to this. However, I couldn't make this mechanism
to work for bbconf. I tried to set BB_PRESERVE_ENV, BB_ENV_WHITELIST and BB_ENV_EXTRAWHITE without any luck.
bitbake still crashes if HOME is used in bbconf.

--
Regards,
Ed

[PATCH] bitbake: Allow environment variables and ~ in bblayers

[Ed Bartosh]

Enabled processing of environment variables in bblayers paths using
bitbake apporved_variables API.

Enabled processing of ~ if HOME environment variable is approved.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/cookerdata.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py
index 6c11a60..05b3986 100644
--- a/bitbake/lib/bb/cookerdata.py
+++ b/bitbake/lib/bb/cookerdata.py
@@ -25,6 +25,7 @@
 import os, sys
 from functools import wraps
 import logging
+import string
 import bb
 from bb import data
 import bb.parse
@@ -269,8 +270,13 @@ class CookerDataBuilder(object):
             layers = (data.getVar('BBLAYERS', True) or "").split()
 
             data = bb.data.createCopy(data)
+            approved = dict((var, data.getVar(var, True)) \
+                                  for var in bb.utils.approved_variables())
             for layer in layers:
                 parselog.debug(2, "Adding layer %s", layer)
+                layer = string.Template(layer).safe_substitute(approved)
+                if 'HOME' in approved:
+                    layer = os.path.expanduser(layer)
                 data.setVar('LAYERDIR', layer)
                 data = parse_config_file(os.path.join(layer, "conf", "layer.conf"), data)
                 data.expandVarref('LAYERDIR')
-- 
2.1.4

Re: [PATCH] bitbake: Allow environment variables and ~ in bblayers

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1822 bytes --]

On Wed, Apr 15, 2015 at 7:43 AM, Ed Bartosh <ed.bartosh@linux.intel.com>
wrote:

> Enabled processing of environment variables in bblayers paths using
> bitbake apporved_variables API.
>
> Enabled processing of ~ if HOME environment variable is approved.
>
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  bitbake/lib/bb/cookerdata.py | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py
> index 6c11a60..05b3986 100644
> --- a/bitbake/lib/bb/cookerdata.py
> +++ b/bitbake/lib/bb/cookerdata.py
> @@ -25,6 +25,7 @@
>  import os, sys
>  from functools import wraps
>  import logging
> +import string
>  import bb
>  from bb import data
>  import bb.parse
> @@ -269,8 +270,13 @@ class CookerDataBuilder(object):
>              layers = (data.getVar('BBLAYERS', True) or "").split()
>
>              data = bb.data.createCopy(data)
> +            approved = dict((var, data.getVar(var, True)) \
> +                                  for var in bb
> .utils.approved_variables())
>              for layer in layers:
>                  parselog.debug(2, "Adding layer %s", layer)
> +                layer = string.Template(layer).safe_substitute(approved)
> +                if 'HOME' in approved:
> +                    layer = os.path.expanduser(layer)
>                  data.setVar('LAYERDIR', layer)
>                  data = parse_config_file(os.path.join(layer, "conf",
> "layer.conf"), data)
>                  data.expandVarref('LAYERDIR')
>

Why the use of string.Template here? We already have a variable expansion
mechanism.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2979 bytes --]

[PATCH] bitbake: Allow ~ in bblayers

[Ed Bartosh]

Implemented processing of ~ in bblayer's paths if HOME
environment variable is approved.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/cookerdata.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py
index 6c11a60..bf08c63 100644
--- a/bitbake/lib/bb/cookerdata.py
+++ b/bitbake/lib/bb/cookerdata.py
@@ -269,8 +269,11 @@ class CookerDataBuilder(object):
             layers = (data.getVar('BBLAYERS', True) or "").split()
 
             data = bb.data.createCopy(data)
+            approved = bb.utils.approved_variables()
             for layer in layers:
                 parselog.debug(2, "Adding layer %s", layer)
+                if 'HOME' in approved:
+                    layer = os.path.expanduser(layer)
                 data.setVar('LAYERDIR', layer)
                 data = parse_config_file(os.path.join(layer, "conf", "layer.conf"), data)
                 data.expandVarref('LAYERDIR')
-- 
2.1.4

Re: [PATCH] bitbake: Allow ~ in bblayers

[Richard Purdie]

On Thu, 2015-04-16 at 21:28 +0300, Ed Bartosh wrote:
> Implemented processing of ~ in bblayer's paths if HOME
> environment variable is approved.
> 
> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
> ---
>  bitbake/lib/bb/cookerdata.py | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py
> index 6c11a60..bf08c63 100644
> --- a/bitbake/lib/bb/cookerdata.py
> +++ b/bitbake/lib/bb/cookerdata.py
> @@ -269,8 +269,11 @@ class CookerDataBuilder(object):
>              layers = (data.getVar('BBLAYERS', True) or "").split()
>  
>              data = bb.data.createCopy(data)
> +            approved = bb.utils.approved_variables()
>              for layer in layers:
>                  parselog.debug(2, "Adding layer %s", layer)
> +                if 'HOME' in approved:

can we change this to 

if 'HOME' in approved and '~' in layer:

simply so that we can more easily remember what this code is doing in
the future?

Cheers,

Richard

> +                    layer = os.path.expanduser(layer)
>                  data.setVar('LAYERDIR', layer)
>                  data = parse_config_file(os.path.join(layer, "conf", "layer.conf"), data)
>                  data.expandVarref('LAYERDIR')
> -- 
> 2.1.4
> 

Re: [PATCH] bitbake: Allow ~ in bblayers

[Gary Thomas]

On 2015-04-17 04:41, Richard Purdie wrote:
> On Thu, 2015-04-16 at 21:28 +0300, Ed Bartosh wrote:
>> Implemented processing of ~ in bblayer's paths if HOME
>> environment variable is approved.
>>
>> Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
>> ---
>>   bitbake/lib/bb/cookerdata.py | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py
>> index 6c11a60..bf08c63 100644
>> --- a/bitbake/lib/bb/cookerdata.py
>> +++ b/bitbake/lib/bb/cookerdata.py
>> @@ -269,8 +269,11 @@ class CookerDataBuilder(object):
>>               layers = (data.getVar('BBLAYERS', True) or "").split()
>>
>>               data = bb.data.createCopy(data)
>> +            approved = bb.utils.approved_variables()
>>               for layer in layers:
>>                   parselog.debug(2, "Adding layer %s", layer)
>> +                if 'HOME' in approved:
>
> can we change this to
>
> if 'HOME' in approved and '~' in layer:
>
> simply so that we can more easily remember what this code is doing in
> the future?

+1

This change also has the side effect of not causing an expansion of
all layers if HOME is in the approved list which would seem a useful
optimization.

>
> Cheers,
>
> Richard
>
>> +                    layer = os.path.expanduser(layer)
>>                   data.setVar('LAYERDIR', layer)
>>                   data = parse_config_file(os.path.join(layer, "conf", "layer.conf"), data)
>>                   data.expandVarref('LAYERDIR')
>> --
>> 2.1.4
>>
>
>

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

[PATCH] bitbake: Allow ~ in bblayers

[Ed Bartosh]

Implemented processing of ~ in bblayer's paths if HOME
environment variable is approved.

Signed-off-by: Ed Bartosh <ed.bartosh@linux.intel.com>
---
 bitbake/lib/bb/cookerdata.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/bitbake/lib/bb/cookerdata.py b/bitbake/lib/bb/cookerdata.py
index 6c11a60..630ee27 100644
--- a/bitbake/lib/bb/cookerdata.py
+++ b/bitbake/lib/bb/cookerdata.py
@@ -269,8 +269,11 @@ class CookerDataBuilder(object):
             layers = (data.getVar('BBLAYERS', True) or "").split()
 
             data = bb.data.createCopy(data)
+            approved = bb.utils.approved_variables()
             for layer in layers:
                 parselog.debug(2, "Adding layer %s", layer)
+                if 'HOME' in approved and '~' in layer:
+                    layer = os.path.expanduser(layer)
                 data.setVar('LAYERDIR', layer)
                 data = parse_config_file(os.path.join(layer, "conf", "layer.conf"), data)
                 data.expandVarref('LAYERDIR')
-- 
2.1.4