[PATCH 0/3] Extensible SDK: 3 fixes

[PATCH 0/3] Extensible SDK: 3 fixes

[Chen Qi]

The first two have been sent before.

The following changes since commit 7ffe10df73cc20d10fcd41b121074445273bd60e:

  license_class: license_create_manifest improvment (2015-05-09 22:26:02 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib ChenQi/ext-sdk-3-fixes
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=ChenQi/ext-sdk-3-fixes

Chen Qi (3):
  populate_sdk_ext: install the latest buildtools-tarball
  populate_sdk_ext: consider custom configuration in local.conf
  copy_buildsystem: make sure bitbake directory is copied

 meta/classes/populate_sdk_ext.bbclass | 27 ++++++++++++++++++++++++++-
 meta/lib/oe/copy_buildsystem.py       |  7 +++----
 2 files changed, 29 insertions(+), 5 deletions(-)

-- 
1.9.1

[PATCH 1/3] populate_sdk_ext: install the latest buildtools-tarball

[Chen Qi]

If we do `bitbake buildtools-tarball' and then after one day do `bitbake
core-image-minimal -c populate_sdk_ext', we would meet errors like below.

| install: cannot stat '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/
poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone
-1.8+snapshot-20150429.sh': No such file or directory

The problem is that the output name for buildtools-tarball has ${DATE} in it.
So if populate_sdk_ext task is executed but buildtools-tarball is not rebuilt,
the above error appears.

Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the
install_tools() function, we should find the latest buildtools-tarball based
on the modification time and install it.

[YOCTO #7674]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/populate_sdk_ext.bbclass | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index dc2c58e..2fc4c11 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -168,7 +168,9 @@ install_tools() {
 	ln -sr ${SDK_OUTPUT}/${SDKPATH}/${scriptrelpath}/recipetool ${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}/recipetool
 	touch ${SDK_OUTPUT}/${SDKPATH}/.devtoolbase
 
-	install ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKGARCH}-buildtools-nativesdk-standalone-${DISTRO_VERSION}.sh ${SDK_OUTPUT}/${SDKPATH}
+	# find latest buildtools-tarball and install it
+	buildtools_path=`ls -t1 ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKGARCH}-buildtools-nativesdk-standalone-*.sh | head -n1`
+	install $buildtools_path ${SDK_OUTPUT}/${SDKPATH}
 
 	install ${SDK_DEPLOY}/${BUILD_ARCH}-nativesdk-libc.tar.bz2 ${SDK_OUTPUT}/${SDKPATH}
 }
-- 
1.9.1

[PATCH 2/3] populate_sdk_ext: consider custom configuration in local.conf

[Chen Qi]

Copy the contents of local.conf under TOPDIR into the final generated
local.conf. In this way, custom settings are also made into the final
local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.

Before this change, installing extensible SDK would usually report failure
when preparing the build system if the user has custom configuration for
DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in local.conf
also don't get built correctly.

This patch solves the above problem.

A blacklist mechanism is also introduced so that we can blacklist variables that
should not be copied into the final local.conf file. Currently, the blacklist
contains 'TMPDIR', 'SSTATE_DIR', 'DL_DIR', 'STAMPS_DIR', 'BASE_WORKDIR' and
'DEPLOY_DIR'. What these variables have in common is that they are set in
bitbake.conf using '?=' or '??='.

In addition, we check to avoid any setting that might lead to host path
bleeding into SDK's configuration.

[YOCTO #7616]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/populate_sdk_ext.bbclass | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 2fc4c11..49ba26b 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = " ${SDK_TARGETS}"
 SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
 
 SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
+SDK_META_CONF_BLACKLIST ?= "TMPDIR DL_DIR SSTATE_DIR STAMPS_DIR BASE_WORKDIR DEPLOY_DIR"
 
 SDK_TARGETS ?= "${PN}"
 OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"
@@ -114,6 +115,28 @@ python copy_buildsystem () {
         f.write('# this configuration provides, it is strongly suggested that you set\n')
         f.write('# up a proper instance of the full build system and use that instead.\n\n')
 
+        # Copy configurations from the current local.conf
+        builddir = d.getVar('TOPDIR', True)
+        with open(builddir + '/conf/local.conf', 'r') as lf:
+            varblacklist = d.getVar('SDK_META_CONF_BLACKLIST', True).split()
+            skip = False
+            for line in lf:
+                line = line.lstrip()
+                if line.startswith('#'):
+                    continue
+                # avoid host path bleeding into SDK configuration
+                if line.find('"/') != -1:
+                    continue
+                for varname in varblacklist:
+                    if line.startswith(varname):
+                        skip = True
+                        break
+                if not skip:
+                    f.write(line)
+                skip = False
+        f.write('\n')
+
+        # Configurations in local.conf which are specific for extensible SDK
         f.write('INHERIT += "%s"\n\n' % 'uninative')
         f.write('CONF_VERSION = "%s"\n\n' % d.getVar('CONF_VERSION'))
 
-- 
1.9.1

[PATCH 3/3] copy_buildsystem: make sure bitbake directory is copied

[Chen Qi]

The previous code assumes that bitbake/ directory is under the core layer.
This is the case for Yocto project. But users might clone oe-core and bitbake
separately. So we use bb.__file__ to locate the bitbake directory to make sure
it's copied into the extensible SDK.

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/lib/oe/copy_buildsystem.py | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/meta/lib/oe/copy_buildsystem.py b/meta/lib/oe/copy_buildsystem.py
index cf7fada..979578c 100644
--- a/meta/lib/oe/copy_buildsystem.py
+++ b/meta/lib/oe/copy_buildsystem.py
@@ -28,11 +28,10 @@ class BuildSystem(object):
         layers.append(corebase)
 
         corebase_files = self.d.getVar('COREBASE_FILES', True).split()
-
-        # bitbake belongs in corebase so make sure it goes there
-        if "bitbake" not in corebase_files:
-            corebase_files.append("bitbake")
         corebase_files = [corebase + '/' +x for x in corebase_files]
+        # Make sure bitbake goes in
+        bitbake_dir = bb.__file__.rsplit('/', 3)[0]
+        corebase_files.append(bitbake_dir)
 
         for layer in layers:
             layerconf = os.path.join(layer, 'conf', 'layer.conf')
-- 
1.9.1

Re: [PATCH 1/3] populate_sdk_ext: install the latest buildtools-tarball

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1277 bytes --]

On Tue, May 26, 2015 at 11:09 PM, Chen Qi <Qi.Chen@windriver.com> wrote:

> If we do `bitbake buildtools-tarball' and then after one day do `bitbake
> core-image-minimal -c populate_sdk_ext', we would meet errors like below.
>
> | install: cannot stat
> '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/
>
> poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone
> -1.8+snapshot-20150429.sh': No such file or directory
>
> The problem is that the output name for buildtools-tarball has ${DATE} in
> it.
> So if populate_sdk_ext task is executed but buildtools-tarball is not
> rebuilt,
> the above error appears.
>
> Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the
> install_tools() function, we should find the latest buildtools-tarball
> based
> on the modification time and install it.
>
> [YOCTO #7674]
>
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>

Hmm, if buildtools-tarball is required, why not just pull it in via
dependencies and build a current one, rather than assuming/forcing the user
to have done so themselves?
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1803 bytes --]

Re: [PATCH 1/3] populate_sdk_ext: install the latest buildtools-tarball

[Randy Witt]

On 05/27/2015 11:54 AM, Christopher Larson wrote:
> On Tue, May 26, 2015 at 11:09 PM, Chen Qi <Qi.Chen@windriver.com> wrote:
>
>> If we do `bitbake buildtools-tarball' and then after one day do `bitbake
>> core-image-minimal -c populate_sdk_ext', we would meet errors like below.
>>
>> | install: cannot stat
>> '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/
>>
>> poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone
>> -1.8+snapshot-20150429.sh': No such file or directory
>>
>> The problem is that the output name for buildtools-tarball has ${DATE} in
>> it.
>> So if populate_sdk_ext task is executed but buildtools-tarball is not
>> rebuilt,
>> the above error appears.
>>
>> Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the
>> install_tools() function, we should find the latest buildtools-tarball
>> based
>> on the modification time and install it.
>>
>> [YOCTO #7674]
>>
>> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>>
>
> Hmm, if buildtools-tarball is required, why not just pull it in via
> dependencies and build a current one, rather than assuming/forcing the user
> to have done so themselves?

It is a dependency and will be built if it hasn't been built. But since

TOOLCHAIN_OUTPUTNAME ?= 
"${SDK_NAME}-buildtools-nativesdk-standalone-${DISTRO_VERSION}"

in buildtools-tarball.bb the filename will be different each time we try to copy 
it, unless we force buildtools-tarball to be rebuilt each time the sdk is 
constructed.

${DATE} being used in DISTRO_VERSION is misleading in my opinion because 
different dates don't necessarily mean different contents. So we could change 
DISTRO_VERSION to not use date, or buildtools-tarball.bb to not use DISTRO_VERSION.

>
>

Re: [PATCH 1/3] populate_sdk_ext: install the latest buildtools-tarball

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 2492 bytes --]

On Wed, May 27, 2015 at 2:26 PM, Randy Witt <randy.e.witt@linux.intel.com>
wrote:

> On 05/27/2015 11:54 AM, Christopher Larson wrote:
>
>> On Tue, May 26, 2015 at 11:09 PM, Chen Qi <Qi.Chen@windriver.com> wrote:
>>
>>  If we do `bitbake buildtools-tarball' and then after one day do `bitbake
>>> core-image-minimal -c populate_sdk_ext', we would meet errors like below.
>>>
>>> | install: cannot stat
>>> '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/
>>>
>>>
>>> poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone
>>> -1.8+snapshot-20150429.sh': No such file or directory
>>>
>>> The problem is that the output name for buildtools-tarball has ${DATE} in
>>> it.
>>> So if populate_sdk_ext task is executed but buildtools-tarball is not
>>> rebuilt,
>>> the above error appears.
>>>
>>> Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the
>>> install_tools() function, we should find the latest buildtools-tarball
>>> based
>>> on the modification time and install it.
>>>
>>> [YOCTO #7674]
>>>
>>> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>>>
>>>
>> Hmm, if buildtools-tarball is required, why not just pull it in via
>> dependencies and build a current one, rather than assuming/forcing the
>> user
>> to have done so themselves?
>>
>
> It is a dependency and will be built if it hasn't been built. But since
>
> TOOLCHAIN_OUTPUTNAME ?=
> "${SDK_NAME}-buildtools-nativesdk-standalone-${DISTRO_VERSION}"
>
> in buildtools-tarball.bb the filename will be different each time we try
> to copy it, unless we force buildtools-tarball to be rebuilt each time the
> sdk is constructed.
>
> ${DATE} being used in DISTRO_VERSION is misleading in my opinion because
> different dates don't necessarily mean different contents. So we could
> change DISTRO_VERSION to not use date, or buildtools-tarball.bb to not
> use DISTRO_VERSION.


I expect it’d be good if in this case DATE wasn’t whitelisted, so recipes
which use a DISTRO_VERSION which includes a date get rebuilt for a new day,
and if you don’t want that, you can remove DATE from DISTRO_VERSION.

Actually, it’d be interesting to try to rig some form of auto-incrementing
distro version based on distro metadata checksum changes or something.. Hmm.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 3434 bytes --]

Re: [PATCH 2/3] populate_sdk_ext: consider custom configuration in local.conf

[Paul Eggleton]

Hi Qi,

On Wednesday 27 May 2015 14:09:39 Chen Qi wrote:
> Copy the contents of local.conf under TOPDIR into the final generated
> local.conf. In this way, custom settings are also made into the final
> local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.
> 
> Before this change, installing extensible SDK would usually report failure
> when preparing the build system if the user has custom configuration for
> DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in
> local.conf also don't get built correctly.
> 
> This patch solves the above problem.
> 
> A blacklist mechanism is also introduced so that we can blacklist variables
> that should not be copied into the final local.conf file. Currently, the
> blacklist contains 'TMPDIR', 'SSTATE_DIR', 'DL_DIR', 'STAMPS_DIR',
> 'BASE_WORKDIR' and 'DEPLOY_DIR'. What these variables have in common is
> that they are set in bitbake.conf using '?=' or '??='.
> 
> In addition, we check to avoid any setting that might lead to host path
> bleeding into SDK's configuration.
> 
> [YOCTO #7616]
> 
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  meta/classes/populate_sdk_ext.bbclass | 23 +++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
> 
> diff --git a/meta/classes/populate_sdk_ext.bbclass
> b/meta/classes/populate_sdk_ext.bbclass index 2fc4c11..49ba26b 100644
> --- a/meta/classes/populate_sdk_ext.bbclass
> +++ b/meta/classes/populate_sdk_ext.bbclass
> @@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = "
> ${SDK_TARGETS}" SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
> 
>  SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
> +SDK_META_CONF_BLACKLIST ?= "TMPDIR DL_DIR SSTATE_DIR STAMPS_DIR
> BASE_WORKDIR DEPLOY_DIR"
> 
>  SDK_TARGETS ?= "${PN}"
>  OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"
> @@ -114,6 +115,28 @@ python copy_buildsystem () {
>          f.write('# this configuration provides, it is strongly suggested
> that you set\n') f.write('# up a proper instance of the full build system
> and use that instead.\n\n')
> 
> +        # Copy configurations from the current local.conf
> +        builddir = d.getVar('TOPDIR', True)
> +        with open(builddir + '/conf/local.conf', 'r') as lf:
> +            varblacklist = d.getVar('SDK_META_CONF_BLACKLIST',
> True).split() +            skip = False
> +            for line in lf:
> +                line = line.lstrip()
> +                if line.startswith('#'):
> +                    continue
> +                # avoid host path bleeding into SDK configuration
> +                if line.find('"/') != -1:
> +                    continue
> +                for varname in varblacklist:
> +                    if line.startswith(varname):
> +                        skip = True
> +                        break
> +                if not skip:
> +                    f.write(line)
> +                skip = False
> +        f.write('\n')
> +
> +        # Configurations in local.conf which are specific for extensible
> SDK f.write('INHERIT += "%s"\n\n' % 'uninative')
>          f.write('CONF_VERSION = "%s"\n\n' % d.getVar('CONF_VERSION'))

I've been thinking about this; since I've recently added edit_metadata() in 
bitbake's lib/bb/utils.py, it would make sense to use that rather than adding 
another piece of code that effectively parses bitbake configuration files (though 
I know this isn't adding much). I know this series has been going back and 
forth for a while now and the delays have largely been my fault, so I'd be 
happy to make this change and resend the series - it's up to you.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 2/3] populate_sdk_ext: consider custom configuration in local.conf

[ChenQi]

On 06/04/2015 10:11 PM, Paul Eggleton wrote:
> Hi Qi,
>
> On Wednesday 27 May 2015 14:09:39 Chen Qi wrote:
>> Copy the contents of local.conf under TOPDIR into the final generated
>> local.conf. In this way, custom settings are also made into the final
>> local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.
>>
>> Before this change, installing extensible SDK would usually report failure
>> when preparing the build system if the user has custom configuration for
>> DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in
>> local.conf also don't get built correctly.
>>
>> This patch solves the above problem.
>>
>> A blacklist mechanism is also introduced so that we can blacklist variables
>> that should not be copied into the final local.conf file. Currently, the
>> blacklist contains 'TMPDIR', 'SSTATE_DIR', 'DL_DIR', 'STAMPS_DIR',
>> 'BASE_WORKDIR' and 'DEPLOY_DIR'. What these variables have in common is
>> that they are set in bitbake.conf using '?=' or '??='.
>>
>> In addition, we check to avoid any setting that might lead to host path
>> bleeding into SDK's configuration.
>>
>> [YOCTO #7616]
>>
>> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
>> ---
>>   meta/classes/populate_sdk_ext.bbclass | 23 +++++++++++++++++++++++
>>   1 file changed, 23 insertions(+)
>>
>> diff --git a/meta/classes/populate_sdk_ext.bbclass
>> b/meta/classes/populate_sdk_ext.bbclass index 2fc4c11..49ba26b 100644
>> --- a/meta/classes/populate_sdk_ext.bbclass
>> +++ b/meta/classes/populate_sdk_ext.bbclass
>> @@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = "
>> ${SDK_TARGETS}" SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
>>
>>   SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
>> +SDK_META_CONF_BLACKLIST ?= "TMPDIR DL_DIR SSTATE_DIR STAMPS_DIR
>> BASE_WORKDIR DEPLOY_DIR"
>>
>>   SDK_TARGETS ?= "${PN}"
>>   OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"
>> @@ -114,6 +115,28 @@ python copy_buildsystem () {
>>           f.write('# this configuration provides, it is strongly suggested
>> that you set\n') f.write('# up a proper instance of the full build system
>> and use that instead.\n\n')
>>
>> +        # Copy configurations from the current local.conf
>> +        builddir = d.getVar('TOPDIR', True)
>> +        with open(builddir + '/conf/local.conf', 'r') as lf:
>> +            varblacklist = d.getVar('SDK_META_CONF_BLACKLIST',
>> True).split() +            skip = False
>> +            for line in lf:
>> +                line = line.lstrip()
>> +                if line.startswith('#'):
>> +                    continue
>> +                # avoid host path bleeding into SDK configuration
>> +                if line.find('"/') != -1:
>> +                    continue
>> +                for varname in varblacklist:
>> +                    if line.startswith(varname):
>> +                        skip = True
>> +                        break
>> +                if not skip:
>> +                    f.write(line)
>> +                skip = False
>> +        f.write('\n')
>> +
>> +        # Configurations in local.conf which are specific for extensible
>> SDK f.write('INHERIT += "%s"\n\n' % 'uninative')
>>           f.write('CONF_VERSION = "%s"\n\n' % d.getVar('CONF_VERSION'))
> I've been thinking about this; since I've recently added edit_metadata() in
> bitbake's lib/bb/utils.py, it would make sense to use that rather than adding
> another piece of code that effectively parses bitbake configuration files (though
> I know this isn't adding much). I know this series has been going back and
> forth for a while now and the delays have largely been my fault, so I'd be
> happy to make this change and resend the series - it's up to you.
>
> Cheers,
> Paul
>

Hi Paul,

Thanks for your suggestion.
I've been looking into edit_metadata(). But it seems that it's used to 
modify *specific* variables.
In this case, we don't know which variables we are going to skip in 
local.conf. We just know that they start with '/'. So I don't see how to 
do it.
I'm not sure if I understand things wrong. Please point it out directly 
if I've missed something.

Best Regards,
Chen Qi

Re: [PATCH 2/3] populate_sdk_ext: consider custom configuration in local.conf

[Paul Eggleton]

On Tuesday 09 June 2015 14:41:53 ChenQi wrote:
> On 06/04/2015 10:11 PM, Paul Eggleton wrote:
> > On Wednesday 27 May 2015 14:09:39 Chen Qi wrote:
> >> Copy the contents of local.conf under TOPDIR into the final generated
> >> local.conf. In this way, custom settings are also made into the final
> >> local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.
> >> 
> >> Before this change, installing extensible SDK would usually report
> >> failure
> >> when preparing the build system if the user has custom configuration for
> >> DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in
> >> local.conf also don't get built correctly.
> >> 
> >> This patch solves the above problem.
> >> 
> >> A blacklist mechanism is also introduced so that we can blacklist
> >> variables
> >> that should not be copied into the final local.conf file. Currently, the
> >> blacklist contains 'TMPDIR', 'SSTATE_DIR', 'DL_DIR', 'STAMPS_DIR',
> >> 'BASE_WORKDIR' and 'DEPLOY_DIR'. What these variables have in common is
> >> that they are set in bitbake.conf using '?=' or '??='.
> >> 
> >> In addition, we check to avoid any setting that might lead to host path
> >> bleeding into SDK's configuration.
> >> 
> >> [YOCTO #7616]
> >> 
> >> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> >> ---
> >> 
> >>   meta/classes/populate_sdk_ext.bbclass | 23 +++++++++++++++++++++++
> >>   1 file changed, 23 insertions(+)
> >> 
> >> diff --git a/meta/classes/populate_sdk_ext.bbclass
> >> b/meta/classes/populate_sdk_ext.bbclass index 2fc4c11..49ba26b 100644
> >> --- a/meta/classes/populate_sdk_ext.bbclass
> >> +++ b/meta/classes/populate_sdk_ext.bbclass
> >> @@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = "
> >> ${SDK_TARGETS}" SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
> >> 
> >>   SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
> >> 
> >> +SDK_META_CONF_BLACKLIST ?= "TMPDIR DL_DIR SSTATE_DIR STAMPS_DIR
> >> BASE_WORKDIR DEPLOY_DIR"
> >> 
> >>   SDK_TARGETS ?= "${PN}"
> >>   OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"
> >> 
> >> @@ -114,6 +115,28 @@ python copy_buildsystem () {
> >> 
> >>           f.write('# this configuration provides, it is strongly
> >>           suggested
> >> 
> >> that you set\n') f.write('# up a proper instance of the full build system
> >> and use that instead.\n\n')
> >> 
> >> +        # Copy configurations from the current local.conf
> >> +        builddir = d.getVar('TOPDIR', True)
> >> +        with open(builddir + '/conf/local.conf', 'r') as lf:
> >> +            varblacklist = d.getVar('SDK_META_CONF_BLACKLIST',
> >> True).split() +            skip = False
> >> +            for line in lf:
> >> +                line = line.lstrip()
> >> +                if line.startswith('#'):
> >> +                    continue
> >> +                # avoid host path bleeding into SDK configuration
> >> +                if line.find('"/') != -1:
> >> +                    continue
> >> +                for varname in varblacklist:
> >> +                    if line.startswith(varname):
> >> +                        skip = True
> >> +                        break
> >> +                if not skip:
> >> +                    f.write(line)
> >> +                skip = False
> >> +        f.write('\n')
> >> +
> >> +        # Configurations in local.conf which are specific for extensible
> >> SDK f.write('INHERIT += "%s"\n\n' % 'uninative')
> >> 
> >>           f.write('CONF_VERSION = "%s"\n\n' % d.getVar('CONF_VERSION'))
> > 
> > I've been thinking about this; since I've recently added edit_metadata()
> > in
> > bitbake's lib/bb/utils.py, it would make sense to use that rather than
> > adding another piece of code that effectively parses bitbake
> > configuration files (though I know this isn't adding much). I know this
> > series has been going back and forth for a while now and the delays have
> > largely been my fault, so I'd be happy to make this change and resend the
> > series - it's up to you.
> > 
> > Cheers,
> > Paul
> 
> Hi Paul,
> 
> Thanks for your suggestion.
> I've been looking into edit_metadata(). But it seems that it's used to
> modify *specific* variables.
> In this case, we don't know which variables we are going to skip in
> local.conf. We just know that they start with '/'. So I don't see how to
> do it.

Well, what you pass in actually goes into a regex, so you can just match
any variable with it - you just need to take care that it doesn't match
characters it shouldn't. Here's an example I just tested - you can place
it in scripts/ in order to try it out:

------------- snip ------------
import sys
import os

scripts_path = os.path.dirname(os.path.realpath(__file__))
lib_path = scripts_path + '/lib'
sys.path = sys.path + [lib_path]

import scriptpath
bitbakepath = scriptpath.add_bitbake_lib_path()

import bb.utils

whitelist = ['BBMASK']
def handle_var(varname, origvalue, op, newlines):
    print '%s "%s"' % (varname, origvalue)
    if '/' in origvalue and (not ':/' in value or varname in whitelist):
        newlines.append('# Removed original setting of %s\n' % varname)
        return None, op, 0, True
    else:
        return origvalue, op, 0, True

varlist = ['[^#=+ ]*']
with open('conf/local.conf', 'r') as f:
    oldlines = f.readlines()
(updated, newlines) = bb.utils.edit_metadata(oldlines, varlist, handle_var)

with open('conf/local.conf.newedit', 'w') as f:
    for line in newlines:
        f.write(line)

------------- snip ------------

We really ought not to have to care about matching #, that's probably
a bug in edit_metadata().  While looking at my own local.conf I noticed that
BBMASK might have / in it and we'd probably want to allow it through; I
don't know if there are other reasonable examples. I suspect we ought
to have a variable for this to allow for some flexibility. As in the example
we probably also want to default to allowing variable values that contain
URLs as well.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre