[PATCH 0/2] populate_sdk_ext: two fixes

[PATCH 0/2] populate_sdk_ext: two fixes

[Chen Qi]

The following changes since commit 4dd4b96b6d60246338bb30ede9f3ab1b2e757be9:

  libxfont: Security Advisory - libxfont - CVE-2015-1804 (2015-04-28 07:56:01 +0100)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib ChenQi/populate_sdk_ext_2_fixes
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=ChenQi/populate_sdk_ext_2_fixes

Chen Qi (2):
  populate_sdk_ext: consider custom configuration in local.conf
  populate_sdk_ext: install the latest buildtools-tarball

 meta/classes/populate_sdk_ext.bbclass | 32 +++++++++++++++++++++++++++++++-
 1 file changed, 31 insertions(+), 1 deletion(-)

-- 
1.9.1

[PATCH 1/2] populate_sdk_ext: consider custom configuration in local.conf

[Chen Qi]

Copy the contents of local.conf under TOPDIR into the final generated
local.conf. In this way, custom settings are also made into the final
local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.

Before this change, installing extensible SDK would usually report failure
when preparing the build system if the user has custom configuration for
DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in local.conf
also don't get built correctly.

This patch solves the above problem.

A blacklist mechanism is also introduced so that we can blacklist variables that
should not be copied into the final local.conf file. Currently, the blacklist
contains 'SSTATE_DIR' and 'DL_DIR'.

[YOCTO #7616]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/populate_sdk_ext.bbclass | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index dc2c58e..7925ece 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = " ${SDK_TARGETS}"
 SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
 
 SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
+SDK_META_CONF_BLACKLIST ?= "DL_DIR SSTATE_DIR"
 
 SDK_TARGETS ?= "${PN}"
 OE_INIT_ENV_SCRIPT ?= "oe-init-build-env"
@@ -114,6 +115,25 @@ python copy_buildsystem () {
         f.write('# this configuration provides, it is strongly suggested that you set\n')
         f.write('# up a proper instance of the full build system and use that instead.\n\n')
 
+        # Copy configurations from the current local.conf
+        builddir = d.getVar('TOPDIR', True)
+        with open(builddir + '/conf/local.conf', 'r') as lf:
+            varblacklist = d.getVar('SDK_META_CONF_BLACKLIST', True).split()
+            skip = False
+            for line in lf:
+                line = line.lstrip()
+                if line.startswith('#'):
+                    continue
+                for varname in varblacklist:
+                    if line.startswith(varname):
+                        skip = True
+                        break
+                if not skip:
+                    f.write(line)
+                skip = False
+        f.write('\n')
+
+        # Configurations in local.conf which are specific for extensible SDK
         f.write('INHERIT += "%s"\n\n' % 'uninative')
         f.write('CONF_VERSION = "%s"\n\n' % d.getVar('CONF_VERSION'))
 
-- 
1.9.1

[PATCH 2/2] populate_sdk_ext: install the latest buildtools-tarball

[Chen Qi]

If we do `bitbake buildtools-tarball' and then after one day do `bitbake
core-image-minimal -c populate_sdk_ext', we would meet errors like below.

| install: cannot stat '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/
poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalone
-1.8+snapshot-20150429.sh': No such file or directory

The problem is that the output name for buildtools-tarball has ${DATE} in it.
So if populate_sdk_ext task is executed but buildtools-tarball is not rebuilt,
the above error appears.

Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the
install_tools() function, we should find the latest buildtools-tarball based
on the modification time and install it.

[YOCTO #7674]

Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
---
 meta/classes/populate_sdk_ext.bbclass | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 7925ece..0e4ee31 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -188,7 +188,17 @@ install_tools() {
 	ln -sr ${SDK_OUTPUT}/${SDKPATH}/${scriptrelpath}/recipetool ${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}/recipetool
 	touch ${SDK_OUTPUT}/${SDKPATH}/.devtoolbase
 
-	install ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKGARCH}-buildtools-nativesdk-standalone-${DISTRO_VERSION}.sh ${SDK_OUTPUT}/${SDKPATH}
+	# find latest buildtools-tarball and install it
+	maxseconds=0
+	buildtools_path=""
+	for f in `ls ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKGARCH}-buildtools-nativesdk-standalone-*.sh`; do
+		seconds=`stat -c "%Y" $f`
+		if [ $seconds -gt $maxseconds ]; then
+			maxseconds=$seconds
+			buildtools_path="$f"
+		fi
+	done
+	install $buildtools_path ${SDK_OUTPUT}/${SDKPATH}
 
 	install ${SDK_DEPLOY}/${BUILD_ARCH}-nativesdk-libc.tar.bz2 ${SDK_OUTPUT}/${SDKPATH}
 }
-- 
1.9.1

Re: [PATCH 1/2] populate_sdk_ext: consider custom configuration in local.conf

[Paul Eggleton]

Hi Qi,

On Wednesday 29 April 2015 16:08:36 Chen Qi wrote:
> Copy the contents of local.conf under TOPDIR into the final generated
> local.conf. In this way, custom settings are also made into the final
> local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.
> 
> Before this change, installing extensible SDK would usually report failure
> when preparing the build system if the user has custom configuration for
> DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in
> local.conf also don't get built correctly.
> 
> This patch solves the above problem.
> 
> A blacklist mechanism is also introduced so that we can blacklist variables
> that should not be copied into the final local.conf file. Currently, the
> blacklist contains 'SSTATE_DIR' and 'DL_DIR'.
> 
> [YOCTO #7616]
> 
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  meta/classes/populate_sdk_ext.bbclass | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> diff --git a/meta/classes/populate_sdk_ext.bbclass
> b/meta/classes/populate_sdk_ext.bbclass index dc2c58e..7925ece 100644
> --- a/meta/classes/populate_sdk_ext.bbclass
> +++ b/meta/classes/populate_sdk_ext.bbclass
> @@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = "
> ${SDK_TARGETS}" SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
> 
>  SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
> +SDK_META_CONF_BLACKLIST ?= "DL_DIR SSTATE_DIR"

I'm not totally opposed to this approach but we are going to have to get the 
default list right if we are going to do it - e.g. you've missed the fairly 
obvious TMPDIR. There are quite a large number of other variables the user 
might set that should be blacklisted - potentially any variable that points to 
a location where the build system would write files (DEPLOY_DIR, DEPLOY_DIR_*, 
WORKDIR, T, S, B, ...). We might go a step further and use the data dictionary 
to check if the value evaluates to somewhere outside of ${TMPDIR} (if we first 
make a copy of d and set TMPDIR to a dummy value); this way we would allow 
adjustments within TMPDIR but not outside of it.

My other concern (perhaps a pet issue of mine) is that people are far too 
reliant on putting things in local.conf that they really shouldn't. On the 
other hand building a potentially broken SDK or at least one that doesn't 
match up with the image is worse, but I'm still concerned about the way 
local.conf is often abused.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 2/2] populate_sdk_ext: install the latest buildtools-tarball

[Paul Eggleton]

On Wednesday 29 April 2015 16:08:37 Chen Qi wrote:
> If we do `bitbake buildtools-tarball' and then after one day do `bitbake
> core-image-minimal -c populate_sdk_ext', we would meet errors like below.
> 
> | install: cannot stat
> | '/buildarea2/chenqi/poky/build-systemd/tmp/deploy/sdk/
> 
> poky-glibc-x86_64-buildtools-tarball-core2-64-buildtools-nativesdk-standalon
> e -1.8+snapshot-20150429.sh': No such file or directory
> 
> The problem is that the output name for buildtools-tarball has ${DATE} in
> it. So if populate_sdk_ext task is executed but buildtools-tarball is not
> rebuilt, the above error appears.
> 
> Instead of hardcoding ${DISTRO_VERSION} which consists of ${DATE} in the
> install_tools() function, we should find the latest buildtools-tarball based
> on the modification time and install it.
> 
> [YOCTO #7674]
> 
> Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> ---
>  meta/classes/populate_sdk_ext.bbclass | 12 +++++++++++-
>  1 file changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/meta/classes/populate_sdk_ext.bbclass
> b/meta/classes/populate_sdk_ext.bbclass index 7925ece..0e4ee31 100644
> --- a/meta/classes/populate_sdk_ext.bbclass
> +++ b/meta/classes/populate_sdk_ext.bbclass
> @@ -188,7 +188,17 @@ install_tools() {
>  	ln -sr ${SDK_OUTPUT}/${SDKPATH}/${scriptrelpath}/recipetool
> ${SDK_OUTPUT}/${SDKPATHNATIVE}${bindir_nativesdk}/recipetool touch
> ${SDK_OUTPUT}/${SDKPATH}/.devtoolbase
> 
> -	install
> ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKG
> ARCH}-buildtools-nativesdk-standalone-${DISTRO_VERSION}.sh
> ${SDK_OUTPUT}/${SDKPATH} +	# find latest buildtools-tarball and install it
> +	maxseconds=0
> +	buildtools_path=""
> +	for f in `ls
> ${SDK_DEPLOY}/${DISTRO}-${TCLIBC}-${SDK_ARCH}-buildtools-tarball-${TUNE_PKG
> ARCH}-buildtools-nativesdk-standalone-*.sh`; do +		seconds=`stat -c 
"%Y" $f`
> +		if [ $seconds -gt $maxseconds ]; then
> +			maxseconds=$seconds
> +			buildtools_path="$f"
> +		fi
> +	done
> +	install $buildtools_path ${SDK_OUTPUT}/${SDKPATH}

Surely it would be simpler to just `ls -t ... | head -n1` ?

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 1/2] populate_sdk_ext: consider custom configuration in local.conf

[Richard Purdie]

On Wed, 2015-04-29 at 09:54 +0100, Paul Eggleton wrote:
> Hi Qi,
> 
> On Wednesday 29 April 2015 16:08:36 Chen Qi wrote:
> > Copy the contents of local.conf under TOPDIR into the final generated
> > local.conf. In this way, custom settings are also made into the final
> > local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx, etc.
> > 
> > Before this change, installing extensible SDK would usually report failure
> > when preparing the build system if the user has custom configuration for
> > DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in
> > local.conf also don't get built correctly.
> > 
> > This patch solves the above problem.
> > 
> > A blacklist mechanism is also introduced so that we can blacklist variables
> > that should not be copied into the final local.conf file. Currently, the
> > blacklist contains 'SSTATE_DIR' and 'DL_DIR'.
> > 
> > [YOCTO #7616]
> > 
> > Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> > ---
> >  meta/classes/populate_sdk_ext.bbclass | 20 ++++++++++++++++++++
> >  1 file changed, 20 insertions(+)
> > 
> > diff --git a/meta/classes/populate_sdk_ext.bbclass
> > b/meta/classes/populate_sdk_ext.bbclass index dc2c58e..7925ece 100644
> > --- a/meta/classes/populate_sdk_ext.bbclass
> > +++ b/meta/classes/populate_sdk_ext.bbclass
> > @@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = "
> > ${SDK_TARGETS}" SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
> > 
> >  SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
> > +SDK_META_CONF_BLACKLIST ?= "DL_DIR SSTATE_DIR"
> 
> I'm not totally opposed to this approach but we are going to have to get the 
> default list right if we are going to do it - e.g. you've missed the fairly 
> obvious TMPDIR. There are quite a large number of other variables the user 
> might set that should be blacklisted - potentially any variable that points to 
> a location where the build system would write files (DEPLOY_DIR, DEPLOY_DIR_*, 
> WORKDIR, T, S, B, ...). We might go a step further and use the data dictionary 
> to check if the value evaluates to somewhere outside of ${TMPDIR} (if we first 
> make a copy of d and set TMPDIR to a dummy value); this way we would allow 
> adjustments within TMPDIR but not outside of it.
> 
> My other concern (perhaps a pet issue of mine) is that people are far too 
> reliant on putting things in local.conf that they really shouldn't. On the 
> other hand building a potentially broken SDK or at least one that doesn't 
> match up with the image is worse, but I'm still concerned about the way 
> local.conf is often abused.

One thing we could do programatically is to build the bash config hash
with and without local.conf. If they match, local.conf isn't influencing
the build checksums. If they don't match, local.conf has something
problematic in it and we could error and tell the user to fix it?

This would also cover site.conf, auto.conf and the other conf files we
can detect.

Cheers,

Richard

Re: [PATCH 1/2] populate_sdk_ext: consider custom configuration in local.conf

[Paul Eggleton]

On Wednesday 29 April 2015 09:59:37 Richard Purdie wrote:
> On Wed, 2015-04-29 at 09:54 +0100, Paul Eggleton wrote:
> > On Wednesday 29 April 2015 16:08:36 Chen Qi wrote:
> > > Copy the contents of local.conf under TOPDIR into the final generated
> > > local.conf. In this way, custom settings are also made into the final
> > > local.conf like IMAGE_INSTALL, DISTRO_FEATURES, VIRTUAL-RUNTIME_xxx,
> > > etc.
> > > 
> > > Before this change, installing extensible SDK would usually report
> > > failure
> > > when preparing the build system if the user has custom configuration for
> > > DISTRO_FEATURES in local.conf. Also, items in IMAGE_INSTALL_append in
> > > local.conf also don't get built correctly.
> > > 
> > > This patch solves the above problem.
> > > 
> > > A blacklist mechanism is also introduced so that we can blacklist
> > > variables
> > > that should not be copied into the final local.conf file. Currently, the
> > > blacklist contains 'SSTATE_DIR' and 'DL_DIR'.
> > > 
> > > [YOCTO #7616]
> > > 
> > > Signed-off-by: Chen Qi <Qi.Chen@windriver.com>
> > > ---
> > > 
> > >  meta/classes/populate_sdk_ext.bbclass | 20 ++++++++++++++++++++
> > >  1 file changed, 20 insertions(+)
> > > 
> > > diff --git a/meta/classes/populate_sdk_ext.bbclass
> > > b/meta/classes/populate_sdk_ext.bbclass index dc2c58e..7925ece 100644
> > > --- a/meta/classes/populate_sdk_ext.bbclass
> > > +++ b/meta/classes/populate_sdk_ext.bbclass
> > > @@ -16,6 +16,7 @@ SDK_RDEPENDS_append_task-populate-sdk-ext = "
> > > ${SDK_TARGETS}" SDK_RELOCATE_AFTER_INSTALL_task-populate-sdk-ext = "0"
> > > 
> > >  SDK_META_CONF_WHITELIST ?= "MACHINE DISTRO PACKAGE_CLASSES"
> > > 
> > > +SDK_META_CONF_BLACKLIST ?= "DL_DIR SSTATE_DIR"
> > 
> > I'm not totally opposed to this approach but we are going to have to get
> > the default list right if we are going to do it - e.g. you've missed the
> > fairly obvious TMPDIR. There are quite a large number of other variables
> > the user might set that should be blacklisted - potentially any variable
> > that points to a location where the build system would write files
> > (DEPLOY_DIR, DEPLOY_DIR_*, WORKDIR, T, S, B, ...). We might go a step
> > further and use the data dictionary to check if the value evaluates to
> > somewhere outside of ${TMPDIR} (if we first make a copy of d and set
> > TMPDIR to a dummy value); this way we would allow adjustments within
> > TMPDIR but not outside of it.
> > 
> > My other concern (perhaps a pet issue of mine) is that people are far too
> > reliant on putting things in local.conf that they really shouldn't. On the
> > other hand building a potentially broken SDK or at least one that doesn't
> > match up with the image is worse, but I'm still concerned about the way
> > local.conf is often abused.
> 
> One thing we could do programatically is to build the bash config hash
> with and without local.conf. If they match, local.conf isn't influencing
> the build checksums. If they don't match, local.conf has something
> problematic in it and we could error and tell the user to fix it?
> 
> This would also cover site.conf, auto.conf and the other conf files we
> can detect.

Hmm, yes... this might be another check we could turn on when switching from 
development -> production (what I mentioned at OEDAM).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre