[PATCH v2 0/5] Extensible SDK fixes

[PATCH v2 0/5] Extensible SDK fixes

[Paul Eggleton]

Changes since v1:
 * Simplify "prevent do_rootfs from executing on install" fix to avoid
   creating a class that we don't need - just put the stubbing out of
   do_rootfs in each bbappend file.
 * Print a warning within the env setup script if the OE env setup
   script has previously been sourced, because things won't work as
   you might expect if that's been done.

The following changes since commit 10e5df3503632a6e1c54612055b19f7258c3ae2f:

  lib/oe/image.py: Fix dependency handling for compressed types (2015-10-14 18:08:22 +0300)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib paule/extsdkfixes2
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/extsdkfixes2

Paul Eggleton (5):
  classes/populate_sdk_ext: consistent indentation
  classes/populate_sdk_ext: prevent do_rootfs from executing on install
  classes/populate_sdk_ext: add note to env setup script
  classes/populate_sdk_ext: detect and warn if running in OE environment
  toolchain-shar-extract.sh: print full-length title underline

 meta/classes/populate_sdk_ext.bbclass | 18 +++++++++++++-----
 meta/files/ext-sdk-prepare.sh         | 16 ++++++++++++++++
 meta/files/toolchain-shar-extract.sh  |  5 +++--
 3 files changed, 32 insertions(+), 7 deletions(-)
 create mode 100644 meta/files/ext-sdk-prepare.sh

-- 
2.1.0

[PATCH v2 1/5] classes/populate_sdk_ext: consistent indentation

[Paul Eggleton]

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index c244d93..90033ba 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -238,11 +238,11 @@ sdk_ext_postinst() {
 	mv *-nativesdk-libc.tar.* $target_sdk_dir/`dirname ${oe_init_build_env_path}`
 
 	if [ "$prepare_buildsystem" != "no" ]; then
-	    printf "Preparing build system...\n"
-	    # dash which is /bin/sh on Ubuntu will not preserve the
-	    # current working directory when first ran, nor will it set $1 when
-	    # sourcing a script. That is why this has to look so ugly.
-	    sh -c ". buildtools/environment-setup* > preparing_build_system.log && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> preparing_build_system.log && bitbake ${SDK_TARGETS} >> preparing_build_system.log" || { echo "SDK preparation failed: see `pwd`/preparing_build_system.log" ; exit 1 ; }
+		printf "Preparing build system...\n"
+		# dash which is /bin/sh on Ubuntu will not preserve the
+		# current working directory when first ran, nor will it set $1 when
+		# sourcing a script. That is why this has to look so ugly.
+		sh -c ". buildtools/environment-setup* > preparing_build_system.log && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> preparing_build_system.log && bitbake ${SDK_TARGETS} >> preparing_build_system.log" || { echo "SDK preparation failed: see `pwd`/preparing_build_system.log" ; exit 1 ; }
 	fi
 	echo done
 }
-- 
2.1.0

[PATCH v2 2/5] classes/populate_sdk_ext: prevent do_rootfs from executing on install

[Paul Eggleton]

In order to prepare the build system within the extensible SDK, we
actually go ahead and build the targets specified by SDK_TARGETS (by
default the image the SDK was built for). Assuming that's an image, we
don't actually need to build the image itself - we just need to have
everything done up to the point before building the image, so that we
have everything needed in the sysroot.

In order to do this, create temporary bbappends for each of the targets
in the workspace layer that stub out do_rootfs. This is a little bit of
a hack but is the least intrusive fix at this point. To make things a
bit tidier, I have split out the preparation commands into a separate
script so we can run that in the appropriate environment rather than all
the commands separately.

Fixes [YOCTO #7590].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass |  5 ++++-
 meta/files/ext-sdk-prepare.sh         | 16 ++++++++++++++++
 2 files changed, 20 insertions(+), 1 deletion(-)
 create mode 100644 meta/files/ext-sdk-prepare.sh

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 90033ba..0feb3b2 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -203,6 +203,8 @@ install_tools() {
 	install $buildtools_path ${SDK_OUTPUT}/${SDKPATH}
 
 	install ${SDK_DEPLOY}/${BUILD_ARCH}-nativesdk-libc.tar.bz2 ${SDK_OUTPUT}/${SDKPATH}
+
+	install -m 0755 ${COREBASE}/meta/files/ext-sdk-prepare.sh ${SDK_OUTPUT}/${SDKPATH}
 }
 
 # Since bitbake won't run as root it doesn't make sense to try and install
@@ -242,8 +244,9 @@ sdk_ext_postinst() {
 		# dash which is /bin/sh on Ubuntu will not preserve the
 		# current working directory when first ran, nor will it set $1 when
 		# sourcing a script. That is why this has to look so ugly.
-		sh -c ". buildtools/environment-setup* > preparing_build_system.log && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> preparing_build_system.log && bitbake ${SDK_TARGETS} >> preparing_build_system.log" || { echo "SDK preparation failed: see `pwd`/preparing_build_system.log" ; exit 1 ; }
+		sh -c ". buildtools/environment-setup* > preparing_build_system.log && cd $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >> preparing_build_system.log && $target_sdk_dir/ext-sdk-prepare.sh $target_sdk_dir '${SDK_TARGETS}' >> preparing_build_system.log 2>&1" || { echo "SDK preparation failed: see `pwd`/preparing_build_system.log" ; exit 1 ; }
 	fi
+	rm -f $target_sdk_dir/ext-sdk-prepare.sh
 	echo done
 }
 
diff --git a/meta/files/ext-sdk-prepare.sh b/meta/files/ext-sdk-prepare.sh
new file mode 100644
index 0000000..a5a46be
--- /dev/null
+++ b/meta/files/ext-sdk-prepare.sh
@@ -0,0 +1,16 @@
+#!/bin/sh
+
+# Prepare the build system within the extensible SDK
+
+target_sdk_dir="$1"
+sdk_targets="$2"
+
+# Avoid actually building images during this phase
+# This is a hack, to be sure, but we really don't need to do this here
+for sdktarget in $sdk_targets ; do
+	bbappend=`recipetool newappend $target_sdk_dir/workspace $sdktarget`
+	printf 'python do_rootfs_forcevariable () {\n    bb.utils.mkdirhier(d.getVar("IMAGE_ROOTFS", True))\n}\n' > $bbappend
+	printf "Created bbappend %s\n" "$bbappend"
+done
+bitbake $sdk_targets || exit 1
+rm -rf $target_sdk_dir/workspace/appends/*
-- 
2.1.0

[PATCH v2 3/5] classes/populate_sdk_ext: add note to env setup script

[Paul Eggleton]

Print a note at the end of the environment setup script pointing to
devtool.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 0feb3b2..060a44a 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -236,6 +236,8 @@ sdk_ext_postinst() {
 	# so put it at the end of $PATH.
 	echo "export PATH=\$PATH:$target_sdk_dir/sysroots/${SDK_SYS}/${bindir_nativesdk}" >> $env_setup_script
 
+	echo "printf 'SDK environment now set up; additionally you may now run devtool to perform development tasks.\nRun devtool --help for further details.\n'" >> $env_setup_script
+
 	# For now this is where uninative.bbclass expects the tarball
 	mv *-nativesdk-libc.tar.* $target_sdk_dir/`dirname ${oe_init_build_env_path}`
 
-- 
2.1.0

[PATCH v2 4/5] classes/populate_sdk_ext: detect and warn if running in OE environment

[Paul Eggleton]

If you run the extensible SDK environment setup script in a shell
session where oe-init-build-env has been run already, and attempt to use
the two together, strange things happen - you may not even be running
devtool from the extensible SDK, but the OE tree. This isn't a supported
use case anyway, so show a warning recommending starting a new shell
session.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/populate_sdk_ext.bbclass | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta/classes/populate_sdk_ext.bbclass b/meta/classes/populate_sdk_ext.bbclass
index 060a44a..249ec36 100644
--- a/meta/classes/populate_sdk_ext.bbclass
+++ b/meta/classes/populate_sdk_ext.bbclass
@@ -238,6 +238,9 @@ sdk_ext_postinst() {
 
 	echo "printf 'SDK environment now set up; additionally you may now run devtool to perform development tasks.\nRun devtool --help for further details.\n'" >> $env_setup_script
 
+	# Warn if trying to use external bitbake and the ext SDK together
+	echo "(which bitbake > /dev/null 2>&1 && echo 'WARNING: attempting to use the extensible SDK in an environment set up to run bitbake - this may lead to unexpected results. Please source this script in a new shell session instead.') || true" >> $env_setup_script
+
 	# For now this is where uninative.bbclass expects the tarball
 	mv *-nativesdk-libc.tar.* $target_sdk_dir/`dirname ${oe_init_build_env_path}`
 
-- 
2.1.0

[PATCH v2 5/5] toolchain-shar-extract.sh: print full-length title underline

[Paul Eggleton]

Print a line that's the full length of the title that it's underneath.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/files/toolchain-shar-extract.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/meta/files/toolchain-shar-extract.sh b/meta/files/toolchain-shar-extract.sh
index 2ffc2d1..568b580 100644
--- a/meta/files/toolchain-shar-extract.sh
+++ b/meta/files/toolchain-shar-extract.sh
@@ -68,8 +68,9 @@ while getopts ":yd:nDRS" OPT; do
 	esac
 done
 
-echo "@SDK_TITLE@ installer version @SDK_VERSION@"
-echo "==========================================================="
+titlestr="@SDK_TITLE@ installer version @SDK_VERSION@"
+printf "%s\n" "$titlestr"
+printf "%${#titlestr}s\n" | tr " " "="
 
 if [ $verbose = 1 ] ; then
 	set -x
-- 
2.1.0

Re: [PATCH v2 2/5] classes/populate_sdk_ext: prevent do_rootfs from executing on install

[Paul Eggleton]

On Wednesday 14 October 2015 21:23:31 Paul Eggleton wrote:
> In order to prepare the build system within the extensible SDK, we
> actually go ahead and build the targets specified by SDK_TARGETS (by
> default the image the SDK was built for). Assuming that's an image, we
> don't actually need to build the image itself - we just need to have
> everything done up to the point before building the image, so that we
> have everything needed in the sysroot.
> 
> In order to do this, create temporary bbappends for each of the targets
> in the workspace layer that stub out do_rootfs. This is a little bit of
> a hack but is the least intrusive fix at this point. To make things a
> bit tidier, I have split out the preparation commands into a separate
> script so we can run that in the appropriate environment rather than all
> the commands separately.
> 
> Fixes [YOCTO #7590].
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
>  meta/classes/populate_sdk_ext.bbclass |  5 ++++-
>  meta/files/ext-sdk-prepare.sh         | 16 ++++++++++++++++
>  2 files changed, 20 insertions(+), 1 deletion(-)
>  create mode 100644 meta/files/ext-sdk-prepare.sh
> 
> diff --git a/meta/classes/populate_sdk_ext.bbclass
> b/meta/classes/populate_sdk_ext.bbclass index 90033ba..0feb3b2 100644
> --- a/meta/classes/populate_sdk_ext.bbclass
> +++ b/meta/classes/populate_sdk_ext.bbclass
> @@ -203,6 +203,8 @@ install_tools() {
>  	install $buildtools_path ${SDK_OUTPUT}/${SDKPATH}
> 
>  	install ${SDK_DEPLOY}/${BUILD_ARCH}-nativesdk-libc.tar.bz2
> ${SDK_OUTPUT}/${SDKPATH} +
> +	install -m 0755 ${COREBASE}/meta/files/ext-sdk-prepare.sh
> ${SDK_OUTPUT}/${SDKPATH} }
> 
>  # Since bitbake won't run as root it doesn't make sense to try and install
> @@ -242,8 +244,9 @@ sdk_ext_postinst() {
>  		# dash which is /bin/sh on Ubuntu will not preserve the
>  		# current working directory when first ran, nor will it set $1 when
>  		# sourcing a script. That is why this has to look so ugly.
> -		sh -c ". buildtools/environment-setup* > preparing_build_system.log 
&& cd
> $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir
> && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >>
> preparing_build_system.log && bitbake ${SDK_TARGETS} >>
> preparing_build_system.log" || { echo "SDK preparation failed: see
> `pwd`/preparing_build_system.log" ; exit 1 ; } +		sh -c ".
> buildtools/environment-setup* > preparing_build_system.log && cd
> $target_sdk_dir/`dirname ${oe_init_build_env_path}` && set $target_sdk_dir
> && . $target_sdk_dir/${oe_init_build_env_path} $target_sdk_dir >>
> preparing_build_system.log && $target_sdk_dir/ext-sdk-prepare.sh
> $target_sdk_dir '${SDK_TARGETS}' >> preparing_build_system.log 2>&1" || {
> echo "SDK preparation failed: see `pwd`/preparing_build_system.log" ; exit
> 1 ; } fi
> +	rm -f $target_sdk_dir/ext-sdk-prepare.sh
>  	echo done
>  }
> 
> diff --git a/meta/files/ext-sdk-prepare.sh b/meta/files/ext-sdk-prepare.sh
> new file mode 100644
> index 0000000..a5a46be
> --- /dev/null
> +++ b/meta/files/ext-sdk-prepare.sh
> @@ -0,0 +1,16 @@
> +#!/bin/sh
> +
> +# Prepare the build system within the extensible SDK
> +
> +target_sdk_dir="$1"
> +sdk_targets="$2"
> +
> +# Avoid actually building images during this phase
> +# This is a hack, to be sure, but we really don't need to do this here
> +for sdktarget in $sdk_targets ; do
> +	bbappend=`recipetool newappend $target_sdk_dir/workspace $sdktarget`
> +	printf 'python do_rootfs_forcevariable () {\n   
> bb.utils.mkdirhier(d.getVar("IMAGE_ROOTFS", True))\n}\n' > $bbappend
> +	printf "Created bbappend %s\n" "$bbappend"
> +done
> +bitbake $sdk_targets || exit 1
> +rm -rf $target_sdk_dir/workspace/appends/*

Unfortunately this doesn't avoid the error I was trying to avoid because 
do_bootimg still executes:

https://bugzilla.yoctoproject.org/show_bug.cgi?id=7629

I'd better send a v3.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre