[PATCH 0/2] Populate site config files content to rootfs

[PATCH 0/2] Populate site config files content to rootfs

[Lianhao Lu]

This series of patches fixed the bug #892. Currently, the site config 
files used in the cross compiling environment generated by meta-toolchain 
partly come from SITECONFIG_SYSROOT_CACHE, where each package will put 
their own site config cache file during their populate-sysroot task.

However, there is no way to guarantee the meta-toolchain to collect these 
site config cache files after all other packages have finished their 
populate-sysroot tasks. Since we now encourage users to use the same rootfs 
for both running and cross-compiling, by populating the site config files'
content into rootfs and having the CONFIG_SITE pointing to the rootfs, we 
can gurantee the cross compiling environment not missing any site config 
cache files' content.

The following changes since commit a1f87ec65fa1a6d5ce9a010548dbe7c01ab9b711:
  Kumar Gala (1):
        eglibc: force GLIBC_EXTRA_OECONF to not be uset for nativesdk

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib llu/bug892
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=llu/bug892

Lianhao Lu (2):
  image: populate site config files into roootfs.
  meta-toolchain/environment: Use site config in rootfs.

 meta/classes/image.bbclass                 |    1 +
 meta/classes/populate_sdk.bbclass          |    2 +-
 meta/classes/siteinfo.bbclass              |    4 +++-
 meta/classes/toolchain-scripts.bbclass     |   26 ++++++++++++++++++++------
 meta/recipes-core/meta/meta-environment.bb |    6 +-----
 meta/recipes-core/meta/meta-toolchain.bb   |    4 +---
 6 files changed, 27 insertions(+), 16 deletions(-)

[PATCH 1/2] image: populate site config files into roootfs.

[Lianhao Lu]

[YOCTO #892] populate the site config files content into the file in
{target}/cross_site_config in rootfs, which can be used by the cross
toolchain in meta-toolchain when the rootfs is used as sysroot.

Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
---
 meta/classes/image.bbclass             |    1 +
 meta/classes/siteinfo.bbclass          |    4 +++-
 meta/classes/toolchain-scripts.bbclass |   26 ++++++++++++++++++++------
 3 files changed, 24 insertions(+), 7 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 54eb78b..1104693 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -1,4 +1,5 @@
 inherit rootfs_${IMAGE_PKGTYPE}
+inherit toolchain-scripts
 
 IMAGETEST ?= "dummy"
 inherit imagetest-${IMAGETEST}
diff --git a/meta/classes/siteinfo.bbclass b/meta/classes/siteinfo.bbclass
index a61b5e5..02294c4 100644
--- a/meta/classes/siteinfo.bbclass
+++ b/meta/classes/siteinfo.bbclass
@@ -118,7 +118,7 @@ python () {
         bb.fatal("Please add your architecture to siteinfo.bbclass")
 }
 
-def siteinfo_get_files(d):
+def siteinfo_get_files(d, no_cache = False):
     sitedata = siteinfo_data(d)
     sitefiles = ""
     for path in d.getVar("BBPATH", True).split(":"):
@@ -127,6 +127,8 @@ def siteinfo_get_files(d):
             if os.path.exists(filename):
                 sitefiles += filename + " "
 
+    if no_cache: return sitefiles
+
     # Now check for siteconfig cache files
     path_siteconfig = bb.data.getVar('SITECONFIG_SYSROOTCACHE', d, 1)
     if os.path.isdir(path_siteconfig):
diff --git a/meta/classes/toolchain-scripts.bbclass b/meta/classes/toolchain-scripts.bbclass
index f7b52be..96c8968 100644
--- a/meta/classes/toolchain-scripts.bbclass
+++ b/meta/classes/toolchain-scripts.bbclass
@@ -9,7 +9,9 @@ toolchain_create_sdk_env_script () {
 	echo 'export PATH=${SDKPATHNATIVE}${bindir_nativesdk}:${SDKPATHNATIVE}${bindir_nativesdk}/${MULTIMACH_TARGET_SYS}:$PATH' >> $script
 	echo 'export PKG_CONFIG_SYSROOT_DIR=${SDKTARGETSYSROOT}' >> $script
 	echo 'export PKG_CONFIG_PATH=${SDKTARGETSYSROOT}${libdir}/pkgconfig' >> $script
-	echo 'export CONFIG_SITE=${SDKPATH}/site-config-${MULTIMACH_TARGET_SYS}' >> $script
+	echo 'export OECORE_NATIVE_SYSROOT="${SDKPATHNATIVE}"' >> $script
+	echo 'export OECORE_TARGET_SYSROOT="${SDKTARGETSYSROOT}"' >> $script
+	echo 'export CONFIG_SITE="$OECORE_TARGET_SYSROOT${target_datadir}/cross_site_config"' >> $script
 	echo 'export CC=${TARGET_PREFIX}gcc' >> $script
 	echo 'export CXX=${TARGET_PREFIX}g++' >> $script
 	echo 'export GDB=${TARGET_PREFIX}gdb' >> $script
@@ -26,8 +28,6 @@ toolchain_create_sdk_env_script () {
 	echo 'export CXXFLAGS="${TARGET_CC_ARCH} --sysroot=${SDKTARGETSYSROOT}"' >> $script
 	echo 'export LDFLAGS="${TARGET_LD_ARCH} --sysroot=${SDKTARGETSYSROOT}"' >> $script
 	echo 'export CPPFLAGS="${TARGET_CC_ARCH} --sysroot=${SDKTARGETSYSROOT}"' >> $script
-	echo 'export OECORE_NATIVE_SYSROOT="${SDKPATHNATIVE}"' >> $script
-	echo 'export OECORE_TARGET_SYSROOT="${SDKTARGETSYSROOT}"' >> $script
 	echo 'export OECORE_ACLOCAL_OPTS="-I ${SDKPATHNATIVE}/usr/share/aclocal"' >> $script
 	echo 'export POKY_DISTRO_VERSION="${DISTRO_VERSION}"' >> $script
 	echo 'export POKY_SDK_VERSION="${SDK_VERSION}"' >> $script
@@ -77,7 +77,9 @@ toolchain_create_sdk_env_script_for_installer () {
 	echo 'export PATH=${SDKPATHNATIVE}${bindir_nativesdk}:${SDKPATHNATIVE}${bindir_nativesdk}/'"${multimach_target_sys}"':$PATH' >> $script
 	echo 'export PKG_CONFIG_SYSROOT_DIR=##SDKTARGETSYSROOT##' >> $script
 	echo 'export PKG_CONFIG_PATH=##SDKTARGETSYSROOT##${target_libdir}/pkgconfig' >> $script
-	echo 'export CONFIG_SITE=${SDKPATH}/site-config-'"${multimach_target_sys}" >> $script
+	echo 'export OECORE_NATIVE_SYSROOT="${SDKPATHNATIVE}"' >> $script
+	echo 'export OECORE_TARGET_SYSROOT="##SDKTARGETSYSROOT##"' >> $script
+	echo 'export CONFIG_SITE="$OECORE_TARGET_SYSROOT${target_datadir}/cross_site_config"' >> $script
 	echo 'export CC=${TARGET_PREFIX}gcc' >> $script
 	echo 'export CXX=${TARGET_PREFIX}g++' >> $script
 	echo 'export GDB=${TARGET_PREFIX}gdb' >> $script
@@ -94,8 +96,6 @@ toolchain_create_sdk_env_script_for_installer () {
 	echo 'export CXXFLAGS="${TARGET_CC_ARCH} --sysroot=##SDKTARGETSYSROOT##"' >> $script
 	echo 'export LDFLAGS="${TARGET_LD_ARCH} --sysroot=##SDKTARGETSYSROOT##"' >> $script
 	echo 'export CPPFLAGS="${TARGET_CC_ARCH} --sysroot=##SDKTARGETSYSROOT##"' >> $script
-	echo 'export OECORE_NATIVE_SYSROOT="${SDKPATHNATIVE}"' >> $script
-	echo 'export OECORE_TARGET_SYSROOT="##SDKTARGETSYSROOT##"' >> $script
         echo 'export OECORE_ACLOCAL_OPTS="-I ${SDKPATHNATIVE}/usr/share/acloal"' >> $script
 	echo 'export POKY_DISTRO_VERSION="${DISTRO_VERSION}"' >> $script
 	echo 'export POKY_SDK_VERSION="${SDK_VERSION}"' >> $script
@@ -124,3 +124,17 @@ toolchain_create_sdk_version () {
 	echo 'Metadata Revision: ${METADATA_REVISION}' >> $versionfile
 	echo 'Timestamp: ${DATETIME}' >> $versionfile
 }
+
+IMAGE_PREPROCESS_COMMAND += "toolchain_create_rootfs_siteconfig ${IMAGE_ROOTFS}; "
+#Use := to avoid cross-canadian polution
+COMMON_CONFIG_SITE := "${@siteinfo_get_files(d,True)}"
+# create the site config file in the rootfs 
+toolchain_create_rootfs_siteconfig () {
+	local allfiles='${COMMON_CONFIG_SITE}'
+	for file in ${SITECONFIG_SYSROOTCACHE}/*; do
+		if [ -f $file ]; then
+			allfiles="$allfiles $file"
+		fi
+	done
+	toolchain_create_sdk_siteconfig $1${datadir}/cross_site_config $allfiles
+}
-- 
1.7.0.4

Re: [PATCH 1/2] image: populate site config files into roootfs.

[Phil Blundell]

On Thu, 2011-08-11 at 22:31 +0800, Lianhao Lu wrote:
> [YOCTO #892] populate the site config files content into the file in
> {target}/cross_site_config in rootfs, which can be used by the cross
> toolchain in meta-toolchain when the rootfs is used as sysroot.

Is that going to cause ${datadir}/cross_site_config to get created in
all rootfs images?  That would be undesirable, but I couldn't obviously
see a flag to turn it on and off.

p.

Re: [PATCH 1/2] image: populate site config files into roootfs.

[Lu, Lianhao]

Phil Blundell wrote on 2011-08-11:
> On Thu, 2011-08-11 at 22:31 +0800, Lianhao Lu wrote:
>> [YOCTO #892] populate the site config files content into the file in
>> {target}/cross_site_config in rootfs, which can be used by the cross
>> toolchain in meta-toolchain when the rootfs is used as sysroot.
> 
> Is that going to cause ${datadir}/cross_site_config to get created in
> all rootfs images?  That would be undesirable, but I couldn't
> obviously see a flag to turn it on and off.
> 

We can add flag to turn it on and off. But the problem is that in that case, the user need to build 2 image rootfs, one for running and one for cross compiling.

Best Regards,
Lianhao

Re: [PATCH 1/2] image: populate site config files into roootfs.

[Saul Wold]

On 08/11/2011 06:09 PM, Lu, Lianhao wrote:
> Phil Blundell wrote on 2011-08-11:
>> On Thu, 2011-08-11 at 22:31 +0800, Lianhao Lu wrote:
>>> [YOCTO #892] populate the site config files content into the file in
>>> {target}/cross_site_config in rootfs, which can be used by the cross
>>> toolchain in meta-toolchain when the rootfs is used as sysroot.
>>
>> Is that going to cause ${datadir}/cross_site_config to get created in
>> all rootfs images?  That would be undesirable, but I couldn't
>> obviously see a flag to turn it on and off.
>>
>
> We can add flag to turn it on and off. But the problem is that in that case, the user need to build 2 image rootfs, one for running and one for cross compiling.
>
Lianhao,

This is only required for toolchain, is there a way to create the 
externally from the rootfs and package it with the ADT?  The problem 
here is that you will add space to the image (not sure how big), maybe 
this only make set to be and IMAGE_FEATURE for -dev or -sdk images, just 
as we add the -dev and -dbg packages to only those images?

SaU!

> Best Regards,
> Lianhao
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

Re: [PATCH 1/2] image: populate site config files into roootfs.

[Lu, Lianhao]

Saul Wold wrote on 2011-08-12:
> On 08/11/2011 06:09 PM, Lu, Lianhao wrote:
>> Phil Blundell wrote on 2011-08-11:
>>> On Thu, 2011-08-11 at 22:31 +0800, Lianhao Lu wrote:
>>>> [YOCTO #892] populate the site config files content into the file
>>>> in {target}/cross_site_config in rootfs, which can be used by the
>>>> cross toolchain in meta-toolchain when the rootfs is used as sysroot.
>>> 
>>> Is that going to cause ${datadir}/cross_site_config to get created
>>> in all rootfs images?  That would be undesirable, but I couldn't
>>> obviously see a flag to turn it on and off.
>>> 
>> 
>> We can add flag to turn it on and off. But the problem is that in that
>> case, the user need to build 2 image rootfs, one for running and one
>> for cross compiling.
>> 
> Lianhao,
> 
> This is only required for toolchain, is there a way to create the
> externally from the rootfs and package it with the ADT?  The problem
> here is that you will add space to the image (not sure how big), maybe
> this only make set to be and IMAGE_FEATURE for -dev or -sdk images,
> just as we add the -dev and -dbg packages to only those images?
> 

Its about 50KB for core-image-sato-sdk. Having them in a separate package seems not feasible now, because we can't make sure the package is generated after all other packages have finished their populate-sysroot tasks. But it's a good idea to populate the site config files only into the -dev or -sdk images. I'll rework the patch.

Best Regards,
Lianhao

[PATCH 2/2] meta-toolchain/environment: Use site config in rootfs.

[Lianhao Lu]

[YOCTO #892] Use the site config file in rootfs to set up the build
environment.

Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
---
 meta/classes/populate_sdk.bbclass          |    2 +-
 meta/recipes-core/meta/meta-environment.bb |    6 +-----
 meta/recipes-core/meta/meta-toolchain.bb   |    4 +---
 3 files changed, 3 insertions(+), 9 deletions(-)

diff --git a/meta/classes/populate_sdk.bbclass b/meta/classes/populate_sdk.bbclass
index 0f3591b..f755b81 100644
--- a/meta/classes/populate_sdk.bbclass
+++ b/meta/classes/populate_sdk.bbclass
@@ -47,7 +47,7 @@ fakeroot do_populate_sdk() {
 	ln -s /etc/ld.so.cache ${SDK_OUTPUT}/${SDKPATHNATIVE}/etc/ld.so.cache
 
 	# Setup site file for external use
-	toolchain_create_sdk_siteconfig ${SDK_OUTPUT}/${SDKPATH}/site-config-${MULTIMACH_TARGET_SYS} ${CONFIG_SITE}
+	toolchain_create_rootfs_siteconfig ${SDK_OUTPUT}/${SDKTARGETSYSROOT}
 
 	toolchain_create_sdk_env_script
 
diff --git a/meta/recipes-core/meta/meta-environment.bb b/meta/recipes-core/meta/meta-environment.bb
index 114727c..b62ec0e 100644
--- a/meta/recipes-core/meta/meta-environment.bb
+++ b/meta/recipes-core/meta/meta-environment.bb
@@ -2,13 +2,11 @@ DESCRIPTION = "Package of environment files for SDK"
 LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=3f40d7994397109285ec7b81fdeb3b58 \
                     file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
 LICENSE = "MIT"
-PR = "r6"
+PR = "r7"
 
 EXCLUDE_FROM_WORLD = "1"
 
 inherit toolchain-scripts
-# get target config site before inheritting cross-canadian
-TARGET_CONFIG_SITE := "${@siteinfo_get_files(d)}"
 REAL_MULTIMACH_TARGET_SYS = "${TUNE_PKGARCH}${TARGET_VENDOR}-${TARGET_OS}"
 
 SDK_DIR = "${WORKDIR}/sdk"
@@ -23,8 +21,6 @@ do_generate_content() {
     rm -rf ${SDK_OUTPUT}
     mkdir -p ${SDK_OUTPUT}/${SDKPATH}
 
-    toolchain_create_sdk_siteconfig ${SDK_OUTPUT}/${SDKPATH}/site-config-${REAL_MULTIMACH_TARGET_SYS} ${TARGET_CONFIG_SITE}
-
     toolchain_create_sdk_env_script_for_installer ${REAL_MULTIMACH_TARGET_SYS}
 
     # Add version information
diff --git a/meta/recipes-core/meta/meta-toolchain.bb b/meta/recipes-core/meta/meta-toolchain.bb
index 164d717..1058c68 100644
--- a/meta/recipes-core/meta/meta-toolchain.bb
+++ b/meta/recipes-core/meta/meta-toolchain.bb
@@ -1,12 +1,10 @@
 DESCRIPTION = "Meta package for building a installable toolchain"
 LICENSE = "MIT"
 
-PR = "r5"
+PR = "r6"
 
 LIC_FILES_CHKSUM = "file://${COREBASE}/LICENSE;md5=3f40d7994397109285ec7b81fdeb3b58 \
                     file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
 
 IMAGETEST ?= "dummy"
 inherit populate_sdk imagetest-${IMAGETEST}
-
-CONFIG_SITE := "${@siteinfo_get_files(d)}"
-- 
1.7.0.4

Re: [PATCH 0/2] Populate site config files content to rootfs

[Richard Purdie]

On Thu, 2011-08-11 at 22:31 +0800, Lianhao Lu wrote:
> This series of patches fixed the bug #892. Currently, the site config 
> files used in the cross compiling environment generated by meta-toolchain 
> partly come from SITECONFIG_SYSROOT_CACHE, where each package will put 
> their own site config cache file during their populate-sysroot task.
> 
> However, there is no way to guarantee the meta-toolchain to collect these 
> site config cache files after all other packages have finished their 
> populate-sysroot tasks. Since we now encourage users to use the same rootfs 
> for both running and cross-compiling, by populating the site config files'
> content into rootfs and having the CONFIG_SITE pointing to the rootfs, we 
> can gurantee the cross compiling environment not missing any site config 
> cache files' content.

I'm afraid I'm not very keen on this patch since it complicates the
situation and I'm not sure its an improvement. We strive very hard never
to add files to a rootfs which aren't packaged and this approach breaks
that (and always adds them to certain image types which again, I worry
about).

Can't we use the no_cache option to siteinfo_get_files() to address this
bug, yet leave the cache files where they are currently located as part
of the toolchain?

Ultimately we could specific extra cache files to add (such as eglibc)
and ensure we have the correct dependencies to ensure the caches are
generated.

Cheers,

Richard