[PATCH 0/3] Add support for relocatable SDK to ADT installer and some fixes

[PATCH 0/3] Add support for relocatable SDK to ADT installer and some fixes

[Laurentiu Palcu]

Hi,

This patchset adds support for relocating SDK to ADT installer and a couple of
fixes.

Thanks,
Laurentiu

The following changes since commit af847d36375aa53f0c1ee2a00c97ba9f38837d1b:

  bitbake: bitbake: build.py: Add stampdir argument to cached_mtime_noerror (2012-08-16 12:27:41 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib lpalcu/relocatable_adt
  http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=lpalcu/relocatable_adt

Laurentiu Palcu (3):
  populate_sdk_base.bbclass: fix SDK relocation issues
  package.bbclass: change RPATHs for cross-canadian binaries
  adt-installer: add support for relocatable SDK

 meta/classes/package.bbclass                       |    2 +-
 meta/classes/populate_sdk_base.bbclass             |    4 +-
 .../installer/adt-installer/adt_installer          |   11 ++
 .../adt-installer/scripts/adt_installer_internal   |  111 ++++++++++++++++----
 .../installer/adt-installer_1.0.bb                 |   37 +++----
 5 files changed, 126 insertions(+), 39 deletions(-)

-- 
1.7.9.5

[PATCH 1/3] populate_sdk_base.bbclass: fix SDK relocation issues

[Laurentiu Palcu]

The problem appears if multiple setup environment scripts are found. In
order to find only the script we're interested in, I removed globbing in
matching pattern with ${REAL_MULTIMACH_TARGET_SYS} that will be expanded
to the correct string.

Also, fix a problem when changing the scripts/configs. The grep pattern
matched also files that contained "text" in their name.

Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
---
 meta/classes/populate_sdk_base.bbclass |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index 0d19741..a1cb12a 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -154,7 +154,7 @@ echo "done"
 
 echo -n "Setting it up..."
 # fix environment paths
-env_setup_script=$(find $target_sdk_dir -name "environment-setup*")
+env_setup_script=$(find $target_sdk_dir -name "environment-setup-${REAL_MULTIMACH_TARGET_SYS}")
 sed -e "s:$DEFAULT_INSTALL_DIR:$target_sdk_dir:g" -i $env_setup_script
 
 # fix dynamic loader paths in all ELF SDK binaries
@@ -168,7 +168,7 @@ if [ $? -ne 0 ]; then
 fi
 
 # replace ${SDKPATH} with the new prefix in all text files: configs/scripts/etc
-find $native_sysroot -type f -exec file '{}' \;|grep text|cut -d':' -f1|xargs sed -i -e "s:$DEFAULT_INSTALL_DIR:$target_sdk_dir:g"
+find $native_sysroot -type f -exec file '{}' \;|grep ":.*ASCII.*text"|cut -d':' -f1|xargs sed -i -e "s:$DEFAULT_INSTALL_DIR:$target_sdk_dir:g"
 
 echo done
 
-- 
1.7.9.5

[PATCH 2/3] package.bbclass: change RPATHs for cross-canadian binaries

[Laurentiu Palcu]

When building the meta-toolchain, the binaries didn't get relocatable
RPATHs. They were hardcoded to the default path. Hence, if one had
already installed one SDK in the default path and one in another
location, the later toolchain's binaries would search and load libraries
from the first location, ending in a "Segmentation Fault".

[YOCTO #2927]

Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
---
 meta/classes/package.bbclass |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index d122cd9..278f0f0 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -448,7 +448,7 @@ python perform_packagecopy () {
     subprocess.call('tar -cf - -C %s -ps . | tar -xf - -C %s' % (dest, dvar), shell=True)
 
     # replace RPATHs for the nativesdk binaries, to make them relocatable
-    if bb.data.inherits_class('nativesdk', d):
+    if bb.data.inherits_class('nativesdk', d) or bb.data.inherits_class('cross-canadian', d):
         rpath_replace (dvar, d)
 }
 
-- 
1.7.9.5

[PATCH 3/3] adt-installer: add support for relocatable SDK

[Laurentiu Palcu]

Since we made the SDK relocatable, we have to add this functionality to
adt-installer too.

Other:
 - Changed tabs to spaces in the recipe file too.

Signed-off-by: Laurentiu Palcu <laurentiu.palcu@intel.com>
---
 .../installer/adt-installer/adt_installer          |   11 ++
 .../adt-installer/scripts/adt_installer_internal   |  111 ++++++++++++++++----
 .../installer/adt-installer_1.0.bb                 |   37 +++----
 3 files changed, 123 insertions(+), 36 deletions(-)

diff --git a/meta/recipes-devtools/installer/adt-installer/adt_installer b/meta/recipes-devtools/installer/adt-installer/adt_installer
index 1a53eb9..2d252b7 100755
--- a/meta/recipes-devtools/installer/adt-installer/adt_installer
+++ b/meta/recipes-devtools/installer/adt-installer/adt_installer
@@ -332,6 +332,17 @@ if [ -f "$YOCTOADT_INSTALL_LOG_FILE" ]; then
   rm $YOCTOADT_INSTALL_LOG_FILE
 fi
 
+echo -n "Please enter the install location (default: $DEFAULT_INSTALL_FOLDER): "
+read INSTALL_FOLDER
+
+if [ "$INSTALL_FOLDER" = "" ]; then
+    INSTALL_FOLDER=$DEFAULT_INSTALL_FOLDER
+fi
+
+eval INSTALL_FOLDER=$INSTALL_FOLDER
+export INSTALL_FOLDER=$(readlink -m $INSTALL_FOLDER)
+
+clear
 
 usage
 
diff --git a/meta/recipes-devtools/installer/adt-installer/scripts/adt_installer_internal b/meta/recipes-devtools/installer/adt-installer/scripts/adt_installer_internal
index 6201095..a540c0d 100755
--- a/meta/recipes-devtools/installer/adt-installer/scripts/adt_installer_internal
+++ b/meta/recipes-devtools/installer/adt-installer/scripts/adt_installer_internal
@@ -50,8 +50,8 @@ install_native_sdk()
 
 echo_info "\nStart installing selected native ADT for archs: $YOCTOADT_TARGETS..."
 
-# where the packages are installed. Currently only / is supported
-NATIVE_INSTALL_DIR="/"
+# where the packages are installed.
+NATIVE_INSTALL_DIR=$INSTALL_FOLDER
 
 if [ -d "$INSTALL_FOLDER" ]; then
   echo_info "\nNative ADT installation directory \"$INSTALL_FOLDER\" already exists! Continue installation will override its contents!"
@@ -59,28 +59,51 @@ if [ -d "$INSTALL_FOLDER" ]; then
 fi
 
 #Now begin to install native sdk and extract qemu rootfs which needs privilege rights
-echo_info "#######################################################################"
-echo_info "Please note from this point on installation requires sudo password ..."
-echo_info "#######################################################################"
-username='id -nu'
+#depending on the install location
+username=$(id -nu)
+
+# find the owner of the parent
+dir=$NATIVE_INSTALL_DIR
+while [ 1 ]; do
+    if [ -d $dir ]; then
+        owner=$(stat -c %U $dir)
+        break
+   else
+        dir=$(dirname $dir)
+   fi
+done
+
+if [ "$owner" = "$username" ]; then
+    SUDO=""
+else
+    echo_info "#######################################################################"
+    echo_info "Please note from this point on installation requires sudo password ..."
+    echo_info "#######################################################################"
+    SUDO=sudo
+fi
+
 #we need to make this directory firstly since opkg need to use it.
 OPKG_LOCK_DIR="$NATIVE_INSTALL_DIR/$OPKG_LIBDIR/opkg"
 if [ ! -d "$OPKG_LOCK_DIR" ]; then
-  sudo mkdir -p $OPKG_LOCK_DIR
+  $SUDO mkdir -p $OPKG_LOCK_DIR
   echo_info "Successfully create directory $OPKG_LOCK_DIR. "
 #if user delete /opt/xxx, while dangling folders there, report error
 elif [ ! -d "$INSTALL_FOLDER" ]; then
   echo_info "\nDangling opkg cache folder $OPKG_LOCK_DIR detected. Continue installation will remove the folder!"
   confirm_install $1
-  sudo rm -rf $OPKG_LOCK_DIR
-  sudo mkdir -p $OPKG_LOCK_DIR
+  $SUDO rm -rf $OPKG_LOCK_DIR
+  $SUDO mkdir -p $OPKG_LOCK_DIR
 #if user are updating installing, just let him/her go, give her/him prompt
 else
   echo_info "ADT has already been installed. Will update its contents..."
 fi
 
 #first update repository
-OPKG_CMD="sudo -E $LOCAL_OPKG_LOC/bin/opkg-cl"
+if [ "x$SUDO" = "x" ]; then
+    OPKG_CMD="$LOCAL_OPKG_LOC/bin/opkg-cl"
+else
+    OPKG_CMD="sudo -E $LOCAL_OPKG_LOC/bin/opkg-cl"
+fi
 
 echo_info "Updating opkg..."
 $OPKG_CMD -f $OPKG_CONFIG_FILE -o $NATIVE_INSTALL_DIR update &>> $YOCTOADT_INSTALL_LOG_FILE
@@ -117,13 +140,6 @@ for native_target_type in $YOCTOADT_TARGETS; do
 
 done
 
-# Link the ld.so.cache file into the hosts filesystem
-if [ ! -f "$OECORE_NATIVE_SYSROOT/etc/ld.so.cache" ]; then 
-echo_info "Link the ld.so.cache file into the host filesystem"
-sudo ln -s /etc/ld.so.cache $OECORE_NATIVE_SYSROOT/etc/ld.so.cache
-check_result
-fi
-
 if [ "$YOCTOADT_QEMU" == "Y" ] || [ "$YOCTOADT_QEMU" = "y" ]; then
   echo_info "\nInstalling qemu native ..."
   $OPKG_INSTALL_NATIVE_CMD qemu-nativesdk &>> $YOCTOADT_INSTALL_LOG_FILE
@@ -138,6 +154,65 @@ if [ "$YOCTOADT_NFS_UTIL" == "Y" ] || [ "$YOCTOADT_NFS_UTIL" == "y" ]; then
   check_result
 fi
 
+# Lose the ./opt/${DISTRO}/${SDK_VERSION} part, we don't really need to keep
+# the entire directory structure. We could patch opkg to do that but it's far
+# simpler to do that here and achieve the same result.
+# This is done in two steps:
+# Step 1: copy ./opt/${DISTRO}/${SDK_VERSION} contents to $NATIVE_INSTALL_DIR.
+# We cannot use move if $NATIVE_INSTALL_DIR is not empty (for example: contains
+# another SDK)
+$SUDO cp -r $NATIVE_INSTALL_DIR/$DEFAULT_INSTALL_FOLDER/* $NATIVE_INSTALL_DIR
+
+# delete the source directory now
+$SUDO rm -rf $NATIVE_INSTALL_DIR/$DEFAULT_INSTALL_FOLDER/*
+
+# Step 2: Delete the ./opt/${DISTRO}/${SDK_VERSION} directories too, they should be empty
+dir=$NATIVE_INSTALL_DIR/$DEFAULT_INSTALL_FOLDER
+while [ "$dir" != "$NATIVE_INSTALL_DIR" ]; do
+    # if the user chose / as the install folder, then we should leave /opt in place
+    if [ "$dir" = "/opt" ]; then
+        break
+    fi
+
+    # try to delete the directory, only if it's empty
+    $SUDO rmdir $dir
+    if [ $? -ne 0 ]; then
+        break
+    fi
+
+    # go to the next directory
+    dir=$(dirname $dir)
+done
+
+# Link the ld.so.cache file into the hosts filesystem
+if [ ! -f "$OECORE_NATIVE_SYSROOT/etc/ld.so.cache" ]; then
+echo_info "Link the ld.so.cache file into the host filesystem"
+$SUDO ln -s /etc/ld.so.cache $OECORE_NATIVE_SYSROOT/etc/ld.so.cache
+check_result
+fi
+
+# relocate binaries
+echo_info "\nRelocating binaries ..."
+escaped_sdkpath=$(echo $DEFAULT_INSTALL_FOLDER |sed -e "s:[\+\.]:\\\\\\\\\0:g")
+
+# We don't change the script in-place since we may want the user to re-run
+# adt-installer script
+$SUDO sed -e "s:##DEFAULT_INSTALL_DIR##:$escaped_sdkpath:" scripts/relocate_sdk.py > scripts/relocate_sdk_tmp.py
+$SUDO chmod +x scripts/relocate_sdk_tmp.py
+
+dl_path=$(find $OECORE_NATIVE_SYSROOT/lib -name "ld-linux*")
+executable_files=$(find $OECORE_NATIVE_SYSROOT -type f -perm +111)
+
+$SUDO scripts/relocate_sdk_tmp.py $INSTALL_FOLDER $dl_path $executable_files
+check_result
+
+# replace /opt/${DISTRO}/${SDK_VERSION} with the install folder in all configs
+env_setup_script=$(find $NATIVE_INSTALL_DIR -name "environment-setup-*")
+$SUDO sed -i -e "s:$DEFAULT_INSTALL_FOLDER:$NATIVE_INSTALL_DIR:g" $env_setup_script
+
+find $OECORE_NATIVE_SYSROOT -type f -exec file '{}' \;|grep ":.*ASCII.*text"|cut -d':' -f1|\
+    xargs $SUDO sed -i -e "s:$DEFAULT_INSTALL_FOLDER:$NATIVE_INSTALL_DIR:g"
+
 echo_info "\nSuccessfully installed selected native ADT!"
 }
 
@@ -180,7 +255,7 @@ else
 fi
 
 if [ ! -z "$env_filename" ]; then
-  sudo sed -i -e "s%##SDKTARGETSYSROOT##%$target_sysroot%g" $env_filename
+  $SUDO sed -i -e "s%##SDKTARGETSYSROOT##%$target_sysroot%g" $env_filename
 else
   echo_info "[ADT_INST] Error: Failed to find environment script for arch: $1"
   return 1 
diff --git a/meta/recipes-devtools/installer/adt-installer_1.0.bb b/meta/recipes-devtools/installer/adt-installer_1.0.bb
index e1edf2f..d5d0da5 100644
--- a/meta/recipes-devtools/installer/adt-installer_1.0.bb
+++ b/meta/recipes-devtools/installer/adt-installer_1.0.bb
@@ -30,7 +30,7 @@ ALLOW_EMPTY = "1"
 
 PACKAGES = ""
 
-PR = "r9"
+PR = "r10"
 
 ADT_DEPLOY = "${TMPDIR}/deploy/sdk/"
 ADT_DIR = "${WORKDIR}/adt-installer/"
@@ -56,23 +56,24 @@ ADTREPO = "http://adtrepo.yoctoproject.org/${SDK_VERSION}"
 do_populate_adt[umask] = "022"
 
 fakeroot do_populate_adt () {
-	cd ${WORKDIR}
-	mkdir -p ${ADT_DEPLOY}
-	rm -f ${ADT_DEPLOY}/adt-installer.tar.bz2
-	rm -rf ${ADT_DIR}
-	mkdir -p ${ADT_DIR}/opkg/build
-	cp -r opkg ${ADT_DIR}/
-	sed -i -e 's#ADTREPO_URL#${ADTREPO}#' ${ADT_DIR}/opkg/conf/*.conf
-	cp -r trunk ${ADT_DIR}/opkg/build/
-	mv ${ADT_DIR}/opkg/build/trunk ${ADT_DIR}/opkg/build/opkg-svn
-	cp -r scripts ${ADT_DIR}/
-	cp adt_installer ${ADT_DIR}
-	cp adt_installer.conf ${ADT_DIR}
-	sed -i -e 's#YOCTOADT_VERSION#${SDK_VERSION}#' ${ADT_DIR}/adt_installer.conf
-        echo 'SDK_VENDOR=${SDK_VENDOR}' >> ${ADT_DIR}/scripts/data_define
-        echo 'INSTALL_FOLDER=${SDKPATH}' >> ${ADT_DIR}/scripts/data_define
-	tar cfj adt_installer.tar.bz2 adt-installer
-	cp ${WORKDIR}/adt_installer.tar.bz2 ${ADT_DEPLOY}
+    cd ${WORKDIR}
+    mkdir -p ${ADT_DEPLOY}
+    rm -f ${ADT_DEPLOY}/adt-installer.tar.bz2
+    rm -rf ${ADT_DIR}
+    mkdir -p ${ADT_DIR}/opkg/build
+    cp -r opkg ${ADT_DIR}/
+    sed -i -e 's#ADTREPO_URL#${ADTREPO}#' ${ADT_DIR}/opkg/conf/*.conf
+    cp -r trunk ${ADT_DIR}/opkg/build/
+    mv ${ADT_DIR}/opkg/build/trunk ${ADT_DIR}/opkg/build/opkg-svn
+    cp -r scripts ${ADT_DIR}/
+    cp adt_installer ${ADT_DIR}
+    cp adt_installer.conf ${ADT_DIR}
+    sed -i -e 's#YOCTOADT_VERSION#${SDK_VERSION}#' ${ADT_DIR}/adt_installer.conf
+    echo 'SDK_VENDOR=${SDK_VENDOR}' >> ${ADT_DIR}/scripts/data_define
+    echo 'DEFAULT_INSTALL_FOLDER=${SDKPATH}' >> ${ADT_DIR}/scripts/data_define
+    cp ${COREBASE}/scripts/relocate_sdk.py ${ADT_DIR}/scripts/
+    tar cfj adt_installer.tar.bz2 adt-installer
+    cp ${WORKDIR}/adt_installer.tar.bz2 ${ADT_DEPLOY}
 }
 
 do_populate_adt[nostamp] = "1"
-- 
1.7.9.5

Re: [PATCH 3/3] adt-installer: add support for relocatable SDK

[Richard Purdie]

On Fri, 2012-08-17 at 06:44 +0300, Laurentiu Palcu wrote:
> Since we made the SDK relocatable, we have to add this functionality to
> adt-installer too.
> 
> Other:
>  - Changed tabs to spaces in the recipe file too.

Shell functions use tabs, python uses spaces. The OE TSC has thought
long and hard about this and that is the decision. Its not perfect but
the alternatives will cause more problems than any benefit is worth.

Cheers,

Richard

Re: [PATCH 3/3] adt-installer: add support for relocatable SDK

[Laurentiu Palcu]

On 08/17/2012 12:38 PM, Richard Purdie wrote:
> On Fri, 2012-08-17 at 06:44 +0300, Laurentiu Palcu wrote:
>> Since we made the SDK relocatable, we have to add this functionality to
>> adt-installer too.
>>
>> Other:
>>  - Changed tabs to spaces in the recipe file too.
> 
> Shell functions use tabs, python uses spaces. The OE TSC has thought
> long and hard about this and that is the decision. Its not perfect but
> the alternatives will cause more problems than any benefit is worth.
I saw so many warnings lately with "variable do_xxx contains tabs,
please remove these" I got carried by the wave! :| I'll resend the
patches and restore the tabs!

Thanks,
Laurentiu
> 
> Cheers,
> 
> Richard
> 
> 
> 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
> 

Re: [PATCH 3/3] adt-installer: add support for relocatable SDK

[Andreas Müller]

On Fri, Aug 17, 2012 at 11:38 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Fri, 2012-08-17 at 06:44 +0300, Laurentiu Palcu wrote:
>> Since we made the SDK relocatable, we have to add this functionality to
>> adt-installer too.
>>
>> Other:
>>  - Changed tabs to spaces in the recipe file too.
>
> Shell functions use tabs, python uses spaces. The OE TSC has thought
> long and hard about this and that is the decision. Its not perfect but
> the alternatives will cause more problems than any benefit is worth.
Is there some report/log/styleguide helping me to understand why we
use two tab/space strategies instead of having spaces for all?

Andreas
>
> Cheers,
>
> Richard

Re: [PATCH 3/3] adt-installer: add support for relocatable SDK

[Richard Purdie]

On Fri, 2012-08-17 at 12:42 +0200, Andreas Müller wrote:
> On Fri, Aug 17, 2012 at 11:38 AM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Fri, 2012-08-17 at 06:44 +0300, Laurentiu Palcu wrote:
> >> Since we made the SDK relocatable, we have to add this functionality to
> >> adt-installer too.
> >>
> >> Other:
> >>  - Changed tabs to spaces in the recipe file too.
> >
> > Shell functions use tabs, python uses spaces. The OE TSC has thought
> > long and hard about this and that is the decision. Its not perfect but
> > the alternatives will cause more problems than any benefit is worth.
> Is there some report/log/styleguide helping me to understand why we
> use two tab/space strategies instead of having spaces for all?

This is the style the bulk of the metadata uses. We really don't want
patches changing a significant portion of the metadata for cosmetic
reasons. It would create a world of pain for people handling/merging
patches, a lot of needless code churn and significant problems for
people trying to backport changes to older versions of the code base
amongst other reasons.

Cheers,

Richard