[PATCH 1/2] relocate_sdk.py: Fix corruption of sdk binaries

[PATCH 1/2] relocate_sdk.py: Fix corruption of sdk binaries

[Jason Wessel]

There are two cases of corruption that the relocate_sdk.py was not correctly
dealing with.

1) SDK Extras should be left alone
   Extra external binaries included in an SDK that were linked against the
   host's version of /usr/lib/ld-so.so should not get a relocation applied.
   In the case that was discovered these were LSB compliant binaries that
   already worked on many hosts.

2) If the interp section is too small generate an error
   In the case of the qemu user code, it was using its own .ld file
   to link the executables which overrides the default in the nativesdk
   binutils.  This generated host executables which had a interp section
   that was too small to relocate.

   Now the relocate_sdk.py will print an error and continue on such that
   the error can be fixed by a developer without having to do the
   difficult task of debugging why it is crashing or not loading correctly.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 scripts/relocate_sdk.py |   17 +++++++++++++----
 1 files changed, 13 insertions(+), 4 deletions(-)

diff --git a/scripts/relocate_sdk.py b/scripts/relocate_sdk.py
index 74bb7a5..3e3181f 100755
--- a/scripts/relocate_sdk.py
+++ b/scripts/relocate_sdk.py
@@ -66,7 +66,7 @@ def parse_elf_header():
     e_ehsize, e_phentsize, e_phnum, e_shentsize, e_shnum, e_shstrndx =\
         hdr_struct.unpack(elf_header[16:hdr_size])
 
-def change_interpreter():
+def change_interpreter(elf_file_name):
     if arch == 32:
         ph_struct = struct.Struct("<IIIIIIII")
     else:
@@ -89,8 +89,17 @@ def change_interpreter():
         if p_type == 3:
             # PT_INTERP section
             f.seek(p_offset)
-            dl_path = new_dl_path + "\0" * (p_filesz - len(new_dl_path))
-            f.write(dl_path)
+            dl_path = new_dl_path + "\0"
+            # External SDKs with mixed pre-compiled binaries should not get
+            # relocated so look for some variant of /lib
+            fname = f.read(11)
+            if fname.startswith("/lib/") or fname.startswith("/lib64/") or fname.startswith("/lib32/") or fname.startswith("/usr/lib32/") or fname.startswith("/usr/lib32/") or fname.startswith("/usr/lib64/"):
+                break
+            if (len(dl_path) >= p_memsz):
+                print "ERROR: could not relocate %s, interp size = %i and %i is needed." % (elf_file_name, p_memsz, len(dl_path))
+                break
+            f.seek(p_offset)
+            f.write(dl_path )
             break
 
 def change_dl_sysdirs():
@@ -199,7 +208,7 @@ for e in executables_list:
     arch = get_arch()
     if arch:
         parse_elf_header()
-        change_interpreter()
+        change_interpreter(e)
         change_dl_sysdirs()
 
     """ change permissions back """
-- 
1.7.1

[PATCH 2/2] populate_sdk_base.bbclass: Improve debugging capabilities for SDK installer

[Jason Wessel]

After having to debug the SDK installer a few times in
addition to the relocation code the following patch was created
to improve the capabilities around debugging the SDK installer.

1) Add a verbose mode -D which set a set -x to see what
   the SDK installer is doing.

2) Add a mode -S to save the relocation scripts for the purpose
   of debugging them in conjunction with -D

3) Add a mode -R to not execute the relocation scripts for the
   purpose of debugging the relocations.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
---
 meta/classes/populate_sdk_base.bbclass |   40 +++++++++++++++++++++++++++----
 1 files changed, 34 insertions(+), 6 deletions(-)

diff --git a/meta/classes/populate_sdk_base.bbclass b/meta/classes/populate_sdk_base.bbclass
index c587af8..9321445 100644
--- a/meta/classes/populate_sdk_base.bbclass
+++ b/meta/classes/populate_sdk_base.bbclass
@@ -136,7 +136,10 @@ DEFAULT_INSTALL_DIR="${SDKPATH}"
 SUDO_EXEC=""
 target_sdk_dir=""
 answer=""
-while getopts ":yd:" OPT; do
+relocate=1
+savescripts=0
+verbose=0
+while getopts ":yd:DRS" OPT; do
 	case $OPT in
 	y)
 		answer="Y"
@@ -145,15 +148,33 @@ while getopts ":yd:" OPT; do
 	d)
 		target_sdk_dir=$OPTARG
 		;;
+	D)
+		verbose=1
+		;;
+	R)
+		relocate=0
+		savescripts=1
+		;;
+	S)
+		savescripts=1
+		;;
 	*)
 		echo "Usage: $(basename $0) [-y] [-d <dir>]"
 		echo "  -y         Automatic yes to all prompts"
 		echo "  -d <dir>   Install the SDK to <dir>"
+		echo "======== Advanced DEBUGGING ONLY OPTIONS ========"
+		echo "  -S         Save relocation scripts"
+		echo "  -R         Do not relocate executables"
+		echo "  -D         use set -x to see what is going on"
 		exit 1
 		;;
 	esac
 done
 
+if [ $verbose = 1 ] ; then
+	set -x
+fi
+
 printf "Enter target directory for SDK (default: $DEFAULT_INSTALL_DIR): "
 if [ "$target_sdk_dir" = "" ]; then
 	read target_sdk_dir
@@ -231,10 +252,15 @@ if [ "$dl_path" = "" ] ; then
 	exit 1
 fi
 executable_files=$($SUDO_EXEC find $native_sysroot -type f -perm +111)
-$SUDO_EXEC ${env_setup_script%/*}/relocate_sdk.py $target_sdk_dir $dl_path $executable_files
-if [ $? -ne 0 ]; then
-	echo "SDK could not be set up. Relocate script failed. Abort!"
-	exit 1
+echo "#!/bin/bash" > ${env_setup_script%/*}/relocate_sdk.sh
+echo exec $SUDO_EXEC ${env_setup_script%/*}/relocate_sdk.py $target_sdk_dir $dl_path $executable_files >> ${env_setup_script%/*}/relocate_sdk.sh
+chmod 755 ${env_setup_script%/*}/relocate_sdk.sh
+if [ $relocate = 1 ] ; then
+	${env_setup_script%/*}/relocate_sdk.sh
+	if [ $? -ne 0 ]; then
+		echo "SDK could not be set up. Relocate script failed. Abort!"
+		exit 1
+	fi
 fi
 
 # replace ${SDKPATH} with the new prefix in all text files: configs/scripts/etc
@@ -249,7 +275,9 @@ echo done
 
 # delete the relocating script, so that user is forced to re-run the installer
 # if he/she wants another location for the sdk
-$SUDO_EXEC rm ${env_setup_script%/*}/relocate_sdk.py
+if [ $savescripts = 0 ] ; then
+	$SUDO_EXEC rm ${env_setup_script%/*}/relocate_sdk.py ${env_setup_script%/*}/relocate_sdk.sh
+fi
 
 echo "SDK has been successfully set up and is ready to be used."
 
-- 
1.7.1

Re: [PATCH 1/2] relocate_sdk.py: Fix corruption of sdk binaries

[Laurentiu Palcu]

Hi Jason,

On 01/25/2013 02:38 AM, Jason Wessel wrote:
> There are two cases of corruption that the relocate_sdk.py was not correctly
> dealing with.
> 
> 1) SDK Extras should be left alone
>    Extra external binaries included in an SDK that were linked against the
>    host's version of /usr/lib/ld-so.so should not get a relocation applied.
>    In the case that was discovered these were LSB compliant binaries that
>    already worked on many hosts.
> 
> 2) If the interp section is too small generate an error
>    In the case of the qemu user code, it was using its own .ld file
>    to link the executables which overrides the default in the nativesdk
>    binutils.  This generated host executables which had a interp section
>    that was too small to relocate.
I believe there is a patch already in oe-core addressing the qemu issue:

meta/recipes-devtools/qemu/files/relocatable_sdk.patch

Aren't you guys using qemu from oe-core?

> 
>    Now the relocate_sdk.py will print an error and continue on such that
>    the error can be fixed by a developer without having to do the
>    difficult task of debugging why it is crashing or not loading correctly.
> 
> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
> ---
>  scripts/relocate_sdk.py |   17 +++++++++++++----
>  1 files changed, 13 insertions(+), 4 deletions(-)
> 
> diff --git a/scripts/relocate_sdk.py b/scripts/relocate_sdk.py
> index 74bb7a5..3e3181f 100755
> --- a/scripts/relocate_sdk.py
> +++ b/scripts/relocate_sdk.py
> @@ -66,7 +66,7 @@ def parse_elf_header():
>      e_ehsize, e_phentsize, e_phnum, e_shentsize, e_shnum, e_shstrndx =\
>          hdr_struct.unpack(elf_header[16:hdr_size])
>  
> -def change_interpreter():
> +def change_interpreter(elf_file_name):
>      if arch == 32:
>          ph_struct = struct.Struct("<IIIIIIII")
>      else:
> @@ -89,8 +89,17 @@ def change_interpreter():
>          if p_type == 3:
>              # PT_INTERP section
>              f.seek(p_offset)
> -            dl_path = new_dl_path + "\0" * (p_filesz - len(new_dl_path))
Personally, I would prefer you left the zero padding in place.
Otherwise, if installing in a location like /opt/test the .interp
section would look like below. Technically, the dynamic loader would not
care but it would be nice to have a clean .interp section, without
leftover strings in it...

This is how it would look like after relocation:
$ readelf -p .interp qemu-arm

String dump of section '.interp':
  [     0]  /opt/test/sysroots/x86_64-pokysdk-linux/lib/ld-linux-x86-64.so.2
  [    41]  -x86-64.so.2



> -            f.write(dl_path)
> +            dl_path = new_dl_path + "\0"
> +            # External SDKs with mixed pre-compiled binaries should not get
> +            # relocated so look for some variant of /lib
> +            fname = f.read(11)
> +            if fname.startswith("/lib/") or fname.startswith("/lib64/") or fname.startswith("/lib32/") or fname.startswith("/usr/lib32/") or fname.startswith("/usr/lib32/") or fname.startswith("/usr/lib64/"):
> +                break
> +            if (len(dl_path) >= p_memsz):
Do we want to check against the section memory size? I have to admit I
haven't seen, yet, any differences between p_filesz and p_memsz and the
elf manual is not very specific but I think p_filesz would give us the
size this section holds in the elf file itself. And, since we're writing
the string back, we might want to use this to make sure we have space.

Thanks,
Laurentiu

> +                print "ERROR: could not relocate %s, interp size = %i and %i is needed." % (elf_file_name, p_memsz, len(dl_path))
> +                break
> +            f.seek(p_offset)
> +            f.write(dl_path )
>              break
>  
>  def change_dl_sysdirs():
> @@ -199,7 +208,7 @@ for e in executables_list:
>      arch = get_arch()
>      if arch:
>          parse_elf_header()
> -        change_interpreter()
> +        change_interpreter(e)
>          change_dl_sysdirs()
>  
>      """ change permissions back """
> 

Re: [PATCH 2/2] populate_sdk_base.bbclass: Improve debugging capabilities for SDK installer

[Laurentiu Palcu]

On 01/25/2013 02:38 AM, Jason Wessel wrote:
>  executable_files=$($SUDO_EXEC find $native_sysroot -type f -perm +111)
> -$SUDO_EXEC ${env_setup_script%/*}/relocate_sdk.py $target_sdk_dir $dl_path $executable_files
> -if [ $? -ne 0 ]; then
> -	echo "SDK could not be set up. Relocate script failed. Abort!"
> -	exit 1


> +echo "#!/bin/bash" > ${env_setup_script%/*}/relocate_sdk.sh
> +echo exec $SUDO_EXEC ${env_setup_script%/*}/relocate_sdk.py $target_sdk_dir $dl_path $executable_files >> ${env_setup_script%/*}/relocate_sdk.sh
> +chmod 755 ${env_setup_script%/*}/relocate_sdk.sh
The last 3 lines will certainly fail if installation takes place in a
location you don't have rights... So, you'll end up with no
relocate_sdk.sh script.

Thanks,
Laurentiu
> +if [ $relocate = 1 ] ; then
> +	${env_setup_script%/*}/relocate_sdk.sh
> +	if [ $? -ne 0 ]; then
> +		echo "SDK could not be set up. Relocate script failed. Abort!"
> +		exit 1
> +	fi
>  fi

Re: [PATCH 1/2] relocate_sdk.py: Fix corruption of sdk binaries

[Jason Wessel]

[-- Attachment #1: Type: text/plain, Size: 2796 bytes --]

On 01/24/2013 06:38 PM, Jason Wessel wrote:
> There are two cases of corruption that the relocate_sdk.py was not correctly
> dealing with.
>
> 1) SDK Extras should be left alone
>    Extra external binaries included in an SDK that were linked against the
>    host's version of /usr/lib/ld-so.so should not get a relocation applied.
>    In the case that was discovered these were LSB compliant binaries that
>    already worked on many hosts.
>
> 2) If the interp section is too small generate an error
>    In the case of the qemu user code, it was using its own .ld file
>    to link the executables which overrides the default in the nativesdk
>    binutils.  This generated host executables which had a interp section
>    that was too small to relocate.
>
>    Now the relocate_sdk.py will print an error and continue on such that
>    the error can be fixed by a developer without having to do the
>    difficult task of debugging why it is crashing or not loading correctly.
>
> Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
> ---
>  scripts/relocate_sdk.py |   17 +++++++++++++----
>  1 files changed, 13 insertions(+), 4 deletions(-)
>
> diff --git a/scripts/relocate_sdk.py b/scripts/relocate_sdk.py
> index 74bb7a5..3e3181f 100755
> --- a/scripts/relocate_sdk.py
> +++ b/scripts/relocate_sdk.py
> @@ -66,7 +66,7 @@ def parse_elf_header():
>      e_ehsize, e_phentsize, e_phnum, e_shentsize, e_shnum, e_shstrndx =\
>          hdr_struct.unpack(elf_header[16:hdr_size])
>  
> -def change_interpreter():
> +def change_interpreter(elf_file_name):
>      if arch == 32:
>          ph_struct = struct.Struct("<IIIIIIII")
>      else:
> @@ -89,8 +89,17 @@ def change_interpreter():
>          if p_type == 3:
>              # PT_INTERP section
>              f.seek(p_offset)
> -            dl_path = new_dl_path + "\0" * (p_filesz - len(new_dl_path))
> -            f.write(dl_path)
> +            dl_path = new_dl_path + "\0"
> +            # External SDKs with mixed pre-compiled binaries should not get
> +            # relocated so look for some variant of /lib
> +            fname = f.read(11)
> +            if fname.startswith("/lib/") or fname.startswith("/lib64/") or fname.startswith("/lib32/") or fname.startswith("/usr/lib32/") or fname.startswith("/usr/lib32/") or fname.startswith("/usr/lib64/"):
> +                break
> +            if (len(dl_path) >= p_memsz):
> +                print "ERROR: could not relocate %s, interp size = %i and %i is needed." % (elf_file_name, p_memsz, len(dl_path))
> +                break
> +            f.seek(p_offset)
> +            f.write(dl_path )

oops...  Need a yocto check patch for python white space eh?


Attached is the fixed version.

Cheers,
Jason.

[-- Attachment #2: 0001-relocate_sdk.py-Fix-corruption-of-sdk-binaries.patch --]
[-- Type: application/mbox, Size: 2942 bytes --]