[PATCH 0/3] Fixes for bitbake wrapper script

[PATCH 0/3] Fixes for bitbake wrapper script

[Paul Eggleton]

Andreas, if you could review these that would be much appreciated.


The following changes since commit 644b7503c37fd73730dd3d7841463b158b8934ed:

  guile: Deal with hardcoded path issues (2012-03-27 00:28:41 +0100)

are available in the git repository at:
  git://git.openembedded.org/openembedded-core-contrib paule/builddir
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/builddir

Paul Eggleton (3):
  scripts/bitbake: try harder to check if pseudo exists
  scripts/bitbake: allow switching between build directories
  scripts/bitbake: add/fix some comments

 scripts/bitbake |   43 ++++++++++++++++++++++++++++++++-----------
 1 files changed, 32 insertions(+), 11 deletions(-)

-- 
1.7.5.4

[PATCH 1/3] scripts/bitbake: try harder to check if pseudo exists

[Paul Eggleton]

If pseudodone doesn't exist, we can get STAGING_BINDIR_NATIVE by calling
bitbake -e and use that as the path to check for pseudo before we give
up and try to build it explicitly first.

This is useful for people who share TMPDIR between multiple build
directories.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/bitbake |   12 +++++++++---
 1 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/scripts/bitbake b/scripts/bitbake
index 45c8697..1ba1839 100755
--- a/scripts/bitbake
+++ b/scripts/bitbake
@@ -53,14 +53,21 @@ if [ "`pwd`" != "$BUILDDIR" ] ; then
 fi
 
 buildpseudo="1"
-if [ $needpseudo = "1" ] && [ -e "$BUILDDIR/pseudodone" ]; then
-    PSEUDOBINDIR=`cat $BUILDDIR/pseudodone`
+if [ $needpseudo = "1" ]; then
+    if [ -e "$BUILDDIR/pseudodone" ]; then
+        PSEUDOBINDIR=`cat $BUILDDIR/pseudodone`
+    else
+        PSEUDOBINDIR=`bitbake -e | grep STAGING_BINDIR_NATIVE=\" | cut -d '=' -f2 | cut -d '"' -f2`
+    fi
     if [ -e "$PSEUDOBINDIR/pseudo" -a -e "$PSEUDOBINDIR/tar" -a "$needtar" = "1" ]; then
         buildpseudo="0"
     fi
     if [ -e "$PSEUDOBINDIR/pseudo" -a $needtar = "0" ]; then
         buildpseudo="0"
     fi
+    if [ $buildpseudo = "0" -a ! -e "$BUILDDIR/pseudodone" ] ; then
+        echo $PSEUDOBINDIR > $BUILDDIR/pseudodone
+    fi
 fi
 if [ $needpseudo = "0" ]; then
     buildpseudo="0"
@@ -104,7 +111,6 @@ BITBAKE=`which bitbake`
 export PATH=$OLDPATH
 if [ $needpseudo = "1" ]; then
     export PSEUDO_BUILD=2
-    PSEUDOBINDIR=`cat $BUILDDIR/pseudodone`
     PSEUDO_BINDIR=$PSEUDOBINDIR PSEUDO_LIBDIR=$PSEUDOBINDIR/../lib/pseudo/lib PSEUDO_PREFIX=$PSEUDOBINDIR/../../ PSEUDO_DISABLED=1 $PSEUDOBINDIR/pseudo $BITBAKE $@
 else
     export PSEUDO_BUILD=0
-- 
1.7.5.4

[PATCH 2/3] scripts/bitbake: allow switching between build directories

[Paul Eggleton]

The recent addition of the check to ensure the user was in their build
directory disabled the ability to switch between build directories
without re-running the build environment setup script. We can rely
upon checking for conf/bblayers.conf instead, so use this check.

This does allow BUILDDIR (which is normally set by the environment
script) to be unset; however if it is set then it is assumed to be the
correct build directory and will be used in the error message that is
shown when we can't find conf/bblayers.conf.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/bitbake |   15 ++++++++++-----
 1 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/scripts/bitbake b/scripts/bitbake
index 1ba1839..f02e89b 100755
--- a/scripts/bitbake
+++ b/scripts/bitbake
@@ -37,6 +37,16 @@ if [ "$py_v26_check" != "True" ]; then
 	exit 1
 fi
 
+if [ ! -e conf/bblayers.conf ] ; then
+    BDPRINT=""
+    [ -n "$BUILDDIR" ] && BDPRINT=": $BUILDDIR"
+    echo "Unable to find conf/bblayers.conf"
+    echo "BitBake must be run from within your build directory$BDPRINT"
+    exit 1
+elif [ -z "$BUILDDIR" ] ; then
+    BUILDDIR="`pwd`"
+fi
+
 needtar="1"
 TARVERSION=`tar --version | head -n 1 | cut -d ' ' -f 4`
 float_test() {
@@ -47,11 +57,6 @@ float_test() {
 # but earlier versions do not
 float_test "$TARVERSION > 1.23" && needtar="0"
 
-if [ "`pwd`" != "$BUILDDIR" ] ; then
-    echo "BitBake must be run from your build directory: $BUILDDIR"
-    exit 1
-fi
-
 buildpseudo="1"
 if [ $needpseudo = "1" ]; then
     if [ -e "$BUILDDIR/pseudodone" ]; then
-- 
1.7.5.4

[PATCH 3/3] scripts/bitbake: add/fix some comments

[Paul Eggleton]

Add some comments explaining what this script does, fix one grammatical
error in a comment and make the tar-replacement-native comment give the
full reason why it is needed.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 scripts/bitbake |   16 +++++++++++++---
 1 files changed, 13 insertions(+), 3 deletions(-)

diff --git a/scripts/bitbake b/scripts/bitbake
index f02e89b..a8adeb7 100755
--- a/scripts/bitbake
+++ b/scripts/bitbake
@@ -1,10 +1,20 @@
 #!/bin/sh
 
+# This is the bitbake wrapper script that ensures everything is set up
+# correctly in the environment, builds pseudo separately if it hasn't
+# already been built, and then runs bitbake within pseudo.
+
 export BBFETCH2=True
 export BB_ENV_EXTRAWHITE="PSEUDO_BUILD PSEUDO_DISABLED $BB_ENV_EXTRAWHITE"
 
+# For certain operations (i.e. that won't be actually running any tasks)
+# we don't need pseudo
 NO_BUILD_OPTS="--version -h --help -p --parse-only -s --show-versions -e --environment -g --graphviz"
+
+# Some options are useful to pass through to the initial pseudo build if
+# that needs to be run (for debugging)
 PASSTHROUGH_OPTS="-D -DD -DDD -DDDD -v"
+
 needpseudo="1"
 for opt in $@; do
 for key in $NO_BUILD_OPTS; do
@@ -29,7 +39,7 @@ if [ "$py_v3_check" != "" ]; then
 fi
 
 # Similarly, we now have code that doesn't parse correctly with older
-# versions of Python, and rather than fixing that and be eternally
+# versions of Python, and rather than fixing that and being eternally
 # vigilant for any other new feature use, just check the version here.
 py_v26_check=`python -c 'import sys; print sys.version_info >= (2,6,0)'`
 if [ "$py_v26_check" != "True" ]; then
@@ -53,8 +63,8 @@ float_test() {
      echo | awk 'END { exit ( !( '"$1"')); }'
 }
 
-# Tar version 1.24 and onwards handle symlinks in sstate packages correctly
-# but earlier versions do not
+# Tar version 1.24 and onwards handle overwriting symlinks correctly
+# but earlier versions do not; this needs to work properly for sstate
 float_test "$TARVERSION > 1.23" && needtar="0"
 
 buildpseudo="1"
-- 
1.7.5.4

Re: [PATCH 1/3] scripts/bitbake: try harder to check if pseudo exists

[Andreas Oberritter]

On 27.03.2012 13:50, Paul Eggleton wrote:
> If pseudodone doesn't exist, we can get STAGING_BINDIR_NATIVE by calling
> bitbake -e and use that as the path to check for pseudo before we give
> up and try to build it explicitly first.
> 
> This is useful for people who share TMPDIR between multiple build
> directories.
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>

This adds a 3s delay over the previous setup on my build system, which I 
deem acceptable. See one minor suggestion inline.

Acked-by: Andreas Oberritter <obi@opendreambox.org>

> ---
>  scripts/bitbake |   12 +++++++++---
>  1 files changed, 9 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/bitbake b/scripts/bitbake
> index 45c8697..1ba1839 100755
> --- a/scripts/bitbake
> +++ b/scripts/bitbake
> @@ -53,14 +53,21 @@ if [ "`pwd`" != "$BUILDDIR" ] ; then
>  fi
>  
>  buildpseudo="1"
> -if [ $needpseudo = "1" ] && [ -e "$BUILDDIR/pseudodone" ]; then
> -    PSEUDOBINDIR=`cat $BUILDDIR/pseudodone`
> +if [ $needpseudo = "1" ]; then
> +    if [ -e "$BUILDDIR/pseudodone" ]; then
> +        PSEUDOBINDIR=`cat $BUILDDIR/pseudodone`
> +    else
> +        PSEUDOBINDIR=`bitbake -e | grep STAGING_BINDIR_NATIVE=\" | cut -d '=' -f2 | cut -d '"' -f2`

I think using sed might improve readability:

bitbake -e | grep ^STAGING_BINDIR_NATIVE= | sed -e 's,^STAGING_BINDIR_NATIVE="\(.*\)"$,\1,'

This may be a subjective impression, though.

> +    fi
>      if [ -e "$PSEUDOBINDIR/pseudo" -a -e "$PSEUDOBINDIR/tar" -a "$needtar" = "1" ]; then
>          buildpseudo="0"
>      fi
>      if [ -e "$PSEUDOBINDIR/pseudo" -a $needtar = "0" ]; then
>          buildpseudo="0"
>      fi
> +    if [ $buildpseudo = "0" -a ! -e "$BUILDDIR/pseudodone" ] ; then
> +        echo $PSEUDOBINDIR > $BUILDDIR/pseudodone
> +    fi
>  fi
>  if [ $needpseudo = "0" ]; then
>      buildpseudo="0"
> @@ -104,7 +111,6 @@ BITBAKE=`which bitbake`
>  export PATH=$OLDPATH
>  if [ $needpseudo = "1" ]; then
>      export PSEUDO_BUILD=2
> -    PSEUDOBINDIR=`cat $BUILDDIR/pseudodone`
>      PSEUDO_BINDIR=$PSEUDOBINDIR PSEUDO_LIBDIR=$PSEUDOBINDIR/../lib/pseudo/lib PSEUDO_PREFIX=$PSEUDOBINDIR/../../ PSEUDO_DISABLED=1 $PSEUDOBINDIR/pseudo $BITBAKE $@
>  else
>      export PSEUDO_BUILD=0

Re: [PATCH 2/3] scripts/bitbake: allow switching between build directories

[Andreas Oberritter]

On 27.03.2012 13:50, Paul Eggleton wrote:
> The recent addition of the check to ensure the user was in their build
> directory disabled the ability to switch between build directories
> without re-running the build environment setup script. We can rely
> upon checking for conf/bblayers.conf instead, so use this check.
> 
> This does allow BUILDDIR (which is normally set by the environment
> script) to be unset; however if it is set then it is assumed to be the
> correct build directory and will be used in the error message that is
> shown when we can't find conf/bblayers.conf.
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>

Acked-by: Andreas Oberritter <obi@opendreambox.org>

> ---
>  scripts/bitbake |   15 ++++++++++-----
>  1 files changed, 10 insertions(+), 5 deletions(-)
> 
> diff --git a/scripts/bitbake b/scripts/bitbake
> index 1ba1839..f02e89b 100755
> --- a/scripts/bitbake
> +++ b/scripts/bitbake
> @@ -37,6 +37,16 @@ if [ "$py_v26_check" != "True" ]; then
>  	exit 1
>  fi
>  
> +if [ ! -e conf/bblayers.conf ] ; then
> +    BDPRINT=""
> +    [ -n "$BUILDDIR" ] && BDPRINT=": $BUILDDIR"
> +    echo "Unable to find conf/bblayers.conf"
> +    echo "BitBake must be run from within your build directory$BDPRINT"
> +    exit 1
> +elif [ -z "$BUILDDIR" ] ; then
> +    BUILDDIR="`pwd`"
> +fi
> +
>  needtar="1"
>  TARVERSION=`tar --version | head -n 1 | cut -d ' ' -f 4`
>  float_test() {
> @@ -47,11 +57,6 @@ float_test() {
>  # but earlier versions do not
>  float_test "$TARVERSION > 1.23" && needtar="0"
>  
> -if [ "`pwd`" != "$BUILDDIR" ] ; then
> -    echo "BitBake must be run from your build directory: $BUILDDIR"
> -    exit 1
> -fi
> -
>  buildpseudo="1"
>  if [ $needpseudo = "1" ]; then
>      if [ -e "$BUILDDIR/pseudodone" ]; then

Re: [PATCH 3/3] scripts/bitbake: add/fix some comments

[Andreas Oberritter]

On 27.03.2012 13:50, Paul Eggleton wrote:
> Add some comments explaining what this script does, fix one grammatical
> error in a comment and make the tar-replacement-native comment give the
> full reason why it is needed.
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>

Acked-by: Andreas Oberritter <obi@opendreambox.org>

> ---
>  scripts/bitbake |   16 +++++++++++++---
>  1 files changed, 13 insertions(+), 3 deletions(-)
> 
> diff --git a/scripts/bitbake b/scripts/bitbake
> index f02e89b..a8adeb7 100755
> --- a/scripts/bitbake
> +++ b/scripts/bitbake
> @@ -1,10 +1,20 @@
>  #!/bin/sh
>  
> +# This is the bitbake wrapper script that ensures everything is set up
> +# correctly in the environment, builds pseudo separately if it hasn't
> +# already been built, and then runs bitbake within pseudo.
> +
>  export BBFETCH2=True
>  export BB_ENV_EXTRAWHITE="PSEUDO_BUILD PSEUDO_DISABLED $BB_ENV_EXTRAWHITE"
>  
> +# For certain operations (i.e. that won't be actually running any tasks)
> +# we don't need pseudo
>  NO_BUILD_OPTS="--version -h --help -p --parse-only -s --show-versions -e --environment -g --graphviz"
> +
> +# Some options are useful to pass through to the initial pseudo build if
> +# that needs to be run (for debugging)
>  PASSTHROUGH_OPTS="-D -DD -DDD -DDDD -v"
> +
>  needpseudo="1"
>  for opt in $@; do
>  for key in $NO_BUILD_OPTS; do
> @@ -29,7 +39,7 @@ if [ "$py_v3_check" != "" ]; then
>  fi
>  
>  # Similarly, we now have code that doesn't parse correctly with older
> -# versions of Python, and rather than fixing that and be eternally
> +# versions of Python, and rather than fixing that and being eternally
>  # vigilant for any other new feature use, just check the version here.
>  py_v26_check=`python -c 'import sys; print sys.version_info >= (2,6,0)'`
>  if [ "$py_v26_check" != "True" ]; then
> @@ -53,8 +63,8 @@ float_test() {
>       echo | awk 'END { exit ( !( '"$1"')); }'
>  }
>  
> -# Tar version 1.24 and onwards handle symlinks in sstate packages correctly
> -# but earlier versions do not
> +# Tar version 1.24 and onwards handle overwriting symlinks correctly
> +# but earlier versions do not; this needs to work properly for sstate
>  float_test "$TARVERSION > 1.23" && needtar="0"
>  
>  buildpseudo="1"

Re: [PATCH 1/3] scripts/bitbake: try harder to check if pseudo exists

[Paul Eggleton]

On Tuesday 27 March 2012 15:08:28 Andreas Oberritter wrote:
> On 27.03.2012 13:50, Paul Eggleton wrote:
> > +        PSEUDOBINDIR=`bitbake -e | grep STAGING_BINDIR_NATIVE=\" | cut -d
> > '=' -f2 | cut -d '"' -f2`
> I think using sed might improve readability:
> 
> bitbake -e | grep ^STAGING_BINDIR_NATIVE= | sed -e
> 's,^STAGING_BINDIR_NATIVE="\(.*\)"$,\1,'
> 
> This may be a subjective impression, though.

I just copied this from further down in the script. Personally I find the 
original more readable although the sed version spawns less processes.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/3] Fixes for bitbake wrapper script

[Paul Eggleton]

On Tuesday 27 March 2012 12:50:07 Paul Eggleton wrote:
> Andreas, if you could review these that would be much appreciated.
> 
> 
> The following changes since commit 644b7503c37fd73730dd3d7841463b158b8934ed:
> 
>   guile: Deal with hardcoded path issues (2012-03-27 00:28:41 +0100)
> 
> are available in the git repository at:
>   git://git.openembedded.org/openembedded-core-contrib paule/builddir
>  
> http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paul
> e/builddir
> 
> Paul Eggleton (3):
>   scripts/bitbake: try harder to check if pseudo exists
>   scripts/bitbake: allow switching between build directories
>   scripts/bitbake: add/fix some comments
> 
>  scripts/bitbake |   43 ++++++++++++++++++++++++++++++++-----------
>  1 files changed, 32 insertions(+), 11 deletions(-)

FYI I've rebased this patch on top of Mark's recent bitbake wrapper script 
changes that are now in master.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: [PATCH 0/3] Fixes for bitbake wrapper script

[Saul Wold]

On 03/27/2012 04:50 AM, Paul Eggleton wrote:
> Andreas, if you could review these that would be much appreciated.
>
>
> The following changes since commit 644b7503c37fd73730dd3d7841463b158b8934ed:
>
>    guile: Deal with hardcoded path issues (2012-03-27 00:28:41 +0100)
>
> are available in the git repository at:
>    git://git.openembedded.org/openembedded-core-contrib paule/builddir
>    http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=paule/builddir
>
> Paul Eggleton (3):
>    scripts/bitbake: try harder to check if pseudo exists
>    scripts/bitbake: allow switching between build directories
>    scripts/bitbake: add/fix some comments
>
>   scripts/bitbake |   43 ++++++++++++++++++++++++++++++++-----------
>   1 files changed, 32 insertions(+), 11 deletions(-)
>

Merged into OE-Core

Thanks
	Sau!