[PATCH 0/1] check bitbake wrapper/pseudo functionality

[PATCH 0/1] check bitbake wrapper/pseudo functionality

[Paul Eggleton]

From: Paul Eggleton <paul.eggleton@linux.intel.com>

This patch detects basic problems with the bitbake wrapper and pseudo.

(previously sent to poky@yoctoproject.org, forgot to CC)

Fixes [YOCTO #653]

Pull URL: git://git.pokylinux.org/poky-contrib.git
  Branch: paule/master-pseudo-sanity
  Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=paule/master-pseudo-sanity

Thanks,
    Paul Eggleton <paul.eggleton@linux.intel.com>
---


Paul Eggleton (1):
  sanity: detect if bitbake wrapper is not being used or pseudo is
    broken

 meta/classes/sanity.bbclass |   25 ++++++++++++++++++++++++-
 scripts/bitbake             |    3 +++
 scripts/poky-env-internal   |    2 +-
 3 files changed, 28 insertions(+), 2 deletions(-)

[PATCH 1/1] sanity: detect if bitbake wrapper is not being used or pseudo is broken

[Paul Eggleton]

From: Paul Eggleton <paul.eggleton@linux.intel.com>

* Shows a warning during sanity checking if the scripts/bitbake wrapper is
  not being used
* Check to see if pseudo is working during sanity checking, and if it
  isn't an error occurs (if we are using the wrapper script and pseudo
  has been built; otherwise it is a warning).

Fixes [YOCTO #653]

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
---
 meta/classes/sanity.bbclass |   25 ++++++++++++++++++++++++-
 scripts/bitbake             |    3 +++
 scripts/poky-env-internal   |    2 +-
 3 files changed, 28 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
index a6d320b..6e13d2a 100644
--- a/meta/classes/sanity.bbclass
+++ b/meta/classes/sanity.bbclass
@@ -40,7 +40,26 @@ def check_sanity_tmpdir_change(tmpdir):
 def check_sanity_version_change():
     # Sanity checks to be done when SANITY_VERSION changes
     return ""
-    
+
+def check_pseudo_wrapper():
+    import subprocess as sub
+    # Check if bitbake wrapper is being used
+    pseudo_build = os.environ.get( 'PSEUDO_BUILD' )
+    if not pseudo_build:
+        bb.warn("Bitbake has not been run using the bitbake wrapper (scripts/bitbake); this is likely because your PATH has been altered from that normally set up by the poky-init-build-env script. Not using the wrapper may result in failures during package installation, so it is highly recommended that you set your PATH back so that the wrapper script is being executed.")
+
+    if (not pseudo_build) or pseudo_build == '2':
+        # pseudo ought to be working, let's see if it is...
+        p = sub.Popen(['sh', '-c', 'PSEUDO_DISABLED=0 id -u'],stdout=sub.PIPE,stderr=sub.PIPE)
+        out, err = p.communicate()
+        if out.rstrip() != '0':
+            msg = "Pseudo is not functioning correctly, which will cause failures during package installation. Please check your configuration."
+            if pseudo_build == '2':
+                return msg
+            else:
+                bb.warn(msg)
+    return ""
+
 def check_create_long_filename(filepath, pathname):
     testfile = os.path.join(filepath, ''.join([`num`[-1] for num in xrange(1,200)]))
     try:
@@ -187,6 +206,10 @@ def check_sanity(e):
         missing = missing.rstrip(',')
         messages = messages + "Please install following missing utilities: %s\n" % missing
 
+    pseudo_msg = check_pseudo_wrapper()
+    if pseudo_msg != "":
+        messages = messages + pseudo_msg + '\n'
+
     # Check if DISPLAY is set if IMAGETEST is set
     if not data.getVar( 'DISPLAY', e.data, True ) and data.getVar( 'IMAGETEST', e.data, True ) == 'qemu':
         messages = messages + 'qemuimagetest needs a X desktop to start qemu, please set DISPLAY correctly (e.g. DISPLAY=:1.0)\n'
diff --git a/scripts/bitbake b/scripts/bitbake
index cad4dbe..29e7bcd 100755
--- a/scripts/bitbake
+++ b/scripts/bitbake
@@ -27,6 +27,7 @@ OLDPATH=$PATH
 export PATH=`echo $PATH | sed s#[^:]*/scripts:##`
 if [ $buildpseudo = "1" ]; then
     echo "Pseudo is not present but is required, building this first before the main build"
+    export PSEUDO_BUILD=1
     bitbake pseudo-native tar-replacement-native -c populate_sysroot
     ret=$?
     if [ "$ret" != "0" ]; then
@@ -44,9 +45,11 @@ fi
 BITBAKE=`which bitbake`
 export PATH=$OLDPATH
 if [ $needpseudo = "1" ]; then
+    export PSEUDO_BUILD=2
     PSEUDOBINDIR=`cat $BUILDDIR/pseudodone`
     PSEUDO_BINDIR=$PSEUDOBINDIR PSEUDO_LIBDIR=$PSEUDOBINDIR/../lib/pseudo/lib PSEUDO_PREFIX=$PSEUDOBINDIR/../../ PSEUDO_DISABLED=1 $PSEUDOBINDIR/pseudo $BITBAKE $@
 else
+    export PSEUDO_BUILD=0
     $BITBAKE $@
 fi
 ret=$?
diff --git a/scripts/poky-env-internal b/scripts/poky-env-internal
index 724a6e4..5549454 100755
--- a/scripts/poky-env-internal
+++ b/scripts/poky-env-internal
@@ -55,6 +55,6 @@ unset BITBAKEDIR
 export BUILDDIR
 export PATH
 
-export BB_ENV_EXTRAWHITE="MACHINE DISTRO POKYMODE POKYLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS GIT_PROXY_COMMAND PSEUDO_DISABLED"
+export BB_ENV_EXTRAWHITE="MACHINE DISTRO POKYMODE POKYLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS GIT_PROXY_COMMAND PSEUDO_DISABLED PSEUDO_BUILD"
 
 export BBFETCH2=True
-- 
1.7.1

Re: [poky] [PATCH 1/1] sanity: detect if bitbake wrapper is not being used or pseudo is broken

[Koen Kooi]

Op 16 mrt 2011, om 18:49 heeft Paul Eggleton het volgende geschreven:

> From: Paul Eggleton <paul.eggleton@linux.intel.com>
> 
> * Shows a warning during sanity checking if the scripts/bitbake wrapper is
>  not being used
> * Check to see if pseudo is working during sanity checking, and if it
>  isn't an error occurs (if we are using the wrapper script and pseudo
>  has been built; otherwise it is a warning).

Do really want to add *more* poky references into oe-core when the goal is to remove them?

> 
> Fixes [YOCTO #653]
> 
> Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
> meta/classes/sanity.bbclass |   25 ++++++++++++++++++++++++-
> scripts/bitbake             |    3 +++
> scripts/poky-env-internal   |    2 +-
> 3 files changed, 28 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes/sanity.bbclass b/meta/classes/sanity.bbclass
> index a6d320b..6e13d2a 100644
> --- a/meta/classes/sanity.bbclass
> +++ b/meta/classes/sanity.bbclass
> @@ -40,7 +40,26 @@ def check_sanity_tmpdir_change(tmpdir):
> def check_sanity_version_change():
>     # Sanity checks to be done when SANITY_VERSION changes
>     return ""
> -    
> +
> +def check_pseudo_wrapper():
> +    import subprocess as sub
> +    # Check if bitbake wrapper is being used
> +    pseudo_build = os.environ.get( 'PSEUDO_BUILD' )
> +    if not pseudo_build:
> +        bb.warn("Bitbake has not been run using the bitbake wrapper (scripts/bitbake); this is likely because your PATH has been altered from that normally set up by the poky-init-build-env script. Not using the wrapper may result in failures during package installation, so it is highly recommended that you set your PATH back so that the wrapper script is being executed.")
> +
> +    if (not pseudo_build) or pseudo_build == '2':
> +        # pseudo ought to be working, let's see if it is...
> +        p = sub.Popen(['sh', '-c', 'PSEUDO_DISABLED=0 id -u'],stdout=sub.PIPE,stderr=sub.PIPE)
> +        out, err = p.communicate()
> +        if out.rstrip() != '0':
> +            msg = "Pseudo is not functioning correctly, which will cause failures during package installation. Please check your configuration."
> +            if pseudo_build == '2':
> +                return msg
> +            else:
> +                bb.warn(msg)
> +    return ""
> +
> def check_create_long_filename(filepath, pathname):
>     testfile = os.path.join(filepath, ''.join([`num`[-1] for num in xrange(1,200)]))
>     try:
> @@ -187,6 +206,10 @@ def check_sanity(e):
>         missing = missing.rstrip(',')
>         messages = messages + "Please install following missing utilities: %s\n" % missing
> 
> +    pseudo_msg = check_pseudo_wrapper()
> +    if pseudo_msg != "":
> +        messages = messages + pseudo_msg + '\n'
> +
>     # Check if DISPLAY is set if IMAGETEST is set
>     if not data.getVar( 'DISPLAY', e.data, True ) and data.getVar( 'IMAGETEST', e.data, True ) == 'qemu':
>         messages = messages + 'qemuimagetest needs a X desktop to start qemu, please set DISPLAY correctly (e.g. DISPLAY=:1.0)\n'
> diff --git a/scripts/bitbake b/scripts/bitbake
> index cad4dbe..29e7bcd 100755
> --- a/scripts/bitbake
> +++ b/scripts/bitbake
> @@ -27,6 +27,7 @@ OLDPATH=$PATH
> export PATH=`echo $PATH | sed s#[^:]*/scripts:##`
> if [ $buildpseudo = "1" ]; then
>     echo "Pseudo is not present but is required, building this first before the main build"
> +    export PSEUDO_BUILD=1
>     bitbake pseudo-native tar-replacement-native -c populate_sysroot
>     ret=$?
>     if [ "$ret" != "0" ]; then
> @@ -44,9 +45,11 @@ fi
> BITBAKE=`which bitbake`
> export PATH=$OLDPATH
> if [ $needpseudo = "1" ]; then
> +    export PSEUDO_BUILD=2
>     PSEUDOBINDIR=`cat $BUILDDIR/pseudodone`
>     PSEUDO_BINDIR=$PSEUDOBINDIR PSEUDO_LIBDIR=$PSEUDOBINDIR/../lib/pseudo/lib PSEUDO_PREFIX=$PSEUDOBINDIR/../../ PSEUDO_DISABLED=1 $PSEUDOBINDIR/pseudo $BITBAKE $@
> else
> +    export PSEUDO_BUILD=0
>     $BITBAKE $@
> fi
> ret=$?
> diff --git a/scripts/poky-env-internal b/scripts/poky-env-internal
> index 724a6e4..5549454 100755
> --- a/scripts/poky-env-internal
> +++ b/scripts/poky-env-internal
> @@ -55,6 +55,6 @@ unset BITBAKEDIR
> export BUILDDIR
> export PATH
> 
> -export BB_ENV_EXTRAWHITE="MACHINE DISTRO POKYMODE POKYLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS GIT_PROXY_COMMAND PSEUDO_DISABLED"
> +export BB_ENV_EXTRAWHITE="MACHINE DISTRO POKYMODE POKYLIBC http_proxy ftp_proxy https_proxy all_proxy ALL_PROXY no_proxy SSH_AGENT_PID SSH_AUTH_SOCK BB_SRCREV_POLICY SDKMACHINE BB_NUMBER_THREADS GIT_PROXY_COMMAND PSEUDO_DISABLED PSEUDO_BUILD"
> 
> export BBFETCH2=True
> -- 
> 1.7.1
> 
> _______________________________________________
> poky mailing list
> poky@yoctoproject.org
> https://lists.yoctoproject.org/listinfo/poky

Re: [poky] [PATCH 1/1] sanity: detect if bitbake wrapper is not being used or pseudo is broken

[Richard Purdie]

On Wed, 2011-03-16 at 19:25 +0100, Koen Kooi wrote:
> Op 16 mrt 2011, om 18:49 heeft Paul Eggleton het volgende geschreven:
> 
> > From: Paul Eggleton <paul.eggleton@linux.intel.com>
> > 
> > * Shows a warning during sanity checking if the scripts/bitbake wrapper is
> >  not being used
> > * Check to see if pseudo is working during sanity checking, and if it
> >  isn't an error occurs (if we are using the wrapper script and pseudo
> >  has been built; otherwise it is a warning).
> 
> Do really want to add *more* poky references into oe-core when the goal is to remove them?

There are two partially competing objectives at the moment, to get a
high quality Yocto 1.0 released and to work on OECore. Doing both at the
same time is proving rather tricky but its not doing too badly.

In this case the script is still called "poky-init-build-env" in both.
Referencing a script that doesn't exist isn't going to help anyone and
that is the only reference I could spot so I don't think the patch is
unreasonable. It also addresses a serious usability issue people do run
into.

I'm not going to block any attempts rename things *despite* the horrible
amounts of pain its likely to cause me juggling patches for the 1.0
release. Nobody has sent those patches but I think the hesitation is
probably warranted. Why?

I still don't think the TSC has figured out crisply what should and
shouldn't be in OE-Core, whether any scripts should be there period,
whether they are purely a reference or whether we encourage people to
use them. Same question for distro config, I moved them out the way in
the poky repo but I get the feeling we're not ready to do that in OECore
until we split the config into pieces which we might be able to share
some of. 

I starting to think the best way to do this may be to sit together at
the "TSC" meeting before ELC and actually write the commits there and
then on a projector screen to get this figured out.

Cheers,

Richard

Re: [PATCH 0/1] check bitbake wrapper/pseudo functionality

[Richard Purdie]

On Wed, 2011-03-16 at 17:52 +0000, Paul Eggleton wrote:
> From: Paul Eggleton <paul.eggleton@linux.intel.com>
> 
> This patch detects basic problems with the bitbake wrapper and pseudo.
> 
> (previously sent to poky@yoctoproject.org, forgot to CC)
> 
> Fixes [YOCTO #653]
> 
> Pull URL: git://git.pokylinux.org/poky-contrib.git
>   Branch: paule/master-pseudo-sanity
>   Browse: http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=paule/master-pseudo-sanity
> 
> Thanks,
>     Paul Eggleton <paul.eggleton@linux.intel.com>
> ---
> 
> 
> Paul Eggleton (1):
>   sanity: detect if bitbake wrapper is not being used or pseudo is
>     broken

Merged to master, thanks.

Richard