python-native and autotools

python-native and autotools

[Randy MacLeod]

autoconf and automake build as part of the "core" tools that get built 
before any target packages are built whereas python-native isn't in
that set. When automake runs, it uses whatever python it finds.
This will be the python-native version if it has been built. When
doing parallel builds, it is possible to be building the python-native
while automake is running for some other package.  This can cause a
build failure with a signature such as:

   NOTE: package freetype-2.4.9-r0: task do_configure: Succeeded
   NOTE: Running task 1891 of 4319 (ID: 3642, /.../oe-core/meta/
   recipes-graphics/freetype/freetype_2.4.9.bb, do_compile)
   NOTE: Running task 1892 of 4319 (ID: 2327, /.../oe-core/meta/
   recipes-extended/zypper/zypper_git.bb, do_populate_lic)
   ERROR: Function failed: do_configure (see /.../tmp/work/
   x86_64-linux/guile-native-2.0.3-r5/temp/do_configure
   /log.do_configure.1659 for further information)
   ...
   Log data follows:
   | DEBUG: Executing python function sysroot_cleansstate
   | DEBUG: Python function sysroot_cleansstate finished
   --
   | python: error while loading shared libraries: libpython2.7.so.1.0:
   cannot open shared object file: No such file or directory
   | ERROR: Function failed: do_configure (see /.../tmp/work/
   x86_64-linux/guile-native-2.0.3-r5/temp/do_configure
   /log.do_configure.1659 for further information)
   NOTE: package guile-native-2.0.3-r5: task do_configure: Failed
   ...

Other signatures involve missing python files, such as:
   .../tmp/sysroots/x86_64-linux/usr/lib/python2.7/warnings.py",
   line 6, in <module>
     import linecache
   ImportError: No module named linecache

Changing the python install to be atomic might cover this up but
that isn't a very satisfying solution. It would be nice to make
anything that USES autotools depend on python-native, but this
causes problems because autoconf and automake inherit the autotools
class, so they would wind up depending on python, which, itself
inherits autotools.

A solution might be to have autoconf and automake and python inherit a 
version of autotools that does not add a python-native dependency, say
autotools-no-python, and then modify the "real" autotools class to
add the python dependency.

My build stats show that this problem
happens in less than 1 out of 40 builds with:
BB_NUMBER_THREADS = "20"
PARALLEL_MAKE = "-j 20"
Has anyone else seen this problem?

I'm still coming up to speed on bitbake's bootstrap process so
any suggestions would be appreciated.


-- 
# Randy MacLeod. MTS, Linux, Wind River
Direct: 613.963.1350

Re: python-native and autotools

[Khem Raj]

On Thu, Jun 28, 2012 at 12:09 PM, Randy MacLeod
<randy.macleod@windriver.com> wrote:
>
> A solution might be to have autoconf and automake and python inherit a
> version of autotools that does not add a python-native dependency, say
> autotools-no-python, and then modify the "real" autotools class to
> add the python dependency.

why is python depending on autotools that seems wrong to me.

Re: python-native and autotools

[Richard Purdie]

On Thu, 2012-06-28 at 15:09 -0400, Randy MacLeod wrote:
> 
> autoconf and automake build as part of the "core" tools that get built 
> before any target packages are built whereas python-native isn't in
> that set. When automake runs, it uses whatever python it finds.
> This will be the python-native version if it has been built. When
> doing parallel builds, it is possible to be building the python-native
> while automake is running for some other package.  This can cause a
> build failure with a signature such as:
> 
>    NOTE: package freetype-2.4.9-r0: task do_configure: Succeeded
>    NOTE: Running task 1891 of 4319 (ID: 3642, /.../oe-core/meta/
>    recipes-graphics/freetype/freetype_2.4.9.bb, do_compile)
>    NOTE: Running task 1892 of 4319 (ID: 2327, /.../oe-core/meta/
>    recipes-extended/zypper/zypper_git.bb, do_populate_lic)
>    ERROR: Function failed: do_configure (see /.../tmp/work/
>    x86_64-linux/guile-native-2.0.3-r5/temp/do_configure
>    /log.do_configure.1659 for further information)
>    ...
>    Log data follows:
>    | DEBUG: Executing python function sysroot_cleansstate
>    | DEBUG: Python function sysroot_cleansstate finished
>    --
>    | python: error while loading shared libraries: libpython2.7.so.1.0:
>    cannot open shared object file: No such file or directory
>    | ERROR: Function failed: do_configure (see /.../tmp/work/
>    x86_64-linux/guile-native-2.0.3-r5/temp/do_configure
>    /log.do_configure.1659 for further information)
>    NOTE: package guile-native-2.0.3-r5: task do_configure: Failed
>    ...
> 
> Other signatures involve missing python files, such as:
>    .../tmp/sysroots/x86_64-linux/usr/lib/python2.7/warnings.py",
>    line 6, in <module>
>      import linecache
>    ImportError: No module named linecache
> 
> Changing the python install to be atomic might cover this up but
> that isn't a very satisfying solution. It would be nice to make
> anything that USES autotools depend on python-native, but this
> causes problems because autoconf and automake inherit the autotools
> class, so they would wind up depending on python, which, itself
> inherits autotools.
> 
> A solution might be to have autoconf and automake and python inherit a 
> version of autotools that does not add a python-native dependency, say
> autotools-no-python, and then modify the "real" autotools class to
> add the python dependency.
> 
> My build stats show that this problem
> happens in less than 1 out of 40 builds with:
> BB_NUMBER_THREADS = "20"
> PARALLEL_MAKE = "-j 20"
> Has anyone else seen this problem?
> 
> I'm still coming up to speed on bitbake's bootstrap process so
> any suggestions would be appreciated.

We need python installed on the system in order to run bitbake. There
are some things like python itself that have a requirement on
python-native specifically.

The way we solved this same problem with perl was to install it into a
specific directory which we only add to PATH when we really want the
native version. See perlnative.bbclass for an example.

It sounds like we might have to consider something similar for python.

Cheers,

Richard