Building one package needs header from another

Building one package needs header from another

[Paul D. DeRocco]

I'm trying to build python-matplotlib, which is a plotting package for
PyGTK, and I need to compile the GDK rendering backend so that I can put
plots onto the screen. (This is for a Gumstix, and I'm using their Danny
branch metadata.) The backend source file needs to include pygtk/pygtk.h,
which is all over the place in the fragments of the target rootfs since
I've built that for the target system, but isn't anywhere to be found in
the cross-compilation environment, so the compilation fails.

The whole topic of cross-compilation makes me dizzy. How is this situation
dealt with? The source tarball for matplotlib (1.1.0) just assumes that
this header file already exists on the user's system. Is this a normal
thing for which there's a routine solution in Yocto? Or do I have to hack
it by copying one of those files from the target rootfs and including it
in my own metadata, forcibly putting it somewhere that the compiler can
find it?

-- 

Ciao,               Paul D. DeRocco
Paul                mailto:pderocco@ix.netcom.com 
 

Re: Building one package needs header from another

[Burton, Ross]

On 20 August 2013 08:29, Paul D. DeRocco <pderocco@ix.netcom.com> wrote:
> I'm trying to build python-matplotlib, which is a plotting package for
> PyGTK, and I need to compile the GDK rendering backend so that I can put
> plots onto the screen. (This is for a Gumstix, and I'm using their Danny
> branch metadata.) The backend source file needs to include pygtk/pygtk.h,
> which is all over the place in the fragments of the target rootfs since
> I've built that for the target system, but isn't anywhere to be found in
> the cross-compilation environment, so the compilation fails.

If you've built pygtk then the target sysroot should have the headers
in, and for me it does:

ross@melchett /data/poky-master/tmp/sysroots/genericx86
$ find . -name pygtk.h
./usr/include/pygtk-2.0/pygtk/pygtk.h

This is probably a problem with python-matplotlib, can you share the
configure and build logs?  It's probably looking in the wrong place.

Ross

Re: Building one package needs header from another

[Paul D. DeRocco]

> From: Burton, Ross [mailto:ross.burton@intel.com] 
> 
> If you've built pygtk then the target sysroot should have the headers
> in, and for me it does:
> 
> ross@melchett /data/poky-master/tmp/sysroots/genericx86
> $ find . -name pygtk.h
> ./usr/include/pygtk-2.0/pygtk/pygtk.h
> 
> This is probably a problem with python-matplotlib, can you share the
> configure and build logs?  It's probably looking in the wrong place.

Yes, the include files are there. I'm only beginning to grasp a little
about how builds work, and that sysroots/overo is the context in which the
Gumstix cross tools run. But the appropriate directory,
/home/pauld/yocto/build/tmp/sysroots/overo/usr/include/pygtk-2.0, is not
on the compiler command line, and that causes the error.

I've been trying to figure out how the setup.py/setup.cfg (and distutils)
stuff works. The setup.cfg file lists only one possible option for adding
directories, which is basedirlist, but setting that to foo adds
foo/include to the include directories and foo/lib to the library
directories, so that's not appropriate. I tried just setting include_dirs
to the proper directory in the [directories] section, but it had no
effect.

The configure log shows nothing, but the compile log (which shows the
compile error at the end), shows this DEBUG note right at the beginning:

/home/pauld/yocto/build/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/python
-matplotlib-1.1.0-r3/temp/run.do_compile.14201: line 82:
/home/pauld/yocto/build/tmp/sysroots/i686-linux/usr/bin/python: No such
file or directory
basedirlist is: ['/home/pauld/yocto/build/tmp/sysroots/overo/usr/lib']

The message refers to this script in run.do_compile (slightly
reformatted):

do_compile() {
  BUILD_SYS=i686-linux HOST_SYS=arm-poky-linux-gnueabi \
  /home/pauld/yocto/build/tmp/sysroots/i686-linux/usr/bin/python \
      setup.py build || true
  distutils_do_compile
}

This would suggest that setup.py isn't even being run. Yet when I
accidentally put a syntax error into setup.cfg, it barfed. Is setup.cfg
read before setup.py is run, by something else?

So there are two questions: why is there no python in that directory?
(There is a python-native subdirectory containing python.) And once that's
dealt with, how does one add a specific include subdirectory to the
command line via setup.cfg?

Or maybe those aren't the questions. I understand about 0.1% of what's
going on here.

I'm also curious if distutils is something that is used throughout the
bitbake process, or is it something specific to building Python-related
stuff? Can the setup.py/setup.cfg mechanism be used in any recipes?

-- 

Ciao,               Paul D. DeRocco
Paul                mailto:pderocco@ix.netcom.com 

Re: Building one package needs header from another

[Paul Eggleton]

Hi Paul,

On Tuesday 20 August 2013 16:06:54 Paul D. DeRocco wrote:
> > From: Burton, Ross [mailto:ross.burton@intel.com]
> > If you've built pygtk then the target sysroot should have the headers
> > in, and for me it does:
> > 
> > ross@melchett /data/poky-master/tmp/sysroots/genericx86
> > $ find . -name pygtk.h
> > ./usr/include/pygtk-2.0/pygtk/pygtk.h
> > 
> > This is probably a problem with python-matplotlib, can you share the
> > configure and build logs?  It's probably looking in the wrong place.
> 
> Yes, the include files are there. I'm only beginning to grasp a little
> about how builds work, and that sysroots/overo is the context in which the
> Gumstix cross tools run. But the appropriate directory,
> /home/pauld/yocto/build/tmp/sysroots/overo/usr/include/pygtk-2.0, is not
> on the compiler command line, and that causes the error.
> 
> I've been trying to figure out how the setup.py/setup.cfg (and distutils)
> stuff works. The setup.cfg file lists only one possible option for adding
> directories, which is basedirlist, but setting that to foo adds
> foo/include to the include directories and foo/lib to the library
> directories, so that's not appropriate. 

Could you just send it ${STAGING_DIR_HOST}/${prefix} ? We do do that elsewhere 
with similar-behaving configure scripts.

> The configure log shows nothing, but the compile log (which shows the
> compile error at the end), shows this DEBUG note right at the beginning:
> 
> /home/pauld/yocto/build/tmp/work/armv7a-vfp-neon-poky-linux-gnueabi/python
> -matplotlib-1.1.0-r3/temp/run.do_compile.14201: line 82:
> /home/pauld/yocto/build/tmp/sysroots/i686-linux/usr/bin/python: No such
> file or directory
> basedirlist is: ['/home/pauld/yocto/build/tmp/sysroots/overo/usr/lib']
> 
> The message refers to this script in run.do_compile (slightly
> reformatted):
> 
> do_compile() {
>   BUILD_SYS=i686-linux HOST_SYS=arm-poky-linux-gnueabi \
>   /home/pauld/yocto/build/tmp/sysroots/i686-linux/usr/bin/python \
>       setup.py build || true
>   distutils_do_compile
> }
> 
> This would suggest that setup.py isn't even being run. Yet when I
> accidentally put a syntax error into setup.cfg, it barfed. Is setup.cfg
> read before setup.py is run, by something else?
> 
> So there are two questions: why is there no python in that directory?
> (There is a python-native subdirectory containing python.)

To be honest, I'm not sure if it is appropriate, but you could inherit 
pythonnative in the recipe, that might fix this.

> I'm also curious if distutils is something that is used throughout the
> bitbake process, or is it something specific to building Python-related
> stuff? Can the setup.py/setup.cfg mechanism be used in any recipes?

distutils is python-specific, so no.

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre

Re: Building one package needs header from another

[Paul D. DeRocco]

> > On Tuesday 20 August 2013 16:06:54 Paul D. DeRocco wrote:
> > 
> > I've been trying to figure out how the setup.py/setup.cfg 
> > (and distutils)
> > stuff works. The setup.cfg file lists only one possible 
> > option for adding
> > directories, which is basedirlist, but setting that to foo adds
> > foo/include to the include directories and foo/lib to the library
> > directories, so that's not appropriate. 

> From: Paul Eggleton
> 
> Could you just send it ${STAGING_DIR_HOST}/${prefix} ? We do 
> do that elsewhere with similar-behaving configure scripts.

I think I've figured out what's going on. The whole distutils.bbclass
interface to the Python distutils package involves the writing of a
setup.py script to perform all the packaging and unpackaging processes.
The setup.py script for matplotlib has to deal with the fact that
matplotlib depends upon a varying set of libraries depending upon how you
configure it. In this case, because I want to build the GTK rendering
backend, it depends upon the pygtk library. It cleverly uses the
pkg-config command to interrogate each dependent library to find out what
compiler options are needed to compile modules that refer to the library,
but unfortunately, pygtk-2.0 is not a "library" that can be searched by
pkg-config, so the do_compile script coughs up the following nonfatal
error message:

    pkg-config: looking for pygtk-2.0 gtk+-2.0
        * Package pygtk-2.0 was not found in the pkg-config
        * search path. Perhaps you should add the directory
        * containing `pygtk-2.0.pc' to the PKG_CONFIG_PATH
        * environment variable No package 'pygtk-2.0' found

I think I'm just going to have to patch the setup.py file (actually a
subsidiary setupext.py file) to hack the additional pygtk-2.0 include
subdirectory into it. However, setupext.py and setup.py both already have
small patches in the original matplotlib recipe. If I add another patch in
my bbappend, will it always be applied after the one in the original
recipe? Or should I just replace the patch file with one containing its
fixes and my fix?

-- 

Ciao,               Paul D. DeRocco
Paul                mailto:pderocco@ix.netcom.com 

Re: Building one package needs header from another

[Paul Eggleton]

Hi Paul,

Apologies for the delayed reply.

On Friday 23 August 2013 12:52:54 Paul D. DeRocco wrote:
> > > On Tuesday 20 August 2013 16:06:54 Paul D. DeRocco wrote:
> > > 
> > > I've been trying to figure out how the setup.py/setup.cfg
> > > (and distutils)
> > > stuff works. The setup.cfg file lists only one possible
> > > option for adding
> > > directories, which is basedirlist, but setting that to foo adds
> > > foo/include to the include directories and foo/lib to the library
> > > directories, so that's not appropriate.
> > 
> > From: Paul Eggleton
> > 
> > Could you just send it ${STAGING_DIR_HOST}/${prefix} ? We do
> > do that elsewhere with similar-behaving configure scripts.
> 
> I think I've figured out what's going on. The whole distutils.bbclass
> interface to the Python distutils package involves the writing of a
> setup.py script to perform all the packaging and unpackaging processes.
> The setup.py script for matplotlib has to deal with the fact that
> matplotlib depends upon a varying set of libraries depending upon how you
> configure it. In this case, because I want to build the GTK rendering
> backend, it depends upon the pygtk library. It cleverly uses the
> pkg-config command to interrogate each dependent library to find out what
> compiler options are needed to compile modules that refer to the library,
> but unfortunately, pygtk-2.0 is not a "library" that can be searched by
> pkg-config, so the do_compile script coughs up the following nonfatal
> error message:
> 
>     pkg-config: looking for pygtk-2.0 gtk+-2.0
>         * Package pygtk-2.0 was not found in the pkg-config
>         * search path. Perhaps you should add the directory
>         * containing `pygtk-2.0.pc' to the PKG_CONFIG_PATH
>         * environment variable No package 'pygtk-2.0' found
> 
> I think I'm just going to have to patch the setup.py file (actually a
> subsidiary setupext.py file) to hack the additional pygtk-2.0 include
> subdirectory into it.

I'm not very familiar with this particular problem, but if that's the only way 
forward I can't see a huge problem with that. FWIW I have had to do similar 
hacks in setup.py in python recipes I've written in the past.

> However, setupext.py and setup.py both already have
> small patches in the original matplotlib recipe. If I add another patch in
> my bbappend, will it always be applied after the one in the original
> recipe? Or should I just replace the patch file with one containing its
> fixes and my fix?

Patches added in a bbappend will always be applied after the ones in the 
recipe itself, assuming you use SRC_URI += or SRC_URI_append = in your 
bbappend that is (since it's the order in which patches are listed in SRC_URI 
that determines the order in which they will be applied).

Cheers,
Paul

-- 

Paul Eggleton
Intel Open Source Technology Centre