[Buildroot] [RFC] pkg-python infrastructure

[Buildroot] [RFC] pkg-python infrastructure

[Ryan Barnett]

A while back (about a week ago) - Gustavoz Zacarias, Thomas Petazzoni, and 
myself - were having a conversation on the IRC about creating a new 
package type for handling the builds of python packages. If I recall 
correctly, this is what we discussed. I would like to just have some 
record on how we want to handle building python packages.

Thomas P and Gustavoz - please correct/expand on any points that I may be 
missing.

pkg-python infrastructure
-------------------------

 * Would add a new makefile defining generic build steps for python 
packages under package/pkg-python.mk.

 * Would support building for Python 2.7 and Python 3+.

 * Would ensure that cross-compiling of python packages is handled 
correctly.

 * Would support setup.py and setuptools packages.

I don't know when I will be able to get around to doing this but I would 
like to at least capture the points that were made. Also anyone else is 
welcome to pick up this work too :)

Thanks,
-Ryan

------------------------------------------------------------
Ryan J Barnett / Software Engineer / Platform SW
MS 137-157, 855 35th St NE, Cedar Rapids, IA, 52498-3161, US
rjbarnet at rockwellcollins.com
www.rockwellcollins.com 

[Buildroot] [RFC] pkg-python infrastructure

[Thomas De Schampheleire]

Ryan Barnett <rjbarnet@rockwellcollins.com> wrote:
>A while back (about a week ago) - Gustavoz Zacarias, Thomas Petazzoni, and 
>myself - were having a conversation on the IRC about creating a new 
>package type for handling the builds of python packages. If I recall 
>correctly, this is what we discussed. I would like to just have some 
>record on how we want to handle building python packages.
>
>Thomas P and Gustavoz - please correct/expand on any points that I may be 
>missing.
>
>pkg-python infrastructure
>-------------------------
>
> * Would add a new makefile defining generic build steps for python 
>packages under package/pkg-python.mk.
>
> * Would support building for Python 2.7 and Python 3+.
>
> * Would ensure that cross-compiling of python packages is handled 
>correctly.
>
> * Would support setup.py and setuptools packages.
>
>I don't know when I will be able to get around to doing this but I would 
>like to at least capture the points that were made. Also anyone else is 
>welcome to pick up this work too :)

FWIW, I'm definitely in favor of such an infrastructure...

Best regards,
Thomas

[Buildroot] [RFC] pkg-python infrastructure

[Thomas Petazzoni]

Dear Ryan Barnett,

On Fri, 25 Oct 2013 10:52:54 -0500, Ryan Barnett wrote:

> A while back (about a week ago) - Gustavoz Zacarias, Thomas Petazzoni, and 
> myself - were having a conversation on the IRC about creating a new 
> package type for handling the builds of python packages. If I recall 
> correctly, this is what we discussed. I would like to just have some 
> record on how we want to handle building python packages.
> 
> Thomas P and Gustavoz - please correct/expand on any points that I may be 
> missing.
> 
> pkg-python infrastructure
> -------------------------
> 
>  * Would add a new makefile defining generic build steps for python 
> packages under package/pkg-python.mk.
> 
>  * Would support building for Python 2.7 and Python 3+.
> 
>  * Would ensure that cross-compiling of python packages is handled 
> correctly.
> 
>  * Would support setup.py and setuptools packages.
> 
> I don't know when I will be able to get around to doing this but I would 
> like to at least capture the points that were made. Also anyone else is 
> welcome to pick up this work too :)

I definitely confirm that the number of Python modules packages has
grown quite significantly, and that it is certainly time to look at
creating a common infrastructure for those that are based on
setup.py/setuptools. If anyone is interested in working on this, it'd
be great. I just want to warn that it doesn't look like an easy task,
as cross-compilation with Python can be a bit tricky.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [RFC] pkg-python infrastructure

[Gustavo Zacarias]

On 10/25/2013 12:52 PM, Ryan Barnett wrote:

> pkg-python infrastructure
> -------------------------

* Packages should declare if they're python2 and/or python3 compatible.

Right now packages are only for one or the other but not both, this
should be fixed when packages are happy about it (two build/install
phases if enabled).

Regards.

[Buildroot] [RFC] pkg-python infrastructure

[Thomas Petazzoni]

Hello all,

On Fri, 25 Oct 2013 10:52:54 -0500, Ryan Barnett wrote:

> A while back (about a week ago) - Gustavoz Zacarias, Thomas Petazzoni, and 
> myself - were having a conversation on the IRC about creating a new 
> package type for handling the builds of python packages. If I recall 
> correctly, this is what we discussed. I would like to just have some 
> record on how we want to handle building python packages.
> 
> Thomas P and Gustavoz - please correct/expand on any points that I may be 
> missing.

Following this discussion, I've started working on this. The current
status of the work is available at:

 http://git.free-electrons.com/users/thomas-petazzoni/buildroot/log/?h=python-pkg

The most important commit is:

 http://git.free-electrons.com/users/thomas-petazzoni/buildroot/commit/?h=python-pkg&id=5880956ce258b15887049ca5850007395cfd1857

It works quite well with distutils, there are a some issues with
setuptools packages: I'll probably need to distinguish them. There are
also a few more Python packages that have not been converted.

I've limited myself to support Python 2.x for the moment: the idea is
that once all Python packages have been converted to the
infrastructure, it will be easier to extend it to also cover Python 3.x.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [RFC] pkg-python infrastructure

[Samuel Martin]

Hi Thomas, all,


2013/12/5 Thomas Petazzoni <thomas.petazzoni@free-electrons.com>

> Hello all,
>
> On Fri, 25 Oct 2013 10:52:54 -0500, Ryan Barnett wrote:
>
> > A while back (about a week ago) - Gustavoz Zacarias, Thomas Petazzoni,
> and
> > myself - were having a conversation on the IRC about creating a new
> > package type for handling the builds of python packages. If I recall
> > correctly, this is what we discussed. I would like to just have some
> > record on how we want to handle building python packages.
> >
> > Thomas P and Gustavoz - please correct/expand on any points that I may be
> > missing.
>
> Following this discussion, I've started working on this. The current
> status of the work is available at:
>
>
> http://git.free-electrons.com/users/thomas-petazzoni/buildroot/log/?h=python-pkg
>
> The most important commit is:
>
>
> http://git.free-electrons.com/users/thomas-petazzoni/buildroot/commit/?h=python-pkg&id=5880956ce258b15887049ca5850007395cfd1857
>
> It works quite well with distutils, there are a some issues with
> setuptools packages: I'll probably need to distinguish them. There are
> also a few more Python packages that have not been converted.
>
> I've limited myself to support Python 2.x for the moment: the idea is
> that once all Python packages have been converted to the
> infrastructure, it will be easier to extend it to also cover Python 3.x.
>

Nice to see this topic moving! :)

I've quickly reviewed the branch:
* Commit: 5880956ce258b15887049ca5850007395cfd1857 [1,2]
  docs/manual/adding-packages-python.txt:29-30: s/LIBFOO/PYTHON_FOO/


I've given a run:
* python-bottle: install fails because its setup.py does not support
'--executable' option
* python-crc16: install fails because its setup.py does not support
'--executable' option
* python-dpkt: install fails because its setup.py does not support
'--executable' option

After these, I removed "--executable=/usr/bin/python"
from PKG_PYTHON_INSTALL_OPT
* python-protobuf: install fails because it seems checking if the
destination belongs to PYTHONPATH because PYTHONPATH is not correctly set:
<snip>
running install
Checking .pth file support in
/opt/buildroot/output/target/usr/lib/python2.7/site-packages/
/opt/buildroot/output/host/usr/bin/python -E -c pass
TEST FAILED: /opt/buildroot/output/target/usr/lib/python2.7/site-packages/
does NOT support .pth files
error: bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from.  The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

    /opt/buildroot/output/target/usr/lib/python2.7/site-packages/

and your PYTHONPATH environment variable currently contains:

    '/opt/buildroot/output/target/usr/lib/python/site-packages'

Here are some of your options for correcting the problem:
</snip>
The problem seems to be that PYTHON_VERSION_MAJOR is defined afterward
PKG_PYTHON_BUILD_ENV is set...
I've been able to overcome this issue by including
package/python/python.mkat the beginning of package/
pyg-python.mk,
but I'm not sure how clean this is, and how to properly fix it...


So far, I've just tested package builds (no runtim tests yet) and with this
patch [3], I've successfully built all these Python packages:
output/build/host-python-2.7.3/
output/build/host-python-distutilscross-0.1/
output/build/host-python-setuptools-0.6.36/
output/build/python-2.7.3/
output/build/python-bottle-0.11.6/
output/build/python-crc16-0.1.1/
output/build/python-dpkt-1.7/
output/build/python-id3-1.2/
output/build/python-ipy-IPy-0.75/
output/build/python-mad-0.6/
output/build/python-meld3-0.6.8/
output/build/python-netifaces-0.7/
output/build/python-nfc-142/
output/build/python-protobuf-2.4.1/
output/build/python-pygame-f0bb4a4b365d/
output/build/python-pyparsing-1.5.6/
output/build/python-pyro-3.14/
output/build/python-pyzmq-13.1.0/
output/build/python-serial-2.6/
output/build/python-setuptools-0.6.36/
output/build/python-thrift-0.9.0/

[1]
http://git.free-electrons.com/users/thomas-petazzoni/buildroot/commit/?h=python-pkg&id=5880956ce258b15887049ca5850007395cfd1857
[2]
http://git.free-electrons.com/users/thomas-petazzoni/buildroot/tree/docs/manual/adding-packages-python.txt?h=python-pkg&id=5880956ce258b15887049ca5850007395cfd1857#n29
[3] http://code.bulix.org/19qo6e-85143

Regards,

-- 
Samuel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20131205/22e98833/attachment-0001.html>

[Buildroot] [RFC] pkg-python infrastructure

[Thomas Petazzoni]

Dear Samuel Martin,

On Thu, 5 Dec 2013 11:47:06 +0100, Samuel Martin wrote:

> > I've limited myself to support Python 2.x for the moment: the idea is
> > that once all Python packages have been converted to the
> > infrastructure, it will be easier to extend it to also cover Python 3.x.
> 
> Nice to see this topic moving! :)

Yes! Ryan motivated me yesterday evening to have a look at this topic,
so I got started!

> I've quickly reviewed the branch:
> * Commit: 5880956ce258b15887049ca5850007395cfd1857 [1,2]
>   docs/manual/adding-packages-python.txt:29-30: s/LIBFOO/PYTHON_FOO/

Thanks, will fix.

> I've given a run:
> * python-bottle: install fails because its setup.py does not support
> '--executable' option
> * python-crc16: install fails because its setup.py does not support
> '--executable' option
> * python-dpkt: install fails because its setup.py does not support
> '--executable' option
> 
> After these, I removed "--executable=/usr/bin/python"
> from PKG_PYTHON_INSTALL_OPT

Yes, I know this problem. Originally, I wasn't passing --executable=
at install time, when I tested all the basic distutils packages. Then I
went on to some more complex setuptools packages, and one of them was
passing --executable at install time, so I added it. But I did not
retest the previous packages (hence the highly RFC state of the patch
set).

> * python-protobuf: install fails because it seems checking if the
> destination belongs to PYTHONPATH because PYTHONPATH is not correctly set:

I knew this package wasn't building, that's where I stopped yesterday
evening, and decided to notify the list anyway... which apparently was
a good idea because I'm now getting some really good and useful
feedback from you!

> <snip>
> running install
> Checking .pth file support in
> /opt/buildroot/output/target/usr/lib/python2.7/site-packages/
> /opt/buildroot/output/host/usr/bin/python -E -c pass
> TEST FAILED: /opt/buildroot/output/target/usr/lib/python2.7/site-packages/
> does NOT support .pth files
> error: bad install directory or PYTHONPATH
> 
> You are attempting to install a package to a directory that is not
> on PYTHONPATH and which Python does not read ".pth" files from.  The
> installation directory you specified (via --install-dir, --prefix, or
> the distutils default setting) was:
> 
>     /opt/buildroot/output/target/usr/lib/python2.7/site-packages/
> 
> and your PYTHONPATH environment variable currently contains:
> 
>     '/opt/buildroot/output/target/usr/lib/python/site-packages'
> 
> Here are some of your options for correcting the problem:
> </snip>
> The problem seems to be that PYTHON_VERSION_MAJOR is defined afterward
> PKG_PYTHON_BUILD_ENV is set...
> I've been able to overcome this issue by including
> package/python/python.mkat the beginning of package/
> pyg-python.mk,
> but I'm not sure how clean this is, and how to properly fix it...

Including package/python/python.mk is indeed ugly. The good solution is
to delay the evaluation of the variable by adding some more $ at the
right place :-)

> So far, I've just tested package builds (no runtim tests yet) and with this
> patch [3], I've successfully built all these Python packages:
> output/build/host-python-2.7.3/
> output/build/host-python-distutilscross-0.1/
> output/build/host-python-setuptools-0.6.36/
> output/build/python-2.7.3/
> output/build/python-bottle-0.11.6/
> output/build/python-crc16-0.1.1/
> output/build/python-dpkt-1.7/
> output/build/python-id3-1.2/
> output/build/python-ipy-IPy-0.75/
> output/build/python-mad-0.6/
> output/build/python-meld3-0.6.8/
> output/build/python-netifaces-0.7/
> output/build/python-nfc-142/
> output/build/python-protobuf-2.4.1/
> output/build/python-pygame-f0bb4a4b365d/
> output/build/python-pyparsing-1.5.6/
> output/build/python-pyro-3.14/
> output/build/python-pyzmq-13.1.0/
> output/build/python-serial-2.6/
> output/build/python-setuptools-0.6.36/
> output/build/python-thrift-0.9.0/

Cool, thanks!

I'll work on this a bit more, and post a real series.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com