[Buildroot] [PATCH v3 0/5] Python 3, simple, external package support

[Buildroot] [PATCH v3 0/5] Python 3, simple, external package support

[Patrick Gerber]

This patch series refactor python support in buildroot to enable the 
use of external package with both; python2 and python3. The packages
python-bottle and python-serial have been adapted as examples.

To support others external package more work is needed as they are
based on python setuptools. It's no more supported by python3 and
should be replaced by "Distribute". This should be done in another
patch series.

Changes between v2 and v3:

* Same new menu organisation as v2 but with less modifications in 
  package/Config.in (Requested by Thomas Petazzoni)

* New directory called python-common instead of python generic.  
  (Requested by Thomas Petazzoni)

* No more symlink from python to python3 in staging (Requested by 
  Samuel Martin)


Patrick Gerber (5):
  Python menu reorganization
  Add python3 config directory simlink
  Add common declaration to ease the support of both version
  python-bottle: Convert package to be used with python2 or python3
  python-serial: Convert package to be used with python2 or python3

 package/Config.in                      |    3 ++-
 package/python-bottle/python-bottle.mk |    6 +++---
 package/python-common/Config.in        |   35 ++++++++++++++++++++++++++++++++
 package/python-common/python-common.mk |   16 +++++++++++++++
 package/python-serial/Config.in        |    1 -
 package/python-serial/python-serial.mk |    6 +++---
 package/python/Config.in               |   14 -------------
 package/python3/Config.in              |   14 -------------
 package/python3/python3.mk             |    6 ++++++
 9 files changed, 65 insertions(+), 36 deletions(-)
 create mode 100644 package/python-common/Config.in
 create mode 100644 package/python-common/python-common.mk

-- 
1.7.9.5

[Buildroot] [PATCH v3 1/5] Python menu reorganization

[Patrick Gerber]

In order to have a better support for python external package make the python
menu clearer. Python2 and Python3 are now mutually exclusive

Signed-off-by: Patrick Gerber <kpa_info@yahoo.fr>
---
 package/Config.in               |    3 ++-
 package/python-common/Config.in |   35 +++++++++++++++++++++++++++++++++++
 package/python/Config.in        |   14 --------------
 package/python3/Config.in       |   14 --------------
 4 files changed, 37 insertions(+), 29 deletions(-)
 create mode 100644 package/python-common/Config.in

diff --git a/package/Config.in b/package/Config.in
index 2fad94d..5f2c1bd 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -315,9 +315,10 @@ endmenu
 endif
 source "package/microperl/Config.in"
 source "package/php/Config.in"
+source "package/python-common/Config.in"
 source "package/python/Config.in"
 source "package/python3/Config.in"
-if BR2_PACKAGE_PYTHON
+if (BR2_PACKAGE_PYTHON || BR2_PACKAGE_PYTHON3)
 menu "external python modules"
 source "package/python-bottle/Config.in"
 source "package/python-dpkt/Config.in"
diff --git a/package/python-common/Config.in b/package/python-common/Config.in
new file mode 100644
index 0000000..6523c21
--- /dev/null
+++ b/package/python-common/Config.in
@@ -0,0 +1,35 @@
+config BR2_PACKAGE_PYTHON_COMMON
+	bool "Python"
+	depends on BR2_USE_WCHAR
+	# uses fork()
+	depends on BR2_USE_MMU
+	help
+	  The python language interpreter.
+
+	  http://www.python.org/
+
+comment "python3 requires a toolchain with WCHAR support"
+	depends on !BR2_USE_WCHAR
+
+choice
+	depends on BR2_PACKAGE_PYTHON_COMMON
+	prompt "Python interpreter"
+	help
+	  Select which version of Python you would like to use.
+
+config BR2_PACKAGE_PYTHON
+	bool "python2"
+	select BR2_PACKAGE_LIBFFI
+	help
+	  Use Python 2.x 
+
+
+config BR2_PACKAGE_PYTHON3
+	bool "python3"
+	# uses fork()
+	depends on BR2_USE_MMU
+	select BR2_PACKAGE_LIBFFI
+	help
+	  Use Python 3.x 
+
+endchoice
diff --git a/package/python/Config.in b/package/python/Config.in
index 4089cce..b4fd6c2 100644
--- a/package/python/Config.in
+++ b/package/python/Config.in
@@ -1,17 +1,3 @@
-config BR2_PACKAGE_PYTHON
-	bool "python"
-	depends on BR2_USE_WCHAR
-	# uses fork()
-	depends on BR2_USE_MMU
-	select BR2_PACKAGE_LIBFFI
-	help
-	  The python language interpreter.
-
-	  http://www.python.org/
-
-comment "python requires a toolchain with WCHAR support"
-	depends on !BR2_USE_WCHAR
-
 if BR2_PACKAGE_PYTHON
 
 choice
diff --git a/package/python3/Config.in b/package/python3/Config.in
index 5959d31..279f31b 100644
--- a/package/python3/Config.in
+++ b/package/python3/Config.in
@@ -1,17 +1,3 @@
-config BR2_PACKAGE_PYTHON3
-	bool "python3"
-	depends on BR2_USE_WCHAR
-	# uses fork()
-	depends on BR2_USE_MMU
-	select BR2_PACKAGE_LIBFFI
-	help
-	  The python language interpreter.
-
-	  http://www.python.org/
-
-comment "python3 requires a toolchain with WCHAR support"
-	depends on !BR2_USE_WCHAR
-
 if BR2_PACKAGE_PYTHON3
 
 choice
-- 
1.7.9.5

[Buildroot] [PATCH v3 2/5] Add python3 config directory simlink

[Patrick Gerber]

Add a simlink to alias the python3.3/config-3.3m directory to python3.3/config
in the staging directory

Signed-off-by: Patrick Gerber <kpa_info@yahoo.fr>
---
 package/python3/python3.mk |    6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/package/python3/python3.mk b/package/python3/python3.mk
index 7551a7b..8af4d50 100644
--- a/package/python3/python3.mk
+++ b/package/python3/python3.mk
@@ -160,6 +160,12 @@ ifneq ($(BR2_PACKAGE_PYTHON),y)
 PYTHON3_POST_INSTALL_TARGET_HOOKS += PYTHON3_INSTALL_SYMLINK
 endif
 
+define HOST_PYTHON3_INSTALL_SYMLINK
+	ln -fs config-3.3m $(HOST_DIR)/usr/lib/python3.3/config
+endef
+
+HOST_PYTHON3_POST_INSTALL_HOOKS += HOST_PYTHON3_INSTALL_SYMLINK
+
 ifeq ($(BR2_PACKAGE_PYTHON3_PY_ONLY),y)
 define PYTHON3_REMOVE_MODULES_FILES
 	for i in `find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) \
-- 
1.7.9.5

[Buildroot] [PATCH v3 3/5] Add common declaration to ease the support of both version

[Patrick Gerber]

Define a new variable called PYTHON that contains either "python" or
"python3" depending of the selected Python version. This is needed
to easily set dependencies and to call the right version of python
for packages that support both version of python.

Signed-off-by: Patrick Gerber <kpa_info@yahoo.fr>
---
 package/python-common/python-common.mk |   16 ++++++++++++++++
 1 file changed, 16 insertions(+)
 create mode 100644 package/python-common/python-common.mk

diff --git a/package/python-common/python-common.mk b/package/python-common/python-common.mk
new file mode 100644
index 0000000..712b895
--- /dev/null
+++ b/package/python-common/python-common.mk
@@ -0,0 +1,16 @@
+#############################################################
+#
+# python-common
+#
+#############################################################
+# Define some symbol usefull to make python package that works 
+# with python2 and python3. 
+
+ifeq ($(BR2_PACKAGE_PYTHON),y)
+	PYTHON = python
+endif
+
+ifeq ($(BR2_PACKAGE_PYTHON3),y)
+	PYTHON = python3
+endif
+
-- 
1.7.9.5

[Buildroot] [PATCH v3 4/5] python-bottle: Convert package to be used with python2 or python3

[Patrick Gerber]

Signed-off-by: Patrick Gerber <kpa_info@yahoo.fr>
---
 package/python-bottle/python-bottle.mk |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/package/python-bottle/python-bottle.mk b/package/python-bottle/python-bottle.mk
index 370b31b..b8a1847 100644
--- a/package/python-bottle/python-bottle.mk
+++ b/package/python-bottle/python-bottle.mk
@@ -7,16 +7,16 @@
 PYTHON_BOTTLE_VERSION = 0.11.4
 PYTHON_BOTTLE_SOURCE = bottle-$(PYTHON_BOTTLE_VERSION).tar.gz
 PYTHON_BOTTLE_SITE = http://pypi.python.org/packages/source/b/bottle
-PYTHON_BOTTLE_DEPENDENCIES = python
+PYTHON_BOTTLE_DEPENDENCIES = $(PYTHON)
 PYTHON_BOTTLE_LICENSE = MIT
 # README.rst refers to the file "LICENSE" but it's not included
 
 define PYTHON_BOTTLE_BUILD_CMDS
-	(cd $(@D); $(HOST_DIR)/usr/bin/python setup.py build)
+	(cd $(@D); $(HOST_DIR)/usr/bin/$(PYTHON) setup.py build)
 endef
 
 define PYTHON_BOTTLE_INSTALL_TARGET_CMDS
-	(cd $(@D); $(HOST_DIR)/usr/bin/python setup.py install --prefix=$(TARGET_DIR)/usr)
+	(cd $(@D); $(HOST_DIR)/usr/bin/$(PYTHON) setup.py install --prefix=$(TARGET_DIR)/usr)
 endef
 
 $(eval $(generic-package))
-- 
1.7.9.5

[Buildroot] [PATCH v3 5/5] python-serial: Convert package to be used with python2 or python3

[Patrick Gerber]

Signed-off-by: Patrick Gerber <kpa_info@yahoo.fr>
---
 package/python-serial/Config.in        |    1 -
 package/python-serial/python-serial.mk |    6 +++---
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/package/python-serial/Config.in b/package/python-serial/Config.in
index ea090bd..55a9cc1 100644
--- a/package/python-serial/Config.in
+++ b/package/python-serial/Config.in
@@ -1,6 +1,5 @@
 config BR2_PACKAGE_PYTHON_SERIAL
 	bool "python-serial"
-	depends on BR2_PACKAGE_PYTHON
 	help
 	  python-serial is a Python library to access serial ports.
 
diff --git a/package/python-serial/python-serial.mk b/package/python-serial/python-serial.mk
index fb6b31c..6528a6b 100644
--- a/package/python-serial/python-serial.mk
+++ b/package/python-serial/python-serial.mk
@@ -8,14 +8,14 @@ PYTHON_SERIAL_VERSION = 2.6
 PYTHON_SERIAL_SOURCE  = pyserial-$(PYTHON_SERIAL_VERSION).tar.gz
 PYTHON_SERIAL_SITE    = http://pypi.python.org/packages/source/p/pyserial/
 
-PYTHON_SERIAL_DEPENDENCIES = python
+PYTHON_SERIAL_DEPENDENCIES = $(PYTHON)
 
 define PYTHON_SERIAL_BUILD_CMDS
-	(cd $(@D); $(HOST_DIR)/usr/bin/python setup.py build)
+	(cd $(@D); $(HOST_DIR)/usr/bin/$(PYTHON) setup.py build)
 endef
 
 define PYTHON_SERIAL_INSTALL_TARGET_CMDS
-	(cd $(@D); $(HOST_DIR)/usr/bin/python setup.py install --prefix=$(TARGET_DIR)/usr)
+	(cd $(@D); $(HOST_DIR)/usr/bin/$(PYTHON) setup.py install --prefix=$(TARGET_DIR)/usr)
 endef
 
 $(eval $(generic-package))
-- 
1.7.9.5

[Buildroot] [PATCH v3 2/5] Add python3 config directory simlink

[Thomas Petazzoni]

Dear Patrick Gerber,

On Wed, 13 Feb 2013 10:20:34 +0100, Patrick Gerber wrote:
> Add a simlink to alias the python3.3/config-3.3m directory to python3.3/config
> in the staging directory

This description doesn't match what the code is doing. The code creates
a symbolic link in $(HOST_DIR), not $(STAGING_DIR).

Also, this commit log doesn't explain *why* this is needed in the first
place. The commit log only explains what the patch is doing (which we
can easily find out by reading the code), but lacks an explanation of
the underlying reasons.

Thanks,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH v3 0/5] Python 3, simple, external package support

[Maxime Ripard]

Hi Patrick,

Le 13/02/2013 10:20, Patrick Gerber a ?crit :
> This patch series refactor python support in buildroot to enable the 
> use of external package with both; python2 and python3. The packages
> python-bottle and python-serial have been adapted as examples.
> 
> To support others external package more work is needed as they are
> based on python setuptools. It's no more supported by python3 and
> should be replaced by "Distribute". This should be done in another
> patch series.

I don't get why this python-common directory is needed. If the two
python are mutually exclusive, why can't we just declare the PYTHON
variable in their respective package makefiles? The Config.in file is
useless as well, just make a depends on !PYTHON3 in python, and !PYTHON
in python3, like Thomas suggested.

Moreover, I'm pretty concerned about the breakage of many
configurations. You seem to assume here that all external modules can
build on both python and python3, and I'm really not sure about that.

Maxime

-- 
Maxime Ripard, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH v3 2/5] Add python3 config directory simlink

[Patrick]

On 02/13/2013 10:33 AM, Thomas Petazzoni wrote:
> Dear Patrick Gerber,
>
> On Wed, 13 Feb 2013 10:20:34 +0100, Patrick Gerber wrote:
>> Add a simlink to alias the python3.3/config-3.3m directory to python3.3/config
>> in the staging directory
>
> This description doesn't match what the code is doing. The code creates
> a symbolic link in $(HOST_DIR), not $(STAGING_DIR).
>
> Also, this commit log doesn't explain *why* this is needed in the first
> place. The commit log only explains what the patch is doing (which we
> can easily find out by reading the code), but lacks an explanation of
> the underlying reasons.
>
> Thanks,
>
> Thomas
>

When using host python to install an external module, the setup tools 
look for a python3.3/config/pyconfig.h file. With the python3 package 
from buildroot this directory is called config-3.3m. This is why we need 
to alias config to config-3.3m.

Patrick

[Buildroot] [PATCH v3 2/5] Add python3 config directory simlink

[Thomas Petazzoni]

Dear Patrick,

On Wed, 13 Feb 2013 11:07:18 +0100, Patrick wrote:

> When using host python to install an external module, the setup tools 
> look for a python3.3/config/pyconfig.h file. With the python3 package 
> from buildroot this directory is called config-3.3m. This is why we need 
> to alias config to config-3.3m.

Thanks. This explanation should go in the commit log, and maybe even as
a comment in the code explaining why the symlink is needed.

Thanks,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] [PATCH v3 2/5] Add python3 config directory simlink

[Patrick]

On 02/13/2013 11:13 AM, Thomas Petazzoni wrote:
> Dear Patrick,
>
> On Wed, 13 Feb 2013 11:07:18 +0100, Patrick wrote:
>
>> When using host python to install an external module, the setup tools
>> look for a python3.3/config/pyconfig.h file. With the python3 package
>> from buildroot this directory is called config-3.3m. This is why we need
>> to alias config to config-3.3m.
>
> Thanks. This explanation should go in the commit log, and maybe even as
> a comment in the code explaining why the symlink is needed.>

I'll do it.

> Thanks,
>
> Thomas
>

[Buildroot] [PATCH v3 0/5] Python 3, simple, external package support

[Patrick]

Hi Maxime,

On 02/13/2013 10:40 AM, Maxime Ripard wrote:
> Hi Patrick,
>
> Le 13/02/2013 10:20, Patrick Gerber a ?crit :
>> This patch series refactor python support in buildroot to enable the
>> use of external package with both; python2 and python3. The packages
>> python-bottle and python-serial have been adapted as examples.
>>
>> To support others external package more work is needed as they are
>> based on python setuptools. It's no more supported by python3 and
>> should be replaced by "Distribute". This should be done in another
>> patch series.
>
> I don't get why this python-common directory is needed. If the two
> python are mutually exclusive, why can't we just declare the PYTHON
> variable in their respective package makefiles?

You are right it could be done as suggested.

The Config.in file is
> useless as well, just make a depends on !PYTHON3 in python, and !PYTHON
> in python3, like Thomas suggested.

As  I replied to Thomas this result in strange behaviour of the menu. 
When selecting/deselecting one of the python entry the menu do not 
handle properly the "circular reference" and it's almost unusable.

> Moreover, I'm pretty concerned about the breakage of many
> configurations. You seem to assume here that all external modules can
> build on both python and python3, and I'm really not sure about that.

No. All the package that don't support python3 have in their Config.in a 
line like this: depends on BR2_PACKAGE_PYTHON
The same way potential package that support only python3 could have: 
depends on BR2_PACKAGE_PYTHON3

> Maxime
>

Patrick

[Buildroot] [PATCH v3 0/5] Python 3, simple, external package support

[Patrick]

On 02/13/2013 11:20 AM, Patrick wrote:
> Hi Maxime,
>
> On 02/13/2013 10:40 AM, Maxime Ripard wrote:
>> Hi Patrick,
>>
>> I don't get why this python-common directory is needed. If the two
>> python are mutually exclusive, why can't we just declare the PYTHON
>> variable in their respective package makefiles?
>
> You are right it could be done as suggested.
>
> The Config.in file is
>> useless as well, just make a depends on !PYTHON3 in python, and !PYTHON
>> in python3, like Thomas suggested.
>
> As  I replied to Thomas this result in strange behaviour of the menu.
> When selecting/deselecting one of the python entry the menu do not
> handle properly the "circular reference" and it's almost unusable.
>

Maxim, Thomas,

Does it seems acceptable to you if I keep the menu like proposed in this 
patch series (including the file python-common/Config.in) but move the 
PYTHON variable definition in both python.mk and python3.mk and remove 
the python-common.mk file ?

Comments are welcomes

Patrick