[Buildroot] [PATCH 0/8] python: clean up host version logic.

[Buildroot] [PATCH 0/8] python: clean up host version logic.

[Adam Duskett]

Currently, packages that rely on host-python, and are compatible with both
python2 and 3, only build python3 if python3 is selected for the target.

This causes a problem in the following scenario:

- A user does not have a target python selected.
- Package A depends on host-python3
- Package B is compatible with host-python and host-python3

In this scenario, host-python is not needed, as package A and B are
both compatible with python3. However; both host-python and host-python3 will
be built because no target-python has been selected.

To fix this, the following patch series introduces a few minimal changes
that will allow the user to manually select a host version of python.
(Note: I have changed only the packages that I personally use and have tested.)

Adam Duskett (8):
  python: add host-entry
  python3: add host-entry
  python-setuptools: check host-python version
  pkg-waf.mk: check host-python version
  ninja: check host-python version
  libselinux: check host-python version
  setools: check host-python version
  libselinux: check host-python version

 package/Config.in.host                         |  2 ++
 package/libselinux/libselinux.mk               |  2 +-
 package/libsemanage/libsemanage.mk             |  2 +-
 package/ninja/ninja.mk                         |  2 +-
 package/pkg-waf.mk                             | 20 +++++++++++++-------
 package/python-setuptools/python-setuptools.mk | 10 +++++++++-
 package/python/Config.in.host                  |  6 ++++++
 package/python3/Config.in.host                 |  6 ++++++
 package/setools/setools.mk                     |  6 ++++++
 9 files changed, 45 insertions(+), 11 deletions(-)
 create mode 100644 package/python/Config.in.host
 create mode 100644 package/python3/Config.in.host

-- 
2.14.3

[Buildroot] [PATCH 1/8] python: add selectable host entry

[Adam Duskett]

Add a user selectable entry to build python2 as a host package.

Signed-off-by: Adam Duskett <aduskett@gmail.com>
---
 package/Config.in.host        | 1 +
 package/python/Config.in.host | 6 ++++++
 2 files changed, 7 insertions(+)
 create mode 100644 package/python/Config.in.host

diff --git a/package/Config.in.host b/package/Config.in.host
index dd6415bba5..7d908c409c 100644
--- a/package/Config.in.host
+++ b/package/Config.in.host
@@ -41,6 +41,7 @@ menu "Host utilities"
 	source "package/pkgconf/Config.in.host"
 	source "package/pru-software-support/Config.in.host"
 	source "package/pwgen/Config.in.host"
+	source "package/python/Config.in.host"
 	source "package/python-lxml/Config.in.host"
 	source "package/python-six/Config.in.host"
 	source "package/qemu/Config.in.host"
diff --git a/package/python/Config.in.host b/package/python/Config.in.host
new file mode 100644
index 0000000000..0a8d6d179f
--- /dev/null
+++ b/package/python/Config.in.host
@@ -0,0 +1,6 @@
+config BR2_PACKAGE_HOST_PYTHON
+	bool "host python"
+	help
+	  The python language interpreter.
+
+	  http://www.python.org/
-- 
2.14.3

[Buildroot] [PATCH 2/8] python3: add selectable host entry

[Adam Duskett]

Add a user selectable entry to build python3 as a host package.

Signed-off-by: Adam Duskett <aduskett@gmail.com>
---
 package/Config.in.host         | 1 +
 package/python3/Config.in.host | 6 ++++++
 2 files changed, 7 insertions(+)
 create mode 100644 package/python3/Config.in.host

diff --git a/package/Config.in.host b/package/Config.in.host
index 7d908c409c..19d45e9d9e 100644
--- a/package/Config.in.host
+++ b/package/Config.in.host
@@ -42,6 +42,7 @@ menu "Host utilities"
 	source "package/pru-software-support/Config.in.host"
 	source "package/pwgen/Config.in.host"
 	source "package/python/Config.in.host"
+	source "package/python3/Config.in.host"
 	source "package/python-lxml/Config.in.host"
 	source "package/python-six/Config.in.host"
 	source "package/qemu/Config.in.host"
diff --git a/package/python3/Config.in.host b/package/python3/Config.in.host
new file mode 100644
index 0000000000..876661e71b
--- /dev/null
+++ b/package/python3/Config.in.host
@@ -0,0 +1,6 @@
+config BR2_PACKAGE_HOST_PYTHON3
+	bool "host python3"
+	help
+	  The python language interpreter.
+
+	  http://www.python.org/
-- 
2.14.3

[Buildroot] [PATCH 3/8] python-setuptools: check host-python version

[Adam Duskett]

python-setuptools is compatible with both python2 and python3, as such,
there is no need to force python2 as a dependency.

Signed-off-by: Adam Duskett <aduskett@gmail.com>
---
 package/python-setuptools/python-setuptools.mk | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/package/python-setuptools/python-setuptools.mk b/package/python-setuptools/python-setuptools.mk
index 28d4f29b5b..a21494a288 100644
--- a/package/python-setuptools/python-setuptools.mk
+++ b/package/python-setuptools/python-setuptools.mk
@@ -10,10 +10,18 @@ PYTHON_SETUPTOOLS_LICENSE = MIT
 PYTHON_SETUPTOOLS_LICENSE_FILES = LICENSE
 PYTHON_SETUPTOOLS_SETUP_TYPE = setuptools
 
+ifeq ($(BR2_PACKAGE_HOST_PYTHON3),y)
+HOST_PYTHON_SETUPTOOLS_NEEDS_HOST_PYTHON += python3
+HOST_PYTHON_SETUPTOOLS_PYLIBVER = python$(PYTHON3_VERSION_MAJOR)
+else
+HOST_PYTHON_SETUPTOOLS_NEEDS_HOST_PYTHON += python
+HOST_PYTHON_SETUPTOOLS_PYLIBVER = python$(PYTHON_VERSION_MAJOR)
+endif
+
 # recent setuptools versions require bootstrap.py to be invoked
 # before the standard setup process.
 define PYTHON_SETUPTOOLS_RUN_BOOTSTRAP
-	cd  $(@D) && $(HOST_DIR)/bin/python ./bootstrap.py
+	cd  $(@D) && $(HOST_DIR)/bin/$(HOST_PYTHON_SETUPTOOLS_PYLIBVER) ./bootstrap.py
 endef
 
 PYTHON_SETUPTOOLS_PRE_CONFIGURE_HOOKS = PYTHON_SETUPTOOLS_RUN_BOOTSTRAP
-- 
2.14.3

[Buildroot] [PATCH 4/8] pkg-waf.mk: check host-python version

[Adam Duskett]

waf is compatible with both python2 and python3. Now that there are host
entries for python, the make logic can be changed to check if host-python3
has been selected, and if so, depend on it instead of explicitly depending
only on python2.

Signed-off-by: Adam Duskett <aduskett@gmail.com>
---
 package/pkg-waf.mk | 20 +++++++++++++-------
 1 file changed, 13 insertions(+), 7 deletions(-)

diff --git a/package/pkg-waf.mk b/package/pkg-waf.mk
index e5b606f063..8df287074b 100644
--- a/package/pkg-waf.mk
+++ b/package/pkg-waf.mk
@@ -36,9 +36,6 @@
 
 define inner-waf-package
 
-# We need host-python to run waf
-$(2)_DEPENDENCIES += host-python
-
 $(2)_NEEDS_EXTERNAL_WAF ?= NO
 
 # If the package does not have its own waf, use our own.
@@ -49,6 +46,15 @@ else
 $(2)_WAF = ./waf
 endif
 
+# We need a host-python to run waf
+ifeq ($(BR2_PACKAGE_HOST_PYTHON3),y)
+$(2)_DEPENDENCIES += host-python3
+$(2)_WAF_PYTHON_PATH ?= $$(HOST_DIR)/bin/python3
+else
+$(2)_DEPENDENCIES += host-python
+$(2)_WAF_PYTHON_PATH ?= $$(HOST_DIR)/bin/python2
+endif
+
 $(2)_BUILD_OPTS				?=
 $(2)_INSTALL_STAGING_OPTS		?=
 $(2)_INSTALL_TARGET_OPTS		?=
@@ -63,7 +69,7 @@ define $(2)_CONFIGURE_CMDS
 	cd $$(@D) && \
 	$$(TARGET_CONFIGURE_OPTS) \
 	$$($(2)_CONF_ENV) \
-	$$(HOST_DIR)/bin/python2 $$($(2)_WAF) configure \
+	$$($(2)_WAF_PYTHON_PATH) $$($(2)_WAF) configure \
 		--prefix=/usr \
 		--libdir=/usr/lib \
 		$$($(2)_CONF_OPTS) \
@@ -78,7 +84,7 @@ endif
 ifndef $(2)_BUILD_CMDS
 define $(2)_BUILD_CMDS
 	cd $$(@D) && \
-	$$(TARGET_MAKE_ENV) $$(HOST_DIR)/bin/python2 $$($(2)_WAF) \
+	$$(TARGET_MAKE_ENV) $$($(2)_WAF_PYTHON_PATH) $$($(2)_WAF) \
 		build -j $$(PARALLEL_JOBS) $$($(2)_BUILD_OPTS) \
 		$$($(2)_WAF_OPTS)
 endef
@@ -91,7 +97,7 @@ endif
 ifndef $(2)_INSTALL_STAGING_CMDS
 define $(2)_INSTALL_STAGING_CMDS
 	cd $$(@D) && \
-	$$(TARGET_MAKE_ENV) $$(HOST_DIR)/bin/python2 $$($(2)_WAF) \
+	$$(TARGET_MAKE_ENV) $$($(2)_WAF_PYTHON_PATH) $$($(2)_WAF) \
 		install --destdir=$$(STAGING_DIR) \
 		$$($(2)_INSTALL_STAGING_OPTS) \
 		$$($(2)_WAF_OPTS)
@@ -105,7 +111,7 @@ endif
 ifndef $(2)_INSTALL_TARGET_CMDS
 define $(2)_INSTALL_TARGET_CMDS
 	cd $$(@D) && \
-	$$(TARGET_MAKE_ENV) $$(HOST_DIR)/bin/python2 $$($(2)_WAF) \
+	$$(TARGET_MAKE_ENV) $$($(2)_WAF_PYTHON_PATH) $$($(2)_WAF) \
 		install --destdir=$$(TARGET_DIR) \
 		$$($(2)_INSTALL_TARGET_OPTS) \
 		$$($(2)_WAF_OPTS)
-- 
2.14.3

[Buildroot] [PATCH 5/8] ninja: check host-python version

[Adam Duskett]

Check if host-python3 is selected instead of target-python3.

Signed-off-by: Adam Duskett <aduskett@gmail.com>
---
 package/ninja/ninja.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/ninja/ninja.mk b/package/ninja/ninja.mk
index 259a87d216..b58d1a0f43 100644
--- a/package/ninja/ninja.mk
+++ b/package/ninja/ninja.mk
@@ -9,7 +9,7 @@ NINJA_SITE = $(call github,ninja-build,ninja,$(NINJA_VERSION))
 NINJA_LICENSE = Apache-2.0
 NINJA_LICENSE_FILES = COPYING
 
-HOST_NINJA_DEPENDENCIES = $(if $(BR2_PACKAGE_PYTHON3),host-python3,host-python)
+HOST_NINJA_DEPENDENCIES = $(if $(BR2_PACKAGE_HOST_PYTHON3),host-python3,host-python)
 
 define HOST_NINJA_BUILD_CMDS
 	(cd $(@D); ./configure.py --bootstrap)
-- 
2.14.3

[Buildroot] [PATCH 6/8] libselinux: check host-python version

[Adam Duskett]

Check if host-python3 is selected instead of target-python3.

Signed-off-by: Adam Duskett <aduskett@gmail.com>
---
 package/libselinux/libselinux.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/libselinux/libselinux.mk b/package/libselinux/libselinux.mk
index 8ac8000de5..f8e8ea1285 100644
--- a/package/libselinux/libselinux.mk
+++ b/package/libselinux/libselinux.mk
@@ -75,7 +75,7 @@ endef
 HOST_LIBSELINUX_DEPENDENCIES = \
 	host-libsepol host-pcre host-swig
 
-ifeq ($(BR2_PACKAGE_PYTHON3),y)
+ifeq ($(BR2_PACKAGE_HOST_PYTHON3),y)
 HOST_LIBSELINUX_DEPENDENCIES += host-python3
 HOST_LIBSELINUX_PYINC = -I$(HOST_DIR)/include/python$(PYTHON3_VERSION_MAJOR)m/
 HOST_LIBSELINUX_PYLIBVER = python$(PYTHON3_VERSION_MAJOR)
-- 
2.14.3

[Buildroot] [PATCH 7/8] setools: check host-python version

[Adam Duskett]

Check if host-python3 is selected instead of target-python3.

Signed-off-by: Adam Duskett <aduskett@gmail.com>
---
 package/setools/setools.mk | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/package/setools/setools.mk b/package/setools/setools.mk
index 6748c95c23..f4644b091c 100644
--- a/package/setools/setools.mk
+++ b/package/setools/setools.mk
@@ -20,6 +20,12 @@ SETOOLS_PYLIBVER = python$(PYTHON_VERSION_MAJOR)
 SETOOLS_DEPENDENCIES += python-enum34
 endif
 
+ifeq ($(BR2_PACKAGE_HOST_PYTHON3),y)
+HOST_SETOOLS_NEEDS_HOST_PYTHON += python3
+else
+HOST_SETOOLS_NEEDS_HOST_PYTHON += python
+endif
+
 define SETOOLS_FIX_SETUP
 	# By default, setup.py will look for libsepol.a in the host machines
 	# /usr/lib directory. This needs to be changed to the staging directory.
-- 
2.14.3

[Buildroot] [PATCH 8/8] libselinux: check host-python version

[Adam Duskett]

Check if host-python3 is selected instead of target-python3.

Signed-off-by: Adam Duskett <aduskett@gmail.com>
---
 package/libsemanage/libsemanage.mk | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/package/libsemanage/libsemanage.mk b/package/libsemanage/libsemanage.mk
index 829409f527..b9a98c7b6c 100644
--- a/package/libsemanage/libsemanage.mk
+++ b/package/libsemanage/libsemanage.mk
@@ -37,7 +37,7 @@ HOST_LIBSEMANAGE_MAKE_OPTS += \
 	PREFIX=$(HOST_DIR) \
 	SWIG_LIB="$(HOST_DIR)/share/swig/$(SWIG_VERSION)/"
 
-ifeq ($(BR2_PACKAGE_PYTHON3),y)
+ifeq ($(BR2_PACKAGE_HOST_PYTHON3),y)
 HOST_LIBSEMANAGE_DEPENDENCIES += host-python3
 HOST_LIBSEMANAGE_MAKE_OPTS += \
 	PYINC="-I$(HOST_DIR)/include/python$(PYTHON3_VERSION_MAJOR)m/" \
-- 
2.14.3

[Buildroot] [PATCH 0/8] python: clean up host version logic.

[Adam Duskett]

Currently, packages that rely on host-python, and are compatible with both
python2 and 3, only build python3 if python3 is selected for the target.

This causes a problem in the following scenario:

- A user does not have a target python selected.
- Package A depends on host-python3
- Package B is compatible with host-python and host-python3

In this scenario, host-python is not needed, as package A and B are
both compatible with python3. However; both host-python and host-python3 will
be built because no target-python has been selected.

To fix this, the following patch series introduces a few minimal changes
that will allow the user to manually select a host version of python.
(Note: I have changed only the packages that I personally use and have tested.)

Adam Duskett (8):
  python: add host-entry
  python3: add host-entry
  python-setuptools: check host-python version
  pkg-waf.mk: check host-python version
  ninja: check host-python version
  libselinux: check host-python version
  setools: check host-python version
  libselinux: check host-python version

 package/Config.in.host                         |  2 ++
 package/libselinux/libselinux.mk               |  2 +-
 package/libsemanage/libsemanage.mk             |  2 +-
 package/ninja/ninja.mk                         |  2 +-
 package/pkg-waf.mk                             | 20 +++++++++++++-------
 package/python-setuptools/python-setuptools.mk | 10 +++++++++-
 package/python/Config.in.host                  |  6 ++++++
 package/python3/Config.in.host                 |  6 ++++++
 package/setools/setools.mk                     |  6 ++++++
 9 files changed, 45 insertions(+), 11 deletions(-)
 create mode 100644 package/python/Config.in.host
 create mode 100644 package/python3/Config.in.host

-- 
2.14.3

[Buildroot] [PATCH 0/8] python: clean up host version logic.

[Thomas Petazzoni]

Hello,

On Tue,  2 Jan 2018 11:33:29 -0500, Adam Duskett wrote:
> Currently, packages that rely on host-python, and are compatible with both
> python2 and 3, only build python3 if python3 is selected for the target.
> 
> This causes a problem in the following scenario:
> 
> - A user does not have a target python selected.
> - Package A depends on host-python3
> - Package B is compatible with host-python and host-python3
> 
> In this scenario, host-python is not needed, as package A and B are
> both compatible with python3. However; both host-python and host-python3 will
> be built because no target-python has been selected.
> 
> To fix this, the following patch series introduces a few minimal changes
> that will allow the user to manually select a host version of python.
> (Note: I have changed only the packages that I personally use and have tested.)

I will have to think a lot to fully grasp the potential consequences of
your changes.

However, there is one thing that I believe is missing in your patch
series: BR2_PACKAGE_PYTHON3 should select BR2_PACKAGE_HOST_PYTHON3, and
BR2_PACKAGE_PYTHON should select BR2_PACKAGE_HOST_PYTHON.

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 0/8] python: clean up host version logic.

[Yann E. MORIN]

Thomas, Adam, All,

On 2018-01-02 21:40 +0100, Thomas Petazzoni spake thusly:
> On Tue,  2 Jan 2018 11:33:29 -0500, Adam Duskett wrote:
> > Currently, packages that rely on host-python, and are compatible with both
> > python2 and 3, only build python3 if python3 is selected for the target.
> > 
> > This causes a problem in the following scenario:
> > 
> > - A user does not have a target python selected.
> > - Package A depends on host-python3
> > - Package B is compatible with host-python and host-python3
> > 
> > In this scenario, host-python is not needed, as package A and B are
> > both compatible with python3. However; both host-python and host-python3 will
> > be built because no target-python has been selected.
> > 
> > To fix this, the following patch series introduces a few minimal changes
> > that will allow the user to manually select a host version of python.
> > (Note: I have changed only the packages that I personally use and have tested.)
> 
> I will have to think a lot to fully grasp the potential consequences of
> your changes.
> 
> However, there is one thing that I believe is missing in your patch
> series: BR2_PACKAGE_PYTHON3 should select BR2_PACKAGE_HOST_PYTHON3, and
> BR2_PACKAGE_PYTHON should select BR2_PACKAGE_HOST_PYTHON.

Furthermore, I believe the prompt should be hidden, because it does not
make sense for the user to actually select the host version for python.

Besides, the current series allows for building both host-python and
host-python3 in the same configuration, and we do not support this for
now (or we would have made it possible for the target variant too).

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 0/8] python: clean up host version logic.

[Thomas Petazzoni]

Hello,

On Tue, 2 Jan 2018 22:40:32 +0100, Yann E. MORIN wrote:

> > However, there is one thing that I believe is missing in your patch
> > series: BR2_PACKAGE_PYTHON3 should select BR2_PACKAGE_HOST_PYTHON3, and
> > BR2_PACKAGE_PYTHON should select BR2_PACKAGE_HOST_PYTHON.  
> 
> Furthermore, I believe the prompt should be hidden, because it does not
> make sense for the user to actually select the host version for python.

If the prompt is not user-selectable, then I don't see the point of the
patch series. Indeed, you can already do:

<pkg>_DEPENDENCIES = $(if $(BR2_PACKAGE_PYTHON3),host-python3,host-python)

to use host-python3 if already built as part of the python3 build
process, or fallback to host-python otherwise. We already use this
construct in a number of places.

> Besides, the current series allows for building both host-python and
> host-python3 in the same configuration, and we do not support this for
> now (or we would have made it possible for the target variant too).

Of course we support building both host-python and host-python3. What
we don't support is:

 - python and python3 in the same configuration

 - building third-party host python modules for a host python version
   different from the selected target python version

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 4/8] pkg-waf.mk: check host-python version

[Trent Piepho]

On Tue, 2018-01-02 at 10:28 -0500, Adam Duskett wrote:
>  
> +# We need a host-python to run waf
> +ifeq ($(BR2_PACKAGE_HOST_PYTHON3),y)
> +$(2)_DEPENDENCIES += host-python3
> +$(2)_WAF_PYTHON_PATH ?= $$(HOST_DIR)/bin/python3
> +else
> +$(2)_DEPENDENCIES += host-python
> +$(2)_WAF_PYTHON_PATH ?= $$(HOST_DIR)/bin/python2
> +endif

PYTHON_PATH above is called PYLIBVER in the five existing definitions
of this variable.   PYTHON_PATH already exists, and it the absolute
path in the target dir of sysconfigdata.

So it would be a lot more consistent to name this PYLIBVER.

It also seems like this variable will be re-created in a lot of
packages, if every package that needs to correctly call the host python
must define it.  I wonder if there could be an easier way for python
package files to call the host python correctly.  Like:

$(HOST_PYTHON_EXE)  Program to run for host python, 2 or 3 as selected.
$(HOST_PYTHON2_EXE) If you must have python 2
$(HOST_PYTHON3_EXE) If you must have python 3