[Buildroot] [RFC] package/python-mako: build host-python-mako as python3 module for mesa3d build.

[Buildroot] [RFC] package/python-mako: build host-python-mako as python3 module for mesa3d build.

[Romain Naour]

Since a long time, it was not possible to build mesa3d from the git
repository due to the missing dependency on host-python-mako package
in mesa3d package.

python-mako module is used by a python script
ir_expression_operation.py to build the header file
ir_expression_operation_constant.h.

Hopefully when building mesa3d from a tarball release using the
autotools build system, this header is alreay generated and the build
continue.

But we are switching mesa3d package to meson build system that check
unconditionnaly if the mako module for python3 interpreter is present.

Even by removing this check, the meson build system call
ir_expression_operation.py script even if
ir_expression_operation_constant.h is already generated.

The problem with host-python-mako and mesa3d is that we need to build
mako module for python3 interpreter. Adding host-python-mako as mesa3d
dependency is not enough since it can be build as host python2 module
when BR2_PACKAGE_PYTHON3 is not selected for the target.

But adding python3 on the target in order to have a python3 module
on the host is not nice.

To build host-python-mako as python3 module we can add this line
in pyhon-mako package:
HOST_PYTHON_MAKO_NEEDS_HOST_PYTHON = python3

But python mako is not available anymore for python2 interpreter.

The problem is more global, how can we build the same python
module for python2 and/or python3 on the host ?

Mako module is also used by gnuradio build system and building
this module for python3 for mesa3d can eventually break the build of
gnuradio when the python interpreter on the host is python2
(to be tested).

Signed-off-by: Romain Naour <romain.naour@smile.fr>
Cc: Asaf Kahlon <asafka7@gmail.com>
Cc: Bernd Kuhls <bernd.kuhls@t-online.de>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Yegor Yefremov <yegorslists@googlemail.com>
---
 package/python-mako/python-mako.mk | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/package/python-mako/python-mako.mk b/package/python-mako/python-mako.mk
index 0d06dc4986..710e4bbe32 100644
--- a/package/python-mako/python-mako.mk
+++ b/package/python-mako/python-mako.mk
@@ -14,5 +14,11 @@ PYTHON_MAKO_LICENSE_FILES = LICENSE
 # In host build, setup.py tries to download markupsafe if it is not installed
 HOST_PYTHON_MAKO_DEPENDENCIES = host-python-markupsafe
 
+# mesa3d python script use python-mako module only for python3.
+# So, build python-mako as python3 module.
+ifeq ($(BR2_PACKAGE_MESA3D),y)
+HOST_PYTHON_MAKO_NEEDS_HOST_PYTHON = python3
+endif
+
 $(eval $(python-package))
 $(eval $(host-python-package))
-- 
2.20.1

[Buildroot] [RFC] package/python-mako: build host-python-mako as python3 module for mesa3d build.

[Thomas Petazzoni]

Hello Romain,

On Sat, 13 Apr 2019 18:46:06 +0200
Romain Naour <romain.naour@smile.fr> wrote:

> But adding python3 on the target in order to have a python3 module
> on the host is not nice.
> 
> To build host-python-mako as python3 module we can add this line
> in pyhon-mako package:
> HOST_PYTHON_MAKO_NEEDS_HOST_PYTHON = python3
> 
> But python mako is not available anymore for python2 interpreter.
> 
> The problem is more global, how can we build the same python
> module for python2 and/or python3 on the host ?

We can't :-)

> Mako module is also used by gnuradio build system and building
> this module for python3 for mesa3d can eventually break the build of
> gnuradio when the python interpreter on the host is python2
> (to be tested).

Indeed, your patch would break situations where mesa3d is enabled, but
another package needs host-python-mako for python2 (and not python3).

We solved this problem for python-setuptools by having
host-python-setuptools and host-python3-setuptools as separate
packages. It's not great, but we don't have any other good solution for
this at the moment I believe.

Alternatively, perhaps for host Python modules we should always install
them for both host-python and host-python3 (if both are enabled, of
course). The gotcha is that since we don't have any Config.in options
for host packages, how can we know if host-python and/or host-python3
are enabled ?

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

[Buildroot] [RFC] package/python-mako: build host-python-mako as python3 module for mesa3d build.

[Romain Naour]

Hi Thomas,

Le 13/04/2019 ? 18:54, Thomas Petazzoni a ?crit?:
> Hello Romain,
> 
> On Sat, 13 Apr 2019 18:46:06 +0200
> Romain Naour <romain.naour@smile.fr> wrote:
> 
>> But adding python3 on the target in order to have a python3 module
>> on the host is not nice.
>>
>> To build host-python-mako as python3 module we can add this line
>> in pyhon-mako package:
>> HOST_PYTHON_MAKO_NEEDS_HOST_PYTHON = python3
>>
>> But python mako is not available anymore for python2 interpreter.
>>
>> The problem is more global, how can we build the same python
>> module for python2 and/or python3 on the host ?
> 
> We can't :-)

My issue would be solved if we can do that :p

> 
>> Mako module is also used by gnuradio build system and building
>> this module for python3 for mesa3d can eventually break the build of
>> gnuradio when the python interpreter on the host is python2
>> (to be tested).
> 
> Indeed, your patch would break situations where mesa3d is enabled, but
> another package needs host-python-mako for python2 (and not python3).

Indeed it break gnuradio:

-- Configuring VOLK support...
-- Build type set to Release.
-- Found PythonInterp: output/host/bin/python2 (found suitable version "2.7.15",
minimum required is "2")
-- 
-- Python checking for python >= 2.7
-- Python checking for python >= 2.7 - found
-- 
-- Python checking for mako >= 0.4.2
-- Python checking for mako >= 0.4.2 - not found
-- 
-- Python checking for six - python 2 and 3 compatibility library
-- Python checking for six - python 2 and 3 compatibility library - found
CMake Error at volk/CMakeLists.txt:88 (message):
  Mako templates required to build VOLK

> 
> We solved this problem for python-setuptools by having
> host-python-setuptools and host-python3-setuptools as separate
> packages. It's not great, but we don't have any other good solution for
> this at the moment I believe.

The problem is that mesa3d is not a python package :)

See the check is mesa build system:
https://cgit.freedesktop.org/mesa/mesa/tree/meson.build?h=mesa-19.0.2#n723

> 
> Alternatively, perhaps for host Python modules we should always install
> them for both host-python and host-python3 (if both are enabled, of
> course). The gotcha is that since we don't have any Config.in options
> for host packages, how can we know if host-python and/or host-python3
> are enabled ?

I don't know... I need to dig deeper into this issue ;)

Best regards,
Romain

> 
> Thomas
> 

[Buildroot] [RFC] package/python-mako: build host-python-mako as python3 module for mesa3d build.

[Arnout Vandecappelle]

On 13/04/2019 18:54, Thomas Petazzoni wrote:
> Alternatively, perhaps for host Python modules we should always install
> them for both host-python and host-python3 (if both are enabled, of
> course). The gotcha is that since we don't have any Config.in options
> for host packages, how can we know if host-python and/or host-python3
> are enabled ?

 We decided in February that having Config.in.host for all packages would add
too much complexity and confusion. We could, however, still do it for python/3
(just like we already have a few blind host package configs). By my count, 52
packages would have to be updated to select BR2_PACKAGE_HOST_PYTHON{,3}. That is
borderline doable, I think.

 The tricky part is detecting when we fail to do the select.

 Regards,
 Arnout

[Buildroot] [RFC] package/python-mako: build host-python-mako as python3 module for mesa3d build.

[Romain Naour]

Hi Thomas, Arnout, All,

Le 13/04/2019 ? 18:46, Romain Naour a ?crit?:
> Since a long time, it was not possible to build mesa3d from the git
> repository due to the missing dependency on host-python-mako package
> in mesa3d package.
> 
> python-mako module is used by a python script
> ir_expression_operation.py to build the header file
> ir_expression_operation_constant.h.
> 
> Hopefully when building mesa3d from a tarball release using the
> autotools build system, this header is alreay generated and the build
> continue.
> 
> But we are switching mesa3d package to meson build system that check
> unconditionnaly if the mako module for python3 interpreter is present.
> 
> Even by removing this check, the meson build system call
> ir_expression_operation.py script even if
> ir_expression_operation_constant.h is already generated.
> 
> The problem with host-python-mako and mesa3d is that we need to build
> mako module for python3 interpreter. Adding host-python-mako as mesa3d
> dependency is not enough since it can be build as host python2 module
> when BR2_PACKAGE_PYTHON3 is not selected for the target.
> 
> But adding python3 on the target in order to have a python3 module
> on the host is not nice.
> 
> To build host-python-mako as python3 module we can add this line
> in pyhon-mako package:
> HOST_PYTHON_MAKO_NEEDS_HOST_PYTHON = python3
> 
> But python mako is not available anymore for python2 interpreter.
> 
> The problem is more global, how can we build the same python
> module for python2 and/or python3 on the host ?
> 
> Mako module is also used by gnuradio build system and building
> this module for python3 for mesa3d can eventually break the build of
> gnuradio when the python interpreter on the host is python2
> (to be tested).
> 
> Signed-off-by: Romain Naour <romain.naour@smile.fr>
> Cc: Asaf Kahlon <asafka7@gmail.com>
> Cc: Bernd Kuhls <bernd.kuhls@t-online.de>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Yegor Yefremov <yegorslists@googlemail.com>
> ---

Actually it's easier to hack the mesa's meson build system than hacking the
python infra...

I have a small patch for mesa that allow to use generated files present in the
release archive (like autotools buils system does).

Best regards,
Romain