* [Buildroot] [PATCH v2 0/3] python-scipy package
@ 2019-09-20 16:36 Esben Haabendal
2019-09-20 16:36 ` [Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type Esben Haabendal
` (2 more replies)
0 siblings, 3 replies; 15+ messages in thread
From: Esben Haabendal @ 2019-09-20 16:36 UTC (permalink / raw)
To: buildroot
Add new python-scipy package and fixup python-numpy so python-scipy can
be cross-compiled.
Changed in v2:
* Added mandatory and lovely comment header in python-scipy.mk file
Esben Haabendal (3):
package/python-numpy: fix setup type
package/python-numpy: fixup numpy distutils for cross compilation
package/python-scipy: new package
DEVELOPERS | 1 +
package/Config.in | 1 +
.../0001-NPY_PKG_CONFIG_PATH.patch | 84 +++++++++++++++++++
package/python-numpy/python-numpy.mk | 25 +++++-
package/python-scipy/Config.in | 21 +++++
package/python-scipy/python-scipy.hash | 9 ++
package/python-scipy/python-scipy.mk | 37 ++++++++
7 files changed, 177 insertions(+), 1 deletion(-)
create mode 100644 package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch
create mode 100644 package/python-scipy/Config.in
create mode 100644 package/python-scipy/python-scipy.hash
create mode 100644 package/python-scipy/python-scipy.mk
--
2.23.0
^ permalink raw reply [flat|nested] 15+ messages in thread* [Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type 2019-09-20 16:36 [Buildroot] [PATCH v2 0/3] python-scipy package Esben Haabendal @ 2019-09-20 16:36 ` Esben Haabendal 2019-09-20 19:11 ` Yegor Yefremov 2019-09-20 16:36 ` [Buildroot] [PATCH v2 2/3] package/python-numpy: fixup numpy distutils for cross compilation Esben Haabendal 2019-09-20 16:36 ` [Buildroot] [PATCH v2 3/3] package/python-scipy: new package Esben Haabendal 2 siblings, 1 reply; 15+ messages in thread From: Esben Haabendal @ 2019-09-20 16:36 UTC (permalink / raw) To: buildroot The numpy package is using distutils, not setuptools. Signed-off-by: Esben Haabendal <esben@geanix.com> --- package/python-numpy/python-numpy.mk | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/package/python-numpy/python-numpy.mk b/package/python-numpy/python-numpy.mk index 3b474efa6e6c..f1f6f8cba407 100644 --- a/package/python-numpy/python-numpy.mk +++ b/package/python-numpy/python-numpy.mk @@ -13,7 +13,7 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \ numpy/linalg/lapack_lite/LICENSE.txt \ tools/npy_tempita/license.txt \ numpy/core/src/multiarray/dragon4.c -PYTHON_NUMPY_SETUP_TYPE = setuptools +PYTHON_NUMPY_SETUP_TYPE = distutils ifeq ($(BR2_PACKAGE_CLAPACK),y) PYTHON_NUMPY_DEPENDENCIES += clapack -- 2.23.0 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type 2019-09-20 16:36 ` [Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type Esben Haabendal @ 2019-09-20 19:11 ` Yegor Yefremov 2019-09-23 6:45 ` Esben Haabendal 0 siblings, 1 reply; 15+ messages in thread From: Yegor Yefremov @ 2019-09-20 19:11 UTC (permalink / raw) To: buildroot Hi Esben, On Fri, Sep 20, 2019 at 6:37 PM Esben Haabendal <esben@geanix.com> wrote: > > The numpy package is using distutils, not setuptools. Are you sure about this? See https://github.com/numpy/numpy/blob/master/numpy/distutils/core.py Yegor > Signed-off-by: Esben Haabendal <esben@geanix.com> > --- > package/python-numpy/python-numpy.mk | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/package/python-numpy/python-numpy.mk b/package/python-numpy/python-numpy.mk > index 3b474efa6e6c..f1f6f8cba407 100644 > --- a/package/python-numpy/python-numpy.mk > +++ b/package/python-numpy/python-numpy.mk > @@ -13,7 +13,7 @@ PYTHON_NUMPY_LICENSE_FILES = LICENSE.txt doc/sphinxext/LICENSE.txt \ > numpy/linalg/lapack_lite/LICENSE.txt \ > tools/npy_tempita/license.txt \ > numpy/core/src/multiarray/dragon4.c > -PYTHON_NUMPY_SETUP_TYPE = setuptools > +PYTHON_NUMPY_SETUP_TYPE = distutils > > ifeq ($(BR2_PACKAGE_CLAPACK),y) > PYTHON_NUMPY_DEPENDENCIES += clapack > -- > 2.23.0 > ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type 2019-09-20 19:11 ` Yegor Yefremov @ 2019-09-23 6:45 ` Esben Haabendal 2019-09-23 13:36 ` Arnout Vandecappelle 0 siblings, 1 reply; 15+ messages in thread From: Esben Haabendal @ 2019-09-23 6:45 UTC (permalink / raw) To: buildroot Hi Yegor Yegor Yefremov <yegorslists@googlemail.com> writes: > On Fri, Sep 20, 2019 at 6:37 PM Esben Haabendal <esben@geanix.com> wrote: >> >> The numpy package is using distutils, not setuptools. > > Are you sure about this? See > https://github.com/numpy/numpy/blob/master/numpy/distutils/core.py https://github.com/numpy/numpy/blob/v1.16.4/setup.py#L392 Looks like a setuptools package to me. /Esben ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type 2019-09-23 6:45 ` Esben Haabendal @ 2019-09-23 13:36 ` Arnout Vandecappelle 2019-09-23 14:09 ` Esben Haabendal 0 siblings, 1 reply; 15+ messages in thread From: Arnout Vandecappelle @ 2019-09-23 13:36 UTC (permalink / raw) To: buildroot On 23/09/2019 08:45, Esben Haabendal wrote: > Hi Yegor > > Yegor Yefremov <yegorslists@googlemail.com> writes: > >> On Fri, Sep 20, 2019 at 6:37 PM Esben Haabendal <esben@geanix.com> wrote: >>> >>> The numpy package is using distutils, not setuptools. >> >> Are you sure about this? See >> https://github.com/numpy/numpy/blob/master/numpy/distutils/core.py > > https://github.com/numpy/numpy/blob/v1.16.4/setup.py#L392 > > Looks like a setuptools package to me. Exactly, and that's how it is defined in Buildroot now. Changing it to distutils would be wrong, no? Regards, Arnout ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type 2019-09-23 13:36 ` Arnout Vandecappelle @ 2019-09-23 14:09 ` Esben Haabendal 0 siblings, 0 replies; 15+ messages in thread From: Esben Haabendal @ 2019-09-23 14:09 UTC (permalink / raw) To: buildroot Arnout Vandecappelle <arnout@mind.be> writes: > On 23/09/2019 08:45, Esben Haabendal wrote: >> Hi Yegor >> >> Yegor Yefremov <yegorslists@googlemail.com> writes: >> >>> On Fri, Sep 20, 2019 at 6:37 PM Esben Haabendal <esben@geanix.com> wrote: >>>> >>>> The numpy package is using distutils, not setuptools. >>> >>> Are you sure about this? See >>> https://github.com/numpy/numpy/blob/master/numpy/distutils/core.py >> >> https://github.com/numpy/numpy/blob/v1.16.4/setup.py#L392 >> >> Looks like a setuptools package to me. > > Exactly, and that's how it is defined in Buildroot now. Changing it to > distutils would be wrong, no? You are absolutely right. EOUTOFCOFFEE. /Esben ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH v2 2/3] package/python-numpy: fixup numpy distutils for cross compilation 2019-09-20 16:36 [Buildroot] [PATCH v2 0/3] python-scipy package Esben Haabendal 2019-09-20 16:36 ` [Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type Esben Haabendal @ 2019-09-20 16:36 ` Esben Haabendal 2019-09-20 16:36 ` [Buildroot] [PATCH v2 3/3] package/python-scipy: new package Esben Haabendal 2 siblings, 0 replies; 15+ messages in thread From: Esben Haabendal @ 2019-09-20 16:36 UTC (permalink / raw) To: buildroot Fix problems using the numpy distutils extension for cross compilation, so that linking with npymath library will use target library when building target packages. Signed-off-by: Esben Haabendal <esben@geanix.com> --- .../0001-NPY_PKG_CONFIG_PATH.patch | 84 +++++++++++++++++++ package/python-numpy/python-numpy.mk | 23 +++++ 2 files changed, 107 insertions(+) create mode 100644 package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch diff --git a/package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch b/package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch new file mode 100644 index 000000000000..cedf5ecc90e4 --- /dev/null +++ b/package/python-numpy/0001-NPY_PKG_CONFIG_PATH.patch @@ -0,0 +1,84 @@ +commit 153fc148eec60e5cbec0e80617f75a3a5dd2a3f8 +Author: Esben Haabendal <esben@geanix.com> +Date: Thu Sep 12 21:59:58 2019 +0200 + + ENH: Allow NPY_PKG_CONFIG_PATH environment variable override + + Allow overriding npy-pkg-config directory using the NPY_PKG_CONFIG_PATH + environment variable, making it easier to use numpy in cross-compilation + setups. + +Upstream-Status: Accepted (scheduled for 1.18.0) + +diff --git a/numpy/distutils/misc_util.py b/numpy/distutils/misc_util.py +index 1e10e92fdadf..0eaaeb7364d4 100644 +--- a/numpy/distutils/misc_util.py ++++ b/numpy/distutils/misc_util.py +@@ -1687,6 +1687,43 @@ class Configuration(object): + + and will be installed as foo.ini in the 'lib' subpath. + ++ Cross-compilation ++ ----------------- ++ When cross-compiling with numpy distutils, it might be necessary to ++ use modified npy-pkg-config files. Using the default/generated files ++ will link with the host libraries (i.e. libnpymath.a). For ++ cross-compilation you of-course need to link with target libraries, ++ while using the host Python installation. ++ ++ You can copy out the numpy/core/lib/npy-pkg-config directory, add a ++ pkgdir value to the .ini files and set NPY_PKG_CONFIG_PATH environment ++ variable to point to the directory with the modified npy-pkg-config ++ files. ++ ++ Example npymath.ini modified for cross-compilation:: ++ ++ [meta] ++ Name=npymath ++ Description=Portable, core math library implementing C99 standard ++ Version=0.1 ++ ++ [variables] ++ pkgname=numpy.core ++ pkgdir=/build/arm-linux-gnueabi/sysroot/usr/lib/python3.7/site-packages/numpy/core ++ prefix=${pkgdir} ++ libdir=${prefix}/lib ++ includedir=${prefix}/include ++ ++ [default] ++ Libs=-L${libdir} -lnpymath ++ Cflags=-I${includedir} ++ Requires=mlib ++ ++ [msvc] ++ Libs=/LIBPATH:${libdir} npymath.lib ++ Cflags=/INCLUDE:${includedir} ++ Requires=mlib ++ + """ + if subst_dict is None: + subst_dict = {} +@@ -2092,9 +2129,22 @@ def get_numpy_include_dirs(): + return include_dirs + + def get_npy_pkg_dir(): +- """Return the path where to find the npy-pkg-config directory.""" ++ """Return the path where to find the npy-pkg-config directory. ++ ++ If the NPY_PKG_CONFIG_PATH environment variable is set, the value of that ++ is returned. Otherwise, a path inside the location of the numpy module is ++ returned. ++ ++ The NPY_PKG_CONFIG_PATH can be useful when cross-compiling, maintaining ++ customized npy-pkg-config .ini files for the cross-compilation ++ environment, and using them when cross-compiling. ++ ++ """ + # XXX: import here for bootstrapping reasons + import numpy ++ d = os.environ.get('NPY_PKG_CONFIG_PATH') ++ if d is not None: ++ return d + d = os.path.join(os.path.dirname(numpy.__file__), + 'core', 'lib', 'npy-pkg-config') + return d diff --git a/package/python-numpy/python-numpy.mk b/package/python-numpy/python-numpy.mk index f1f6f8cba407..3edb34a79ff7 100644 --- a/package/python-numpy/python-numpy.mk +++ b/package/python-numpy/python-numpy.mk @@ -30,6 +30,29 @@ define PYTHON_NUMPY_CONFIGURE_CMDS echo "include_dirs = $(STAGING_DIR)/usr/include" >> $(@D)/site.cfg endef +# The numpy distutils extensions are not very cross friendly. It comes with +# it's own pkg-config look-alike, which we are patching to allow overriding +# where it locates the configuration files. This allows us to use fixed up +# target configuration files, which we make sure includes full path to the +# target staging area, so that when building for target, we actually use the +# target libraries. Without this, target builds using numpy distutils +# extensions (such as fx. python-scipy) will use the host libraries, which +# obviously will not work. +ifeq ($(BR2_PACKAGE_PYTHON3),y) +PYTHON_NUMPY_STAGING_DIR = $(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/numpy +else +PYTHON_NUMPY_STAGING_DIR = $(STAGING_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/site-packages/numpy +endif +PYTHON_NUMPY_NPY_PKG_CONFIG_PATH=$(PYTHON_NUMPY_STAGING_DIR)/core/lib/npy-pkg-config +define PYTHON_NUMPY_FIXUP_NPY_PKG_CONFIG_FILES + sed -i '/pkgdir=/d' \ + $(PYTHON_NUMPY_NPY_PKG_CONFIG_PATH)/npymath.ini + awk -i inplace \ + '/prefix=/ {print "pkgdir='$(PYTHON_NUMPY_STAGING_DIR)/core'"}1' \ + $(PYTHON_NUMPY_NPY_PKG_CONFIG_PATH)/npymath.ini +endef +PYTHON_NUMPY_POST_INSTALL_STAGING_HOOKS += PYTHON_NUMPY_FIXUP_NPY_PKG_CONFIG_FILES + # Some package may include few headers from NumPy, so let's install it # in the staging area. PYTHON_NUMPY_INSTALL_STAGING = YES -- 2.23.0 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH v2 3/3] package/python-scipy: new package 2019-09-20 16:36 [Buildroot] [PATCH v2 0/3] python-scipy package Esben Haabendal 2019-09-20 16:36 ` [Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type Esben Haabendal 2019-09-20 16:36 ` [Buildroot] [PATCH v2 2/3] package/python-numpy: fixup numpy distutils for cross compilation Esben Haabendal @ 2019-09-20 16:36 ` Esben Haabendal 2019-09-20 17:18 ` Romain Naour 2 siblings, 1 reply; 15+ messages in thread From: Esben Haabendal @ 2019-09-20 16:36 UTC (permalink / raw) To: buildroot --- DEVELOPERS | 1 + package/Config.in | 1 + package/python-scipy/Config.in | 21 +++++++++++++++ package/python-scipy/python-scipy.hash | 9 +++++++ package/python-scipy/python-scipy.mk | 37 ++++++++++++++++++++++++++ 5 files changed, 69 insertions(+) create mode 100644 package/python-scipy/Config.in create mode 100644 package/python-scipy/python-scipy.hash create mode 100644 package/python-scipy/python-scipy.mk diff --git a/DEVELOPERS b/DEVELOPERS index 67a0fef0886d..975a197404bb 100644 --- a/DEVELOPERS +++ b/DEVELOPERS @@ -723,6 +723,7 @@ F: package/szip/ N: Esben Haabendal <esben@haabendal.dk> F: boot/gummiboot/ F: package/python-kiwisolver/ +F: package/python-scipy/ N: Etienne Carriere <etienne.carriere@linaro.org> F: boot/optee-os/ diff --git a/package/Config.in b/package/Config.in index dbf297f4df39..818a2abca591 100644 --- a/package/Config.in +++ b/package/Config.in @@ -1056,6 +1056,7 @@ menu "External python modules" source "package/python-scapy/Config.in" source "package/python-scapy3k/Config.in" source "package/python-schedule/Config.in" + source "package/python-scipy/Config.in" source "package/python-sdnotify/Config.in" source "package/python-secretstorage/Config.in" source "package/python-see/Config.in" diff --git a/package/python-scipy/Config.in b/package/python-scipy/Config.in new file mode 100644 index 000000000000..5b8192266843 --- /dev/null +++ b/package/python-scipy/Config.in @@ -0,0 +1,21 @@ +config BR2_PACKAGE_PYTHON_SCIPY + bool "python-scipy" + depends on BR2_PACKAGE_PYTHON3 + depends on BR2_TOOLCHAIN_HAS_FORTRAN + depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS + depends on BR2_PACKAGE_OPENBLAS_ARCH_SUPPORTS + select BR2_PACKAGE_HOST_PYTHON_CYTHON + select BR2_PACKAGE_HOST_PYTHON_NUMPY + select BR2_PACKAGE_PYTHON_NUMPY + select BR2_PACKAGE_OPENBLAS + select BR2_PACKAGE_CLAPACK + help + The SciPy library is one of the core packages that make up the SciPy + stack. It provides many user-friendly and efficient numerical + routines such as routines for numerical integration, interpolation, + optimization, linear algebra and statistics. + + https://www.scipy.org/scipylib/ + +comment "python-scipy needs a toolchain w/ fortran" + depends on !BR2_TOOLCHAIN_HAS_FORTRAN diff --git a/package/python-scipy/python-scipy.hash b/package/python-scipy/python-scipy.hash new file mode 100644 index 000000000000..6b0c3605a3b8 --- /dev/null +++ b/package/python-scipy/python-scipy.hash @@ -0,0 +1,9 @@ +# Locally generated +sha256 4ea68de2840cf7d35c58464412d21f6f154cab7fba610c11002603ee0b9e2372 python-scipy-1.3.1.tar.gz +sha256 904942f7df1dc0f4a48af8d72405854fd96497bd1fe9ddb2c69d1797c22dfcd7 LICENSE.txt +sha256 37e64a498894ac7c3b070023e3689e954a8ecf8a23b90968d09a455f1b4f7b35 scipy/linalg/src/lapack_deprecations/LICENSE +sha256 606209a000716c5f66e33e180ce08434b96ed17db4975ab9723c6b5fbcc89609 scipy/ndimage/LICENSE.txt +sha256 3df9207af2fdb861af0ae3b22026f163b9bcfab4e525dc4943afe2ffa3f77624 scipy/optimize/tnc/LICENSE +sha256 f0cedf52503b2d42b83411a0a16e6fefac346dfad8fddc66f50050150123470c scipy/sparse/linalg/dsolve/SuperLU/License.txt +sha256 0926566f9f406178d1214f8cc796e166b1213dd7c05e0c5b461a8b8ac9e50bbe scipy/sparse/linalg/eigen/arpack/ARPACK/COPYING +sha256 51a5a08d537b34febb851c761496ead461f90b6c3c46a5248780d63870124e9c scipy/spatial/qhull_src/COPYING.txt diff --git a/package/python-scipy/python-scipy.mk b/package/python-scipy/python-scipy.mk new file mode 100644 index 000000000000..74db98ed0ba0 --- /dev/null +++ b/package/python-scipy/python-scipy.mk @@ -0,0 +1,37 @@ +################################################################################ +# +# python-scipy +# +################################################################################ + +PYTHON_SCIPY_VERSION = 1.3.1 +PYTHON_SCIPY_SITE = $(call github,scipy,scipy,v$(PYTHON_SCIPY_VERSION)) +PYTHON_SCIPY_LICENSE = BSD-3-Clause, BSD-2-Clause, BSD, BSD-Style, PSF, \ + Apache-2.0, MIT +PYTHON_SCIPY_LICENSE_FILES = LICENSE.txt \ + scipy/linalg/src/lapack_deprecations/LICENSE \ + scipy/ndimage/LICENSE.txt \ + scipy/optimize/tnc/LICENSE \ + scipy/sparse/linalg/dsolve/SuperLU/License.txt \ + scipy/sparse/linalg/eigen/arpack/ARPACK/COPYING \ + scipy/spatial/qhull_src/COPYING.txt +PYTHON_SCIPY_SETUP_TYPE = setuptools +PYTHON_SCIPY_DEPENDENCIES += host-python-cython host-python-numpy python-numpy \ + openblas clapack + +PYTHON_SCIPY_BUILD_OPTS = --fcompiler=gnu95 +PYTHON_SCIPY_ENV += F90=$(TARGET_CROSS)gfortran F77=$(TARGET_CROSS)gfortran + +define PYTHON_SCIPY_CONFIGURE_CMDS + -rm -f $(@D)/site.cfg + echo "[DEFAULT]" >> $(@D)/site.cfg + echo "library_dirs = $(STAGING_DIR)/usr/lib" >> $(@D)/site.cfg + echo "include_dirs = $(STAGING_DIR)/usr/include" >> $(@D)/site.cfg +endef + +# Use the target numpy pkg-config configuration files modified for target +# cross-compilation. Without this, numpy distutils will cause the linker to +# link with host libnpymath.a. +PYTHON_SCIPY_ENV += NPY_PKG_CONFIG_PATH=$(PYTHON_NUMPY_NPY_PKG_CONFIG_PATH) + +$(eval $(python-package)) -- 2.23.0 ^ permalink raw reply related [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH v2 3/3] package/python-scipy: new package 2019-09-20 16:36 ` [Buildroot] [PATCH v2 3/3] package/python-scipy: new package Esben Haabendal @ 2019-09-20 17:18 ` Romain Naour 2019-09-23 7:05 ` Esben Haabendal 0 siblings, 1 reply; 15+ messages in thread From: Romain Naour @ 2019-09-20 17:18 UTC (permalink / raw) To: buildroot Hi Esben, There is a previous contribution to add python-scipy from Alexandre Payen: http://patchwork.ozlabs.org/patch/1141144/ It would be great if you can work with Alexandre on python-scipy package. Le 20/09/2019 ? 18:36, Esben Haabendal a ?crit?: > --- > DEVELOPERS | 1 + > package/Config.in | 1 + > package/python-scipy/Config.in | 21 +++++++++++++++ > package/python-scipy/python-scipy.hash | 9 +++++++ > package/python-scipy/python-scipy.mk | 37 ++++++++++++++++++++++++++ > 5 files changed, 69 insertions(+) > create mode 100644 package/python-scipy/Config.in > create mode 100644 package/python-scipy/python-scipy.hash > create mode 100644 package/python-scipy/python-scipy.mk > > diff --git a/DEVELOPERS b/DEVELOPERS > index 67a0fef0886d..975a197404bb 100644 > --- a/DEVELOPERS > +++ b/DEVELOPERS > @@ -723,6 +723,7 @@ F: package/szip/ > N: Esben Haabendal <esben@haabendal.dk> > F: boot/gummiboot/ > F: package/python-kiwisolver/ > +F: package/python-scipy/ > > N: Etienne Carriere <etienne.carriere@linaro.org> > F: boot/optee-os/ > diff --git a/package/Config.in b/package/Config.in > index dbf297f4df39..818a2abca591 100644 > --- a/package/Config.in > +++ b/package/Config.in > @@ -1056,6 +1056,7 @@ menu "External python modules" > source "package/python-scapy/Config.in" > source "package/python-scapy3k/Config.in" > source "package/python-schedule/Config.in" > + source "package/python-scipy/Config.in" > source "package/python-sdnotify/Config.in" > source "package/python-secretstorage/Config.in" > source "package/python-see/Config.in" > diff --git a/package/python-scipy/Config.in b/package/python-scipy/Config.in > new file mode 100644 > index 000000000000..5b8192266843 > --- /dev/null > +++ b/package/python-scipy/Config.in > @@ -0,0 +1,21 @@ > +config BR2_PACKAGE_PYTHON_SCIPY > + bool "python-scipy" > + depends on BR2_PACKAGE_PYTHON3 > + depends on BR2_TOOLCHAIN_HAS_FORTRAN > + depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS > + depends on BR2_PACKAGE_OPENBLAS_ARCH_SUPPORTS > + select BR2_PACKAGE_HOST_PYTHON_CYTHON > + select BR2_PACKAGE_HOST_PYTHON_NUMPY There is no such option, the dependency is trigged by adding host-python-numpy in DEPENDENCIES; > + select BR2_PACKAGE_PYTHON_NUMPY > + select BR2_PACKAGE_OPENBLAS > + select BR2_PACKAGE_CLAPACK Python-scipy can be used with lapack instead of Clapack. Clapack is not maintained any more and really out of date. OpenBlas is using a bundled version of Lapack when gFortran compiler is available. Clapack is an old f2c'ed version of lapack, so it can be used without a Fortran Compiler. Since Python-scipy depends on Fortran, Clapack can be removed from python-scipy. Actually there is an existing issue (discussed during the Buildroot hackathon) on *blas providers. We need to fix this before adding new packages using *blas libraries :-/ Best regards, Romain > + help > + The SciPy library is one of the core packages that make up the SciPy > + stack. It provides many user-friendly and efficient numerical > + routines such as routines for numerical integration, interpolation, > + optimization, linear algebra and statistics. > + > + https://www.scipy.org/scipylib/ > + > +comment "python-scipy needs a toolchain w/ fortran" > + depends on !BR2_TOOLCHAIN_HAS_FORTRAN > diff --git a/package/python-scipy/python-scipy.hash b/package/python-scipy/python-scipy.hash > new file mode 100644 > index 000000000000..6b0c3605a3b8 > --- /dev/null > +++ b/package/python-scipy/python-scipy.hash > @@ -0,0 +1,9 @@ > +# Locally generated > +sha256 4ea68de2840cf7d35c58464412d21f6f154cab7fba610c11002603ee0b9e2372 python-scipy-1.3.1.tar.gz > +sha256 904942f7df1dc0f4a48af8d72405854fd96497bd1fe9ddb2c69d1797c22dfcd7 LICENSE.txt > +sha256 37e64a498894ac7c3b070023e3689e954a8ecf8a23b90968d09a455f1b4f7b35 scipy/linalg/src/lapack_deprecations/LICENSE > +sha256 606209a000716c5f66e33e180ce08434b96ed17db4975ab9723c6b5fbcc89609 scipy/ndimage/LICENSE.txt > +sha256 3df9207af2fdb861af0ae3b22026f163b9bcfab4e525dc4943afe2ffa3f77624 scipy/optimize/tnc/LICENSE > +sha256 f0cedf52503b2d42b83411a0a16e6fefac346dfad8fddc66f50050150123470c scipy/sparse/linalg/dsolve/SuperLU/License.txt > +sha256 0926566f9f406178d1214f8cc796e166b1213dd7c05e0c5b461a8b8ac9e50bbe scipy/sparse/linalg/eigen/arpack/ARPACK/COPYING > +sha256 51a5a08d537b34febb851c761496ead461f90b6c3c46a5248780d63870124e9c scipy/spatial/qhull_src/COPYING.txt > diff --git a/package/python-scipy/python-scipy.mk b/package/python-scipy/python-scipy.mk > new file mode 100644 > index 000000000000..74db98ed0ba0 > --- /dev/null > +++ b/package/python-scipy/python-scipy.mk > @@ -0,0 +1,37 @@ > +################################################################################ > +# > +# python-scipy > +# > +################################################################################ > + > +PYTHON_SCIPY_VERSION = 1.3.1 > +PYTHON_SCIPY_SITE = $(call github,scipy,scipy,v$(PYTHON_SCIPY_VERSION)) > +PYTHON_SCIPY_LICENSE = BSD-3-Clause, BSD-2-Clause, BSD, BSD-Style, PSF, \ > + Apache-2.0, MIT > +PYTHON_SCIPY_LICENSE_FILES = LICENSE.txt \ > + scipy/linalg/src/lapack_deprecations/LICENSE \ > + scipy/ndimage/LICENSE.txt \ > + scipy/optimize/tnc/LICENSE \ > + scipy/sparse/linalg/dsolve/SuperLU/License.txt \ > + scipy/sparse/linalg/eigen/arpack/ARPACK/COPYING \ > + scipy/spatial/qhull_src/COPYING.txt > +PYTHON_SCIPY_SETUP_TYPE = setuptools > +PYTHON_SCIPY_DEPENDENCIES += host-python-cython host-python-numpy python-numpy \ > + openblas clapack > + > +PYTHON_SCIPY_BUILD_OPTS = --fcompiler=gnu95 > +PYTHON_SCIPY_ENV += F90=$(TARGET_CROSS)gfortran F77=$(TARGET_CROSS)gfortran > + > +define PYTHON_SCIPY_CONFIGURE_CMDS > + -rm -f $(@D)/site.cfg > + echo "[DEFAULT]" >> $(@D)/site.cfg > + echo "library_dirs = $(STAGING_DIR)/usr/lib" >> $(@D)/site.cfg > + echo "include_dirs = $(STAGING_DIR)/usr/include" >> $(@D)/site.cfg > +endef > + > +# Use the target numpy pkg-config configuration files modified for target > +# cross-compilation. Without this, numpy distutils will cause the linker to > +# link with host libnpymath.a. > +PYTHON_SCIPY_ENV += NPY_PKG_CONFIG_PATH=$(PYTHON_NUMPY_NPY_PKG_CONFIG_PATH) > + > +$(eval $(python-package)) > ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH v2 3/3] package/python-scipy: new package 2019-09-20 17:18 ` Romain Naour @ 2019-09-23 7:05 ` Esben Haabendal 2019-09-23 8:32 ` Alexandre PAYEN 0 siblings, 1 reply; 15+ messages in thread From: Esben Haabendal @ 2019-09-23 7:05 UTC (permalink / raw) To: buildroot Hi Romain Romain Naour <romain.naour@smile.fr> writes: > There is a previous contribution to add python-scipy from Alexandre Payen: > http://patchwork.ozlabs.org/patch/1141144/ > > It would be great if you can work with Alexandre on python-scipy package. Sounds good. Alexandre, do you have time to work on this? > Le 20/09/2019 ? 18:36, Esben Haabendal a ?crit?: >> --- >> DEVELOPERS | 1 + >> package/Config.in | 1 + >> package/python-scipy/Config.in | 21 +++++++++++++++ >> package/python-scipy/python-scipy.hash | 9 +++++++ >> package/python-scipy/python-scipy.mk | 37 ++++++++++++++++++++++++++ >> 5 files changed, 69 insertions(+) >> create mode 100644 package/python-scipy/Config.in >> create mode 100644 package/python-scipy/python-scipy.hash >> create mode 100644 package/python-scipy/python-scipy.mk >> >> diff --git a/DEVELOPERS b/DEVELOPERS >> index 67a0fef0886d..975a197404bb 100644 >> --- a/DEVELOPERS >> +++ b/DEVELOPERS >> @@ -723,6 +723,7 @@ F: package/szip/ >> N: Esben Haabendal <esben@haabendal.dk> >> F: boot/gummiboot/ >> F: package/python-kiwisolver/ >> +F: package/python-scipy/ >> >> N: Etienne Carriere <etienne.carriere@linaro.org> >> F: boot/optee-os/ >> diff --git a/package/Config.in b/package/Config.in >> index dbf297f4df39..818a2abca591 100644 >> --- a/package/Config.in >> +++ b/package/Config.in >> @@ -1056,6 +1056,7 @@ menu "External python modules" >> source "package/python-scapy/Config.in" >> source "package/python-scapy3k/Config.in" >> source "package/python-schedule/Config.in" >> + source "package/python-scipy/Config.in" >> source "package/python-sdnotify/Config.in" >> source "package/python-secretstorage/Config.in" >> source "package/python-see/Config.in" >> diff --git a/package/python-scipy/Config.in b/package/python-scipy/Config.in >> new file mode 100644 >> index 000000000000..5b8192266843 >> --- /dev/null >> +++ b/package/python-scipy/Config.in >> @@ -0,0 +1,21 @@ >> +config BR2_PACKAGE_PYTHON_SCIPY >> + bool "python-scipy" >> + depends on BR2_PACKAGE_PYTHON3 >> + depends on BR2_TOOLCHAIN_HAS_FORTRAN >> + depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS >> + depends on BR2_PACKAGE_OPENBLAS_ARCH_SUPPORTS >> + select BR2_PACKAGE_HOST_PYTHON_CYTHON >> + select BR2_PACKAGE_HOST_PYTHON_NUMPY > > There is no such option, the dependency is trigged by adding host-python-numpy > in DEPENDENCIES; Ah, I wasn't aware of how host-foo packages in DEPENDENCIES works. I will remove the bogus select. >> + select BR2_PACKAGE_PYTHON_NUMPY >> + select BR2_PACKAGE_OPENBLAS >> + select BR2_PACKAGE_CLAPACK > > Python-scipy can be used with lapack instead of Clapack. > Clapack is not maintained any more and really out of date. Yes, but as we currently only supports clapack for python-numpy builds, It seems a bit odd to do it different for python-scipy. What about adding python-scipy first, and then extending both python-numpy and python-scipy to support lapack as (an up-to-date) alternative to clapack afterwards? > OpenBlas is using a bundled version of Lapack when gFortran compiler > is available. Clapack is an old f2c'ed version of lapack, so it can be > used without a Fortran Compiler. Since Python-scipy depends on > Fortran, Clapack can be removed from python-scipy. > > Actually there is an existing issue (discussed during the Buildroot > hackathon) on *blas providers. We need to fix this before adding new > packages using *blas libraries :-/ Ok, so where does this place this work on python-scipy for now? Is it a 100% no-go until *blas providers are reworked? I think it would be a shame to block python-scipy, as it is pretty useful package, and the effort to take it in (looking at this patch series) is pretty minimal. /Esben ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH v2 3/3] package/python-scipy: new package 2019-09-23 7:05 ` Esben Haabendal @ 2019-09-23 8:32 ` Alexandre PAYEN 2019-09-23 8:51 ` Esben Haabendal 0 siblings, 1 reply; 15+ messages in thread From: Alexandre PAYEN @ 2019-09-23 8:32 UTC (permalink / raw) To: buildroot Hi Esben. Le lun. 23 sept. 2019 ? 09:05, Esben Haabendal <esben@geanix.com> a ?crit : > Hi Romain > > Romain Naour <romain.naour@smile.fr> writes: > > > There is a previous contribution to add python-scipy from Alexandre > Payen: > > http://patchwork.ozlabs.org/patch/1141144/ > > > > It would be great if you can work with Alexandre on python-scipy package. > > Sounds good. > > Alexandre, do you have time to work on this? > I have some time if you need yes. > > > Le 20/09/2019 ? 18:36, Esben Haabendal a ?crit : > >> --- > >> DEVELOPERS | 1 + > >> package/Config.in | 1 + > >> package/python-scipy/Config.in | 21 +++++++++++++++ > >> package/python-scipy/python-scipy.hash | 9 +++++++ > >> package/python-scipy/python-scipy.mk | 37 ++++++++++++++++++++++++++ > >> 5 files changed, 69 insertions(+) > >> create mode 100644 package/python-scipy/Config.in > >> create mode 100644 package/python-scipy/python-scipy.hash > >> create mode 100644 package/python-scipy/python-scipy.mk > >> > >> diff --git a/DEVELOPERS b/DEVELOPERS > >> index 67a0fef0886d..975a197404bb 100644 > >> --- a/DEVELOPERS > >> +++ b/DEVELOPERS > >> @@ -723,6 +723,7 @@ F: package/szip/ > >> N: Esben Haabendal <esben@haabendal.dk> > >> F: boot/gummiboot/ > >> F: package/python-kiwisolver/ > >> +F: package/python-scipy/ > >> > >> N: Etienne Carriere <etienne.carriere@linaro.org> > >> F: boot/optee-os/ > >> diff --git a/package/Config.in b/package/Config.in > >> index dbf297f4df39..818a2abca591 100644 > >> --- a/package/Config.in > >> +++ b/package/Config.in > >> @@ -1056,6 +1056,7 @@ menu "External python modules" > >> source "package/python-scapy/Config.in" > >> source "package/python-scapy3k/Config.in" > >> source "package/python-schedule/Config.in" > >> + source "package/python-scipy/Config.in" > >> source "package/python-sdnotify/Config.in" > >> source "package/python-secretstorage/Config.in" > >> source "package/python-see/Config.in" > >> diff --git a/package/python-scipy/Config.in > b/package/python-scipy/Config.in > >> new file mode 100644 > >> index 000000000000..5b8192266843 > >> --- /dev/null > >> +++ b/package/python-scipy/Config.in > >> @@ -0,0 +1,21 @@ > >> +config BR2_PACKAGE_PYTHON_SCIPY > >> + bool "python-scipy" > >> + depends on BR2_PACKAGE_PYTHON3 > >> + depends on BR2_TOOLCHAIN_HAS_FORTRAN > >> + depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS > >> + depends on BR2_PACKAGE_OPENBLAS_ARCH_SUPPORTS > >> + select BR2_PACKAGE_HOST_PYTHON_CYTHON > >> + select BR2_PACKAGE_HOST_PYTHON_NUMPY > > > > There is no such option, the dependency is trigged by adding > host-python-numpy > > in DEPENDENCIES; > > Ah, I wasn't aware of how host-foo packages in DEPENDENCIES works. > I will remove the bogus select. > > >> + select BR2_PACKAGE_PYTHON_NUMPY > >> + select BR2_PACKAGE_OPENBLAS > >> + select BR2_PACKAGE_CLAPACK > > > > Python-scipy can be used with lapack instead of Clapack. > > Clapack is not maintained any more and really out of date. > > Yes, but as we currently only supports clapack for python-numpy builds, > It seems a bit odd to do it different for python-scipy. > > What about adding python-scipy first, and then extending both > python-numpy and python-scipy to support lapack as (an up-to-date) > alternative to clapack afterwards? > This is exatcly what I did. I propose a virtual package for lapack/clapack. But after digging into lapack/clapack, we realise this is much harder than just adding a virtual package. > > > OpenBlas is using a bundled version of Lapack when gFortran compiler > > is available. Clapack is an old f2c'ed version of lapack, so it can be > > used without a Fortran Compiler. Since Python-scipy depends on > > Fortran, Clapack can be removed from python-scipy. > > > > Actually there is an existing issue (discussed during the Buildroot > > hackathon) on *blas providers. We need to fix this before adding new > > packages using *blas libraries :-/ > > Ok, so where does this place this work on python-scipy for now? > Is it a 100% no-go until *blas providers are reworked? > For me, yes, the first setp is to fix those blas providers. There is libflame[1], a lapack implementation and Blis[2], a blas implementation. `libflame includes a compatibility layer, lapack2flame, which includes a complete LAPACK implementation.` `While BLIS exports a new BLAS-like API, it also includes a BLAS compatibility layer which gives application developers access to BLIS implementations via traditional BLAS routine calls.` This have to be test but I guess it could replace lapack/clapack and blas implementation. I suggest you to read the previous work on lapack/clapack and blas[3]. > > I think it would be a shame to block python-scipy, as it is pretty > useful package, and the effort to take it in (looking at this patch > series) is pretty minimal. > > /Esben > Cheers, Alexandre. [1] : https://github.com/flame/libflame [2] : https://github.com/flame/blis [3] : http://buildroot-busybox.2317881.n4.nabble.com/PATCH-1-5-package-python-numpy-fix-occasional-build-and-run-time-failure-with-lapack-td228787.html#a228788 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190923/c7880a74/attachment.html> ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH v2 3/3] package/python-scipy: new package 2019-09-23 8:32 ` Alexandre PAYEN @ 2019-09-23 8:51 ` Esben Haabendal 2019-09-23 12:29 ` Alexandre PAYEN 0 siblings, 1 reply; 15+ messages in thread From: Esben Haabendal @ 2019-09-23 8:51 UTC (permalink / raw) To: buildroot Hi Alexandre Alexandre PAYEN <alexandre.payen@smile.fr> writes: >> Alexandre, do you have time to work on this? >> > I have some time if you need yes. Great :) >> >> + select BR2_PACKAGE_PYTHON_NUMPY >> >> + select BR2_PACKAGE_OPENBLAS >> >> + select BR2_PACKAGE_CLAPACK >> > >> > Python-scipy can be used with lapack instead of Clapack. >> > Clapack is not maintained any more and really out of date. >> >> Yes, but as we currently only supports clapack for python-numpy builds, >> It seems a bit odd to do it different for python-scipy. >> >> What about adding python-scipy first, and then extending both >> python-numpy and python-scipy to support lapack as (an up-to-date) >> alternative to clapack afterwards? >> > This is exatcly what I did. I propose a virtual package for lapack/clapack. > But after digging into lapack/clapack, we realise this is much harder than > just adding a virtual package. What is harder, adding python-scipy first? If so, I think you might been hindered by a cross-compilation unfriendly mechanism in numpy distutils, which I have got a fix into numpy for. Please take a look at https://github.com/numpy/numpy/pull/14495/commits/153fc148eec60e5cbec0e80617f75a3a5dd2a3f8 and how I use that in my python-scipy recipe sent to this list. With that, I think the python-scipy recipe looks pretty simple and innocent, and I don't think that merging it should be a problem for reworking lapack/clpack/*blas later on. >> > OpenBlas is using a bundled version of Lapack when gFortran compiler >> > is available. Clapack is an old f2c'ed version of lapack, so it can be >> > used without a Fortran Compiler. Since Python-scipy depends on >> > Fortran, Clapack can be removed from python-scipy. >> > >> > Actually there is an existing issue (discussed during the Buildroot >> > hackathon) on *blas providers. We need to fix this before adding new >> > packages using *blas libraries :-/ >> >> Ok, so where does this place this work on python-scipy for now? >> Is it a 100% no-go until *blas providers are reworked? >> > For me, yes, the first setp is to fix those blas providers. Ok, but wouldn't it help having the python-scipy in here to have a good use-case for working with all this? > There is libflame[1], a lapack implementation and Blis[2], a blas > implementation. > `libflame includes a compatibility layer, lapack2flame, which includes a > complete LAPACK implementation.` > `While BLIS exports a new BLAS-like API, it also includes a BLAS > compatibility layer which gives application developers access to BLIS > implementations via traditional BLAS routine calls.` > > This have to be test but I guess it could replace lapack/clapack and blas > implementation. > I suggest you to read the previous work on lapack/clapack and blas[3]. I am not at all a lapack user/developer, just trying to build something with it, so please bear with me. You write that the python-scipy requires lapack (and thus implicitl that it does not work with clapack). Why is that? It seems to WORKSFORME, so what am I missing? >> I think it would be a shame to block python-scipy, as it is pretty >> useful package, and the effort to take it in (looking at this patch >> series) is pretty minimal. > [3] : > http://buildroot-busybox.2317881.n4.nabble.com/PATCH-1-5-package-python-numpy-fix-occasional-build-and-run-time-failure-with-lapack-td228787.html#a228788 /Esben ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH v2 3/3] package/python-scipy: new package 2019-09-23 8:51 ` Esben Haabendal @ 2019-09-23 12:29 ` Alexandre PAYEN 2019-09-23 12:46 ` Esben Haabendal 0 siblings, 1 reply; 15+ messages in thread From: Alexandre PAYEN @ 2019-09-23 12:29 UTC (permalink / raw) To: buildroot Le lun. 23 sept. 2019 ? 10:51, Esben Haabendal <esben@geanix.com> a ?crit : > Hi Alexandre > > Alexandre PAYEN <alexandre.payen@smile.fr> writes: > > >> Alexandre, do you have time to work on this? > >> > > I have some time if you need yes. > > Great :) > > >> >> + select BR2_PACKAGE_PYTHON_NUMPY > >> >> + select BR2_PACKAGE_OPENBLAS > >> >> + select BR2_PACKAGE_CLAPACK > >> > > >> > Python-scipy can be used with lapack instead of Clapack. > >> > Clapack is not maintained any more and really out of date. > >> > >> Yes, but as we currently only supports clapack for python-numpy builds, > >> It seems a bit odd to do it different for python-scipy. > >> > >> What about adding python-scipy first, and then extending both > >> python-numpy and python-scipy to support lapack as (an up-to-date) > >> alternative to clapack afterwards? > >> > > This is exatcly what I did. I propose a virtual package for > lapack/clapack. > > But after digging into lapack/clapack, we realise this is much harder > than > > just adding a virtual package. > > What is harder, adding python-scipy first? > No, lapack/clapack and blas. > > If so, I think you might been hindered by a cross-compilation unfriendly > mechanism in numpy distutils, which I have got a fix into numpy for. > Please take a look at > > https://github.com/numpy/numpy/pull/14495/commits/153fc148eec60e5cbec0e80617f75a3a5dd2a3f8 > > and how I use that in my python-scipy recipe sent to this list. > > With that, I think the python-scipy recipe looks pretty simple and > innocent, and I don't think that merging it should be a problem for > reworking lapack/clpack/*blas later on. Yes, there was a cross-compilation problem. But we fix it poorly, I assume your commit do the job. I think adding python-scipy is not relevant because we don't understand lapack/clapack and blas properly. We need to fix blas/lapack before adding packages which uses it. > > >> > OpenBlas is using a bundled version of Lapack when gFortran compiler > >> > is available. Clapack is an old f2c'ed version of lapack, so it can be > >> > used without a Fortran Compiler. Since Python-scipy depends on > >> > Fortran, Clapack can be removed from python-scipy. > >> > > >> > Actually there is an existing issue (discussed during the Buildroot > >> > hackathon) on *blas providers. We need to fix this before adding new > >> > packages using *blas libraries :-/ > >> > >> Ok, so where does this place this work on python-scipy for now? > >> Is it a 100% no-go until *blas providers are reworked? > >> > > For me, yes, the first setp is to fix those blas providers. > > Ok, but wouldn't it help having the python-scipy in here to have a good > use-case for working with all this? > Yes, why not. But python-numpy is already present. It's help testing lapack with or without complex support. > > > There is libflame[1], a lapack implementation and Blis[2], a blas > > implementation. > > `libflame includes a compatibility layer, lapack2flame, which includes a > > complete LAPACK implementation.` > > `While BLIS exports a new BLAS-like API, it also includes a BLAS > > compatibility layer which gives application developers access to BLIS > > implementations via traditional BLAS routine calls.` > > > > This have to be test but I guess it could replace lapack/clapack and blas > > implementation. > > I suggest you to read the previous work on lapack/clapack and blas[3]. > > I am not at all a lapack user/developer, just trying to build something > with it, so please bear with me. > No one really use lapack, python-scipy use it, by building python-scipy, you kind of become a lapack user. > > You write that the python-scipy requires lapack (and thus implicitl that > it does not work with clapack). Why is that? It seems to WORKSFORME, > so what am I missing? > No, clapack is just a C implementation of lapack. It had been generated in C from fortran code with f2c. Clapack is not maintained any more as lapack but there are still old. Also, lapack need a fortran compiler as python-numpy/python-scipy then you should use lapack instead of clapack. > > >> I think it would be a shame to block python-scipy, as it is pretty > >> useful package, and the effort to take it in (looking at this patch > >> series) is pretty minimal. > > > [3] : > > > http://buildroot-busybox.2317881.n4.nabble.com/PATCH-1-5-package-python-numpy-fix-occasional-build-and-run-time-failure-with-lapack-td228787.html#a228788 > > > /Esben > Alexandre. -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190923/58ecd5bd/attachment.html> ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH v2 3/3] package/python-scipy: new package 2019-09-23 12:29 ` Alexandre PAYEN @ 2019-09-23 12:46 ` Esben Haabendal 2019-09-23 12:55 ` Alexandre PAYEN 0 siblings, 1 reply; 15+ messages in thread From: Esben Haabendal @ 2019-09-23 12:46 UTC (permalink / raw) To: buildroot Hi Alexandre Alexandre PAYEN <alexandre.payen@smile.fr> writes: > Le lun. 23 sept. 2019 ? 10:51, Esben Haabendal <esben@geanix.com> a ?crit : > >> Alexandre PAYEN <alexandre.payen@smile.fr> writes: >> >>> This is exatcly what I did. I propose a virtual package for >>> lapack/clapack. But after digging into lapack/clapack, we realise >>> this is much harder than just adding a virtual package. >> >> What is harder, adding python-scipy first? >> > No, lapack/clapack and blas. Ok. So while delaying addition of python-scipy with a relatively simple virtual package change, we now have a much harder task at hand. >> If so, I think you might been hindered by a cross-compilation unfriendly >> mechanism in numpy distutils, which I have got a fix into numpy for. >> Please take a look at >> >> https://github.com/numpy/numpy/pull/14495/commits/153fc148eec60e5cbec0e80617f75a3a5dd2a3f8 >> >> and how I use that in my python-scipy recipe sent to this list. >> >> With that, I think the python-scipy recipe looks pretty simple and >> innocent, and I don't think that merging it should be a problem for >> reworking lapack/clpack/*blas later on. > > Yes, there was a cross-compilation problem. But we fix it poorly, I assume > your commit do the job. I believe it does. > I think adding python-scipy is not relevant because we don't understand > lapack/clapack and blas properly. Well, addition of python-scipy is relevant for me, as I need to have support for it. It is a firm product requirement, so the lack of it requires me to carry it in an overlay or to maintain patches for it. > We need to fix blas/lapack before adding packages which uses it. It might be old, but clapack seems to do the job for python-scipy. So I don't see that we *need* to do something to blas/lapack before adding python-scipy, but it definitely sounds like it would be good to do something about it, with or without python-scipy. >> >> > OpenBlas is using a bundled version of Lapack when gFortran compiler >> >> > is available. Clapack is an old f2c'ed version of lapack, so it can be >> >> > used without a Fortran Compiler. Since Python-scipy depends on >> >> > Fortran, Clapack can be removed from python-scipy. >> >> > >> >> > Actually there is an existing issue (discussed during the Buildroot >> >> > hackathon) on *blas providers. We need to fix this before adding new >> >> > packages using *blas libraries :-/ >> >> >> >> Ok, so where does this place this work on python-scipy for now? >> >> Is it a 100% no-go until *blas providers are reworked? >> >> >> > For me, yes, the first setp is to fix those blas providers. >> >> Ok, but wouldn't it help having the python-scipy in here to have a good >> use-case for working with all this? > > Yes, why not. Ok, let's try to get a simple and clean python-scipy recipe in. >> > There is libflame[1], a lapack implementation and Blis[2], a blas >> > implementation. >> > `libflame includes a compatibility layer, lapack2flame, which includes a >> > complete LAPACK implementation.` >> > `While BLIS exports a new BLAS-like API, it also includes a BLAS >> > compatibility layer which gives application developers access to BLIS >> > implementations via traditional BLAS routine calls.` >> > >> > This have to be test but I guess it could replace lapack/clapack and blas >> > implementation. >> > I suggest you to read the previous work on lapack/clapack and blas[3]. >> >> I am not at all a lapack user/developer, just trying to build something >> with it, so please bear with me. >> > No one really use lapack, python-scipy use it, by building python-scipy, > you kind of become a lapack user. True. But, but it does not make me any more knowledgeable of the API or internals. >> You write that the python-scipy requires lapack (and thus implicitl that >> it does not work with clapack). Why is that? It seems to WORKSFORME, >> so what am I missing? >> > No, clapack is just a C implementation of lapack. It had been generated in > C from fortran code with f2c. > Clapack is not maintained any more as lapack but there are still old. I am with you so far. > Also, lapack need a fortran compiler as python-numpy/python-scipy then > you should use lapack instead of clapack. Well, I don't *have* to use lapack instead of clapack. I could, and you could argue I should. But with this patch series, buildroot can build python-numpy and python-scipy using clapack, for a working numpy/scipy enabled rootfs. That sounds quite useful to me. If I send a slightly modified version of this patch series, will you give it a review and see if we can agree on a form that can get merged now? /Esben ^ permalink raw reply [flat|nested] 15+ messages in thread
* [Buildroot] [PATCH v2 3/3] package/python-scipy: new package 2019-09-23 12:46 ` Esben Haabendal @ 2019-09-23 12:55 ` Alexandre PAYEN 0 siblings, 0 replies; 15+ messages in thread From: Alexandre PAYEN @ 2019-09-23 12:55 UTC (permalink / raw) To: buildroot Hi Esben, Le lun. 23 sept. 2019 ? 14:46, Esben Haabendal <esben@geanix.com> a ?crit : > Hi Alexandre > > Alexandre PAYEN <alexandre.payen@smile.fr> writes: > > > Le lun. 23 sept. 2019 ? 10:51, Esben Haabendal <esben@geanix.com> a > ?crit : > > > >> Alexandre PAYEN <alexandre.payen@smile.fr> writes: > >> > >>> This is exatcly what I did. I propose a virtual package for > >>> lapack/clapack. But after digging into lapack/clapack, we realise > >>> this is much harder than just adding a virtual package. > >> > >> What is harder, adding python-scipy first? > >> > > No, lapack/clapack and blas. > > Ok. So while delaying addition of python-scipy with a relatively simple > virtual package change, we now have a much harder task at hand. > > >> If so, I think you might been hindered by a cross-compilation unfriendly > >> mechanism in numpy distutils, which I have got a fix into numpy for. > >> Please take a look at > >> > >> > https://github.com/numpy/numpy/pull/14495/commits/153fc148eec60e5cbec0e80617f75a3a5dd2a3f8 > >> > >> and how I use that in my python-scipy recipe sent to this list. > >> > >> With that, I think the python-scipy recipe looks pretty simple and > >> innocent, and I don't think that merging it should be a problem for > >> reworking lapack/clpack/*blas later on. > > > > Yes, there was a cross-compilation problem. But we fix it poorly, I > assume > > your commit do the job. > > I believe it does. > > > I think adding python-scipy is not relevant because we don't understand > > lapack/clapack and blas properly. > > Well, addition of python-scipy is relevant for me, as I need to have > support for it. It is a firm product requirement, so the lack of it > requires me to carry it in an overlay or to maintain patches for it. > > > We need to fix blas/lapack before adding packages which uses it. > > It might be old, but clapack seems to do the job for python-scipy. So I > don't see that we *need* to do something to blas/lapack before adding > python-scipy, but it definitely sounds like it would be good to do > something about it, with or without python-scipy. > > >> >> > OpenBlas is using a bundled version of Lapack when gFortran > compiler > >> >> > is available. Clapack is an old f2c'ed version of lapack, so it > can be > >> >> > used without a Fortran Compiler. Since Python-scipy depends on > >> >> > Fortran, Clapack can be removed from python-scipy. > >> >> > > >> >> > Actually there is an existing issue (discussed during the Buildroot > >> >> > hackathon) on *blas providers. We need to fix this before adding > new > >> >> > packages using *blas libraries :-/ > >> >> > >> >> Ok, so where does this place this work on python-scipy for now? > >> >> Is it a 100% no-go until *blas providers are reworked? > >> >> > >> > For me, yes, the first setp is to fix those blas providers. > >> > >> Ok, but wouldn't it help having the python-scipy in here to have a good > >> use-case for working with all this? > > > > Yes, why not. > > Ok, let's try to get a simple and clean python-scipy recipe in. > > >> > There is libflame[1], a lapack implementation and Blis[2], a blas > >> > implementation. > >> > `libflame includes a compatibility layer, lapack2flame, which > includes a > >> > complete LAPACK implementation.` > >> > `While BLIS exports a new BLAS-like API, it also includes a BLAS > >> > compatibility layer which gives application developers access to BLIS > >> > implementations via traditional BLAS routine calls.` > >> > > >> > This have to be test but I guess it could replace lapack/clapack and > blas > >> > implementation. > >> > I suggest you to read the previous work on lapack/clapack and blas[3]. > >> > >> I am not at all a lapack user/developer, just trying to build something > >> with it, so please bear with me. > >> > > No one really use lapack, python-scipy use it, by building python-scipy, > > you kind of become a lapack user. > > True. But, but it does not make me any more knowledgeable of the API or > internals. > > >> You write that the python-scipy requires lapack (and thus implicitl that > >> it does not work with clapack). Why is that? It seems to WORKSFORME, > >> so what am I missing? > >> > > No, clapack is just a C implementation of lapack. It had been generated > in > > C from fortran code with f2c. > > Clapack is not maintained any more as lapack but there are still old. > > I am with you so far. > > > Also, lapack need a fortran compiler as python-numpy/python-scipy then > > you should use lapack instead of clapack. > > Well, I don't *have* to use lapack instead of clapack. I could, and you > could argue I should. > > But with this patch series, buildroot can build python-numpy and > python-scipy using clapack, for a working numpy/scipy enabled rootfs. > That sounds quite useful to me. > > If I send a slightly modified version of this patch series, will you > give it a review and see if we can agree on a form that can get merged > now? > Please, proceed, I will review it. > > /Esben > Alexandre -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190923/0586da5a/attachment.html> ^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2019-09-23 14:09 UTC | newest] Thread overview: 15+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2019-09-20 16:36 [Buildroot] [PATCH v2 0/3] python-scipy package Esben Haabendal 2019-09-20 16:36 ` [Buildroot] [PATCH v2 1/3] package/python-numpy: fix setup type Esben Haabendal 2019-09-20 19:11 ` Yegor Yefremov 2019-09-23 6:45 ` Esben Haabendal 2019-09-23 13:36 ` Arnout Vandecappelle 2019-09-23 14:09 ` Esben Haabendal 2019-09-20 16:36 ` [Buildroot] [PATCH v2 2/3] package/python-numpy: fixup numpy distutils for cross compilation Esben Haabendal 2019-09-20 16:36 ` [Buildroot] [PATCH v2 3/3] package/python-scipy: new package Esben Haabendal 2019-09-20 17:18 ` Romain Naour 2019-09-23 7:05 ` Esben Haabendal 2019-09-23 8:32 ` Alexandre PAYEN 2019-09-23 8:51 ` Esben Haabendal 2019-09-23 12:29 ` Alexandre PAYEN 2019-09-23 12:46 ` Esben Haabendal 2019-09-23 12:55 ` Alexandre PAYEN
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox