[Buildroot] [PATCH v1 1/7] package/python-joblib: new package

[Buildroot] [PATCH v1 1/7] package/python-joblib: new package

[Guillaume W. Bres]

package/python-scikitlearn requires python-joblib
as runtime dependency

Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
---
 DEVELOPERS                               |  1 +
 package/Config.in                        |  1 +
 package/python-joblib/Config.in          |  8 ++++++++
 package/python-joblib/python-joblib.hash |  3 +++
 package/python-joblib/python-joblib.mk   | 20 ++++++++++++++++++++
 5 files changed, 33 insertions(+)
 create mode 100644 package/python-joblib/Config.in
 create mode 100644 package/python-joblib/python-joblib.hash
 create mode 100644 package/python-joblib/python-joblib.mk

diff --git a/DEVELOPERS b/DEVELOPERS
index d2bd0d809a..2c4f9ed76e 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -1191,6 +1191,7 @@ F:	package/mbw/
 F:	package/pixiewps/
 F:	package/python-beniget/
 F:	package/python-gast/
+F:	package/python-joblib/
 F:	package/python-pybind/
 F:	package/python-pythran/
 F:	package/python-scipy/
diff --git a/package/Config.in b/package/Config.in
index d1c098c48f..7f017c92e9 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -1086,6 +1086,7 @@ menu "External python modules"
 	source "package/python-jedi/Config.in"
 	source "package/python-jinja2/Config.in"
 	source "package/python-jmespath/Config.in"
+	source "package/python-joblib/Config.in"
 	source "package/python-json-schema-validator/Config.in"
 	source "package/python-jsonmodels/Config.in"
 	source "package/python-jsonschema/Config.in"
diff --git a/package/python-joblib/Config.in b/package/python-joblib/Config.in
new file mode 100644
index 0000000000..c6c5122c49
--- /dev/null
+++ b/package/python-joblib/Config.in
@@ -0,0 +1,8 @@
+config BR2_PACKAGE_PYTHON_JOBLIB
+	bool "python-joblib"
+	depends on BR2_PACKAGE_PYTHON3
+	help
+	  Joblib is a set of tools to provide lightweight
+	  pipelining in Python.
+
+	  https://joblib.readthedocs.io/en/latest/
diff --git a/package/python-joblib/python-joblib.hash b/package/python-joblib/python-joblib.hash
new file mode 100644
index 0000000000..5023ba872c
--- /dev/null
+++ b/package/python-joblib/python-joblib.hash
@@ -0,0 +1,3 @@
+# Locally calculated
+sha256  4cda464ca712a4814849c9477d7551b696ce215dc05d79fc516460814ef48f4f  python-joblib-1.1.0.tar.gz
+sha256  42612911c1872c5e4b43f6ae0e8ee59467cd350332241cf72ce90640264fae6a  LICENSE.txt
diff --git a/package/python-joblib/python-joblib.mk b/package/python-joblib/python-joblib.mk
new file mode 100644
index 0000000000..97bf9172f2
--- /dev/null
+++ b/package/python-joblib/python-joblib.mk
@@ -0,0 +1,20 @@
+################################################################################
+#
+# python-joblib
+#
+################################################################################
+
+PYTHON_JOBLIB_VERSION = 1.1.0
+PYTHON_JOBLIB_SITE = $(call github,joblib,joblib,$(PYTHON_JOBLIB_VERSION))
+PYTHON_JOBLIB_LICENSE = BSD-3-Clause
+PYTHON_JOBLIB_LICENSE_FILES = LICENSE.txt
+PYTHON_JOBLIB_SETUP_TYPE = setuptools
+
+# avoid running this test which requires
+# chinese caracters to be supported
+define PYTHON_JOBLIB_REMOVE_SPECIAL_ENCODING_TEST
+	rm -f $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages/joblib/test/test_func_inspect_special_encoding.py
+endef
+PYTHON_JOBLIB_POST_INSTALL_TARGET_HOOKS += PYTHON_JOBLIB_REMOVE_SPECIAL_ENCODING_TEST
+
+$(eval $(python-package))
-- 
2.20.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v1 2/7] package/python-threadpoolctl: new package

[Guillaume W. Bres]

package/python-scikitlearn needs python-threadpoolctl as a
runtime dependency.

Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
---
python-threadpoolctl originally builds with "flit", whose
build process merely consists in copying threadpoolctl.py to
the /usr/lib python pool. As we have no knowledge of "flit",
I simply used the buildroot infra to recreate this behavior.
Let me know of any suggestion
---
 DEVELOPERS                                    |  1 +
 package/Config.in                             |  1 +
 package/python-threadpoolctl/Config.in        | 10 +++++++++
 .../python-threadpoolctl.hash                 |  3 +++
 .../python-threadpoolctl.mk                   | 22 +++++++++++++++++++
 5 files changed, 37 insertions(+)
 create mode 100644 package/python-threadpoolctl/Config.in
 create mode 100644 package/python-threadpoolctl/python-threadpoolctl.hash
 create mode 100644 package/python-threadpoolctl/python-threadpoolctl.mk

diff --git a/DEVELOPERS b/DEVELOPERS
index 2c4f9ed76e..21a35cf9d1 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -1195,6 +1195,7 @@ F:	package/python-joblib/
 F:	package/python-pybind/
 F:	package/python-pythran/
 F:	package/python-scipy/
+F:	package/python-threadpoolctl/
 F:	package/reaver/
 F:	support/testing/tests/package/br2-external/python-pybind
 F:	support/testing/tests/package/sample_python_pybind.py
diff --git a/package/Config.in b/package/Config.in
index 7f017c92e9..876d7309eb 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -1274,6 +1274,7 @@ menu "External python modules"
 	source "package/python-termcolor/Config.in"
 	source "package/python-terminaltables/Config.in"
 	source "package/python-texttable/Config.in"
+	source "package/python-threadpoolctl/Config.in"
 	source "package/python-thrift/Config.in"
 	source "package/python-tinycss2/Config.in"
 	source "package/python-tinyrpc/Config.in"
diff --git a/package/python-threadpoolctl/Config.in b/package/python-threadpoolctl/Config.in
new file mode 100644
index 0000000000..127de3bc39
--- /dev/null
+++ b/package/python-threadpoolctl/Config.in
@@ -0,0 +1,10 @@
+config BR2_PACKAGE_PYTHON_THREADPOOLCTL
+	bool "python-threadpoolctl"
+	depends on BR2_PACKAGE_PYTHON3
+	help
+	  Python helpers to limit the number of threads
+	  used in the threadpool-backed of common native
+	  libraries used for scientific computing and
+	  data science (e.g. BLAS and OpenMP).
+
+	  https://github.com/joblib/threadpoolctl
diff --git a/package/python-threadpoolctl/python-threadpoolctl.hash b/package/python-threadpoolctl/python-threadpoolctl.hash
new file mode 100644
index 0000000000..6b6fa72443
--- /dev/null
+++ b/package/python-threadpoolctl/python-threadpoolctl.hash
@@ -0,0 +1,3 @@
+# Locally calculated
+sha256  449b806f9147a3cf28ed7ec6389dbff9c450331d695873ab75dea79fe661f1e7  python-threadpoolctl-3.1.0.tar.gz
+sha256  81ac619075248b06e53660b652d10e485f4675f5d0ae0f97ea22370da1f7e23b  LICENSE
diff --git a/package/python-threadpoolctl/python-threadpoolctl.mk b/package/python-threadpoolctl/python-threadpoolctl.mk
new file mode 100644
index 0000000000..50f77b1b1f
--- /dev/null
+++ b/package/python-threadpoolctl/python-threadpoolctl.mk
@@ -0,0 +1,22 @@
+################################################################################
+#
+# python-threadpoolctl
+#
+################################################################################
+
+PYTHON_THREADPOOLCTL_VERSION = 3.1.0
+PYTHON_THREADPOOLCTL_SITE = $(call github,joblib,threadpoolctl,$(PYTHON_THREADPOOLCTL_VERSION),$(PYTHON_THREADPOOLCTL_VERSION).tar.gz)
+PYTHON_THREADPOOLCTL_LICENSE = BSD-3-Clause
+PYTHON_THREADPOOLCTL_LICENSE_FILES = LICENSE
+
+# threadpoolctl is originally built using 'flit' which
+# simply symlinks $(@D)/threadpoolctl.py to site-packages
+define PYTHON_THREADPOOLCTL_BUILD_CMDS
+endef
+
+define PYTHON_THREADPOOLCTL_INSTALL_TARGET_CMDS
+	cp $(@D)/threadpoolctl.py \
+		$(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
+endef
+
+$(eval $(generic-package))
-- 
2.20.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v1 3/7] package/lapack: host-python-scipy needs host-lapack

[Guillaume W. Bres]

host-python-scipy is a build time dependency of python-scikitlearn

Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
---
 package/lapack/lapack.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/lapack/lapack.mk b/package/lapack/lapack.mk
index 3db2797202..dc7ce1cde0 100644
--- a/package/lapack/lapack.mk
+++ b/package/lapack/lapack.mk
@@ -28,3 +28,4 @@ LAPACK_CONF_OPTS += \
 endif
 
 $(eval $(cmake-package))
+$(eval $(host-cmake-package))
-- 
2.20.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v1 4/7] package/python-pybind: host-python-scipy needs host-python-pybind

[Guillaume W. Bres]

host-python-scipy is a build time dependency of python-scikitlearn

Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
---
 package/python-pybind/python-pybind.mk | 1 +
 1 file changed, 1 insertion(+)

diff --git a/package/python-pybind/python-pybind.mk b/package/python-pybind/python-pybind.mk
index c5bee3d0a7..24b3fc3f68 100644
--- a/package/python-pybind/python-pybind.mk
+++ b/package/python-pybind/python-pybind.mk
@@ -26,3 +26,4 @@ endef
 PYTHON_PYBIND_POST_INSTALL_STAGING_HOOKS += PYTHON_PYBIND_INSTALL_MODULE
 
 $(eval $(cmake-package))
+$(eval $(host-cmake-package))
-- 
2.20.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v1 5/7] package/python-scipy: add host-package

[Guillaume W. Bres]

host-python-scipy is a build time dependency of python-scikitlearn

Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
---
 package/python-scipy/python-scipy.mk | 14 ++++++++++++++
 1 file changed, 14 insertions(+)

diff --git a/package/python-scipy/python-scipy.mk b/package/python-scipy/python-scipy.mk
index a8c23c6c88..d41cf2fd70 100644
--- a/package/python-scipy/python-scipy.mk
+++ b/package/python-scipy/python-scipy.mk
@@ -31,6 +31,12 @@ PYTHON_SCIPY_DEPENDENCIES += \
 	lapack \
 	python-numpy \
 	python-pybind
+HOST_PYTHON_SCIPY_DEPENDENCIES += \
+	host-python-numpy \
+	host-python-pybind \
+	host-python-pythran \
+	host-lapack \
+	zlib
 PYTHON_SCIPY_INSTALL_STAGING = YES
 
 PYTHON_SCIPY_SETUP_TYPE = setuptools
@@ -55,4 +61,12 @@ define PYTHON_SCIPY_CONFIGURE_CMDS
 	echo "include_dirs = $(STAGING_DIR)/usr/include" >> $(@D)/site.cfg
 endef
 
+define HOST_PYTHON_SCIPY_CONFIGURE_CMDS
+	-rm -f $(@D)/site.cfg
+	echo "[DEFAULT]" >> $(@D)/site.cfg
+	echo "library_dirs = $(HOST_DIR)/usr/lib" >> $(@D)/site.cfg
+	echo "include_dirs = $(HOST_DIR)/usr/include" >> $(@D)/site.cfg
+endef
+
 $(eval $(python-package))
+$(eval $(host-python-package))
-- 
2.20.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v1 6/7] package/python-scikitlearn: new package

[Guillaume W. Bres]

Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
---
python-scikitlearn needs scipy both as build time and runtime dependency
---
 DEVELOPERS                                    |  1 +
 package/Config.in                             |  1 +
 package/python-scikitlearn/Config.in          | 24 +++++++++++++++++++
 .../python-scikitlearn.hash                   |  4 ++++
 .../python-scikitlearn/python-scikitlearn.mk  | 21 ++++++++++++++++
 5 files changed, 51 insertions(+)
 create mode 100644 package/python-scikitlearn/Config.in
 create mode 100644 package/python-scikitlearn/python-scikitlearn.hash
 create mode 100644 package/python-scikitlearn/python-scikitlearn.mk

diff --git a/DEVELOPERS b/DEVELOPERS
index 21a35cf9d1..049f9fe186 100644
--- a/DEVELOPERS
+++ b/DEVELOPERS
@@ -1194,6 +1194,7 @@ F:	package/python-gast/
 F:	package/python-joblib/
 F:	package/python-pybind/
 F:	package/python-pythran/
+F:	package/python-scikitlearn/
 F:	package/python-scipy/
 F:	package/python-threadpoolctl/
 F:	package/reaver/
diff --git a/package/Config.in b/package/Config.in
index 876d7309eb..a2ee4f73e5 100644
--- a/package/Config.in
+++ b/package/Config.in
@@ -1236,6 +1236,7 @@ menu "External python modules"
 	source "package/python-scandir/Config.in"
 	source "package/python-scapy/Config.in"
 	source "package/python-schedule/Config.in"
+	source "package/python-scikitlearn/Config.in"
 	source "package/python-scipy/Config.in"
 	source "package/python-sdnotify/Config.in"
 	source "package/python-secretstorage/Config.in"
diff --git a/package/python-scikitlearn/Config.in b/package/python-scikitlearn/Config.in
new file mode 100644
index 0000000000..1fabdf92fe
--- /dev/null
+++ b/package/python-scikitlearn/Config.in
@@ -0,0 +1,24 @@
+config BR2_PACKAGE_PYTHON_SCIKITLEARN
+	bool "python-scikitlearn"
+	depends on BR2_TOOLCHAIN_HAS_FORTRAN # python-scipy
+	depends on BR2_PACKAGE_OPENBLAS_ARCH_SUPPORTS # python-scipy
+	depends on BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS # python-numpy
+	depends on BR2_TOOLCHAIN_USES_GLIBC || BR2_TOOLCHAIN_USES_MUSL # python-numpy
+	select BR2_PACKAGE_PYTHON_NUMPY
+	select BR2_PACKAGE_PYTHON_SCIPY # runtime
+	select BR2_PACKAGE_PYTHON_JOBLIB # runtime
+	select BR2_PACKAGE_PYTHON_THREADPOOLCTL # runtime
+	help
+	  Scikit-Learn is a Python module for machine learning
+	  built on top of Python-Scipy.
+
+	  https://scikit-learn.org/
+
+comment "python-scikitlearn needs a toolchain with fortran"
+	  depends on !BR2_TOOLCHAIN_HAS_FORTRAN
+
+comment "python-scikitlearn needs lib openblas"
+	  depends on !BR2_PACKAGE_OPENBLAS_ARCH_SUPPORTS
+
+comment "python-scikitlearn needs python-numpy"
+	depends on !BR2_PACKAGE_PYTHON_NUMPY_ARCH_SUPPORTS
diff --git a/package/python-scikitlearn/python-scikitlearn.hash b/package/python-scikitlearn/python-scikitlearn.hash
new file mode 100644
index 0000000000..e8c712761a
--- /dev/null
+++ b/package/python-scikitlearn/python-scikitlearn.hash
@@ -0,0 +1,4 @@
+# Locally calculated
+sha256  568e621b9e1479b9ab952a9241db5af2ba3ab4f69d44b8aba3dd7648825e8e5a  python-scikitlearn-1.1.1.tar.gz
+# License files, locally calculated
+sha256  bc2e5553b31344779335ddadac8de6f9d12da6f4ce46e22a8e2820951a1d5458  COPYING
diff --git a/package/python-scikitlearn/python-scikitlearn.mk b/package/python-scikitlearn/python-scikitlearn.mk
new file mode 100644
index 0000000000..00cdcd9ac7
--- /dev/null
+++ b/package/python-scikitlearn/python-scikitlearn.mk
@@ -0,0 +1,21 @@
+################################################################################
+#
+# python-scikitlearn
+#
+################################################################################
+
+PYTHON_SCIKITLEARN_VERSION = 1.1.1
+PYTHON_SCIKITLEARN_SITE = $(call github,scikit-learn,scikit-learn,$(PYTHON_SCIKITLEARN_VERSION))
+PYTHON_SCIKITLEARN_LICENSE = BSD-3-Clause
+PYTHON_SCIKITLEARN_LICENSE_FILES = COPYING
+PYTHON_SCIKITLEARN_SETUP_TYPE = setuptools
+PYTHON_SCIKITLEARN_DEPENDENCIES = \
+	host-python-cython \
+	host-python-scipy \
+	python-joblib \
+	python-threadpoolctl
+
+# disable test execution when cross compiling
+PYTHON_SCIKITLEARN_ENV = PYTHON_CROSSENV=1
+
+$(eval $(python-package))
-- 
2.20.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v1 7/7] support/testing: add python-scikitlearn test

[Guillaume W. Bres]

Signed-off-by: Guillaume W. Bres <guillaume.bressaix@gmail.com>
---
Execute this test with
./support/testing/run-tests -d dl -o test-results -k tests.package.test_python_scikitlearn.TestPythonPy3ScikitLearn
---
 .../package/sample_python_scikitlearn.py      |  4 +++
 .../tests/package/test_python_scikitlearn.py  | 31 +++++++++++++++++++
 2 files changed, 35 insertions(+)
 create mode 100644 support/testing/tests/package/sample_python_scikitlearn.py
 create mode 100644 support/testing/tests/package/test_python_scikitlearn.py

diff --git a/support/testing/tests/package/sample_python_scikitlearn.py b/support/testing/tests/package/sample_python_scikitlearn.py
new file mode 100644
index 0000000000..af9eef87ef
--- /dev/null
+++ b/support/testing/tests/package/sample_python_scikitlearn.py
@@ -0,0 +1,4 @@
+from sklearn import datasets
+
+iris = datasets.load_iris()
+digits = datasets.load_digits()
diff --git a/support/testing/tests/package/test_python_scikitlearn.py b/support/testing/tests/package/test_python_scikitlearn.py
new file mode 100644
index 0000000000..d729961306
--- /dev/null
+++ b/support/testing/tests/package/test_python_scikitlearn.py
@@ -0,0 +1,31 @@
+import os
+
+from tests.package.test_python import TestPythonPackageBase
+
+
+class TestPythonPy3ScikitLearn(TestPythonPackageBase):
+    __test__ = True
+    # We can't use the base configuration, as we need a glibc
+    # toolchain for scipy
+    config = \
+        """
+        BR2_arm=y
+        BR2_TOOLCHAIN_EXTERNAL=y
+        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
+        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV5_EABI_GLIBC_STABLE=y
+        BR2_PACKAGE_PYTHON3=y
+        BR2_PACKAGE_PYTHON_SCIKITLEARN=y
+        BR2_TARGET_ROOTFS_EXT2=y
+        BR2_TARGET_ROOTFS_EXT2_SIZE="120M"
+        # BR2_TARGET_ROOTFS_TAR is not set
+        """
+    sample_scripts = ["tests/package/sample_python_scikitlearn.py"]
+    timeout = 30
+
+    def login(self):
+        ext2_file = os.path.join(self.builddir, "images", "rootfs.ext2")
+        self.emulator.boot(arch="armv5",
+                           kernel="builtin",
+                           options=["-drive", "file=%s,if=scsi,format=raw" % ext2_file],
+                           kernel_cmdline=["rootwait", "root=/dev/sda"])
+        self.emulator.login()
-- 
2.20.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v1 6/7] package/python-scikitlearn: new package

[Thomas Petazzoni via buildroot]

Hello Guillaume,

On Tue, 16 Aug 2022 16:39:42 +0200
"Guillaume W. Bres" <guillaume.bressaix@gmail.com> wrote:

> +PYTHON_SCIKITLEARN_VERSION = 1.1.1
> +PYTHON_SCIKITLEARN_SITE = $(call github,scikit-learn,scikit-learn,$(PYTHON_SCIKITLEARN_VERSION))
> +PYTHON_SCIKITLEARN_LICENSE = BSD-3-Clause
> +PYTHON_SCIKITLEARN_LICENSE_FILES = COPYING
> +PYTHON_SCIKITLEARN_SETUP_TYPE = setuptools
> +PYTHON_SCIKITLEARN_DEPENDENCIES = \
> +	host-python-cython \
> +	host-python-scipy \

Are you sure it needs host-python-scipy? What error do you get if you
don't have host-python-scipy?

From a quick glance at scikitlearn's setup.py script (at
https://github.com/scikit-learn/scikit-learn/blob/main/setup.py#L303),
I have the impression it only tries to import numpy and scipy at
build-time to verify that their version is correct.

Though it's true than then it does import numpy/scipy stuff:

        # These commands require the setup from numpy.distutils because they
        # may use numpy.distutils compiler classes.
        from numpy.distutils.core import setup

        # Monkeypatches CCompiler.spawn to prevent random wheel build errors on Windows
        # The build errors on Windows was because msvccompiler spawn was not threadsafe
        # This fixed can be removed when we build with numpy >= 1.22.2 on Windows.
        # https://github.com/pypa/distutils/issues/5
        # https://github.com/scikit-learn/scikit-learn/issues/22310
        # https://github.com/numpy/numpy/pull/20640
        from numpy.distutils.ccompiler import replace_method
        from distutils.ccompiler import CCompiler
        from sklearn.externals._numpy_compiler_patch import CCompiler_spawn

But the below part is to avoid build errors on Windows. Can you try to
patch this out?

> +	python-joblib \
> +	python-threadpoolctl

Are you sure these are needed at build-time? You marked them as
run-time dependencies in your Config.in file.

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot