* [Buildroot] host-python-numpy - confused about host openblas dependency
@ 2024-01-29 19:54 Graeme Smecher
2024-01-29 20:22 ` Yann E. MORIN
0 siblings, 1 reply; 6+ messages in thread
From: Graeme Smecher @ 2024-01-29 19:54 UTC (permalink / raw)
To: buildroot
Hi all,
Several packages depend on host-python-numpy:
buildroot$ grep host-python-numpy -rl package/
package/python-pythran/python-pythran.mk
package/python-matplotlib/python-matplotlib.mk
package/python-matplotlib/Config.in
package/piglit/piglit.mk
package/piglit/Config.in
package/gnuradio/gnuradio.mk
package/gnuradio/Config.in
package/python-scipy/python-scipy.mk
package/python-scipy/Config.in
package/uhd/uhd.mk
package/uhd/Config.in
However, this package appears to rely on a host-openblas dependency that's implied but never really spelled out.
My builds fail as follows:
buldroot$ make
>>> host-python-numpy 1.25.0 Configuring
rm -rf /path/to/buildroot/output/build/host-python-numpy-1.25.0//build
mkdir -p /path/to/buildroot/output/build/host-python-numpy-1.25.0//build
GIT_DIR=. PATH="/path/to/buildroot/output/host/bin:/path/to/buildroot/output/host/sbin:/home/gsmecher/.local/bin:/home/gsmecher/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin" PKG_CONFIG="/path/to/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/path/to/buildroot/output/host/lib/pkgconfig:/path/to/buildroot/output/host/share/pkgconfig" AR="/usr/bin/ar" AS="/usr/bin/as" LD="/usr/bin/ld" NM="/usr/bin/nm" CC="/usr/bin/gcc" GCC="/usr/bin/gcc" CXX="/usr/bin/g++" CPP="/usr/bin/cpp" OBJCOPY="/usr/bin/objcopy" RANLIB="/usr/bin/ranlib" CPPFLAGS="-I/path/to/buildroot/output/host/include" CFLAGS="-O2 -I/path/to/buildroot/output/host/include" CXXFLAGS="-O2 -I/path/to/buildroot/output/host/include" LDFLAGS="-L/path/to/buildroot/output/host/lib -Wl,-rpath,/path/to/buildroot/output/host/lib" INTLTOOL_PERL=/usr/bin/perl PYTHONNOUSERSITE=y /path/to/buildroot/output/host/bin/me
son setup --prefix=/path/to/buildroot/output/host --libdir=lib --sysconfdir=/path/to/buildroot/output/host/etc --localstatedir=/path/to/buildroot/output/host/var --default-library=shared --buildtype=release --wrap-mode=nodownload -Dstrip=true /path/to/buildroot/output/build/host-python-numpy-1.25.0/ /path/to/buildroot/output/build/host-python-numpy-1.25.0//build
The Meson build system
Version: 1.1.1
Source dir: /path/to/buildroot/output/build/host-python-numpy-1.25.0
Build dir: /path/to/buildroot/output/build/host-python-numpy-1.25.0/build
Build type: native build
Project name: NumPy
Project version: 1.24.0.dev0
C compiler for the host machine: /usr/bin/gcc (gcc 12.2.0 "gcc (Debian 12.2.0-14) 12.2.0")
C linker for the host machine: /usr/bin/gcc ld.bfd 2.40
C++ compiler for the host machine: /usr/bin/g++ (gcc 12.2.0 "g++ (Debian 12.2.0-14) 12.2.0")
C++ linker for the host machine: /usr/bin/g++ ld.bfd 2.40
Cython compiler for the host machine: cython (cython 0.29.36)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python3 found: YES (/path/to/buildroot/output/host/bin/python3)
Found pkg-config: /path/to/buildroot/output/host/bin/pkg-config (1.6.3)
Has header "Python.h" with dependency python-3.11: YES
Compiler for C supports arguments -fno-strict-aliasing: YES
Library m found: YES
Found CMake: /path/to/buildroot/output/host/bin/cmake (3.27.7)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency openblas found: NO (tried pkgconfig and cmake)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency openblas found: YES 0.3.21
Dependency openblas found: YES 0.3.21 (cached)
Program _build_utils/process_src_template.py found: YES (/path/to/buildroot/output/host/bin/python3 /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py)
Program _build_utils/tempita.py found: YES (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py)
output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44: ERROR: Could not get cmake variable and no default provided for <CMakeDependency OpenBLAS: True None>
A full log can be found at /path/to/buildroot/output/build/host-python-numpy-1.25.0/build/meson-logs/meson-log.txt
make: *** [package/pkg-generic.mk:273: /path/to/buildroot/output/build/host-python-numpy-1.25.0/.stamp_configured] Error 1
Any suggestions? There are enough packages depending on host-numpy that I expect it's something unusual on my end. If I force the build to proceed using system pkg-config I can coax it into proceeding - but this is clearly a hack.
thanks,
Graeme
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Buildroot] host-python-numpy - confused about host openblas dependency
2024-01-29 19:54 [Buildroot] host-python-numpy - confused about host openblas dependency Graeme Smecher
@ 2024-01-29 20:22 ` Yann E. MORIN
2024-01-29 20:40 ` [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency gsmecher
0 siblings, 1 reply; 6+ messages in thread
From: Yann E. MORIN @ 2024-01-29 20:22 UTC (permalink / raw)
To: Graeme Smecher; +Cc: buildroot
Graeme, All,
On 2024-01-29 11:54 -0800, Graeme Smecher spake thusly:
> Several packages depend on host-python-numpy:
[--SNIP--]
> However, this package appears to rely on a host-openblas dependency
> that's implied but never really spelled out.
[--SNIP--]
> Any suggestions?
See package/python-numpy/python-numpy.mk: it has proper lapack and blas
handling for the target variant, but indeed the host variant is left in
the dark.
> There are enough packages depending on host-numpy that I expect it's
> something unusual on my end.
Or rather just happenstance that you found it first...
> If I force the build to proceed using system pkg-config I can coax it
> into proceeding - but this is clearly a hack.
Indeed it is not the proper solutoin. The proper way is to edit
package/python-numpy/python-numpy.mk and add:
HOST_PYTHON_NUMPY_CONF_OPTS = -Dblas="" -Dlapack=""
Hopefully they are not needed for the host variant, because we not have
host-lapack or host-openblas.
Care to send a patch, please?
Regards,
Yann E. MORIN.
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency.
2024-01-29 20:22 ` Yann E. MORIN
@ 2024-01-29 20:40 ` gsmecher
2024-01-29 20:58 ` Yann E. MORIN
2024-02-05 18:04 ` Yann E. MORIN
0 siblings, 2 replies; 6+ messages in thread
From: gsmecher @ 2024-01-29 20:40 UTC (permalink / raw)
To: buildroot; +Cc: Graeme Smecher, yann.morin.1998
From: Graeme Smecher <gsmecher@threespeedlogic.com>
Before this patch, host-python-numpy attempted (and failed) to detect
OpenBLAS as follows:
buildroot$ make
>>> host-python-numpy 1.25.0 Configuring
[...]
Run-time dependency openblas found: NO (tried pkgconfig and cmake)
WARNING: CMake Toolchain: Failed to determine CMake compilers state
Run-time dependency openblas found: YES 0.3.21
Dependency openblas found: YES 0.3.21 (cached)
Program _build_utils/process_src_template.py found: YES (/path/to/buildroot/output/host/bin/python3 /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py)
Program _build_utils/tempita.py found: YES (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py)
output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44: ERROR: Could not get cmake variable and no default provided for <CMakeDependency OpenBLAS: True None>
[...]
Two obvious ways to correct this:
- Add some kind of host-lapack / host-openblas dependency to
host-python-numpy, or
- Tell host-python-numpy not to look for (or rely on) external code.
Because the host numpy environment shouldn't be doing any numerical
heavy lifting, we opt for the latter here.
Signed-off-by: Graeme Smecher <gsmecher@threespeedlogic.com>
---
package/python-numpy/python-numpy.mk | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/package/python-numpy/python-numpy.mk b/package/python-numpy/python-numpy.mk
index b52e0da17d..cc5dfc9224 100644
--- a/package/python-numpy/python-numpy.mk
+++ b/package/python-numpy/python-numpy.mk
@@ -36,6 +36,10 @@ else
PYTHON_NUMPY_CONF_OPTS += -Dblas=""
endif
+# Rather than add a host-blas or host-lapack dependencies, just use unoptimized,
+# in-tree code.
+HOST_PYTHON_NUMPY_CONF_OPTS = -Dblas="" -Dlapack=""
+
# Fixup the npymath.ini prefix path with actual target staging area where
# numpy core was built. Without this, target builds using numpy distutils
# extensions like python-scipy, python-numba cannot find -lnpymath since
--
2.39.2
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency.
2024-01-29 20:40 ` [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency gsmecher
@ 2024-01-29 20:58 ` Yann E. MORIN
2024-02-05 18:04 ` Yann E. MORIN
1 sibling, 0 replies; 6+ messages in thread
From: Yann E. MORIN @ 2024-01-29 20:58 UTC (permalink / raw)
To: gsmecher; +Cc: James Hilliard, buildroot
Graeme, All,
+James
On 2024-01-29 12:40 -0800, gsmecher@threespeedlogic.com spake thusly:
> From: Graeme Smecher <gsmecher@threespeedlogic.com>
>
> Before this patch, host-python-numpy attempted (and failed) to detect
> OpenBLAS as follows:
>
> buildroot$ make
> >>> host-python-numpy 1.25.0 Configuring
> [...]
> Run-time dependency openblas found: NO (tried pkgconfig and cmake)
> WARNING: CMake Toolchain: Failed to determine CMake compilers state
> Run-time dependency openblas found: YES 0.3.21
> Dependency openblas found: YES 0.3.21 (cached)
> Program _build_utils/process_src_template.py found: YES (/path/to/buildroot/output/host/bin/python3 /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py)
> Program _build_utils/tempita.py found: YES (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py)
>
> output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44: ERROR: Could not get cmake variable and no default provided for <CMakeDependency OpenBLAS: True None>
> [...]
>
> Two obvious ways to correct this:
>
> - Add some kind of host-lapack / host-openblas dependency to
> host-python-numpy, or
> - Tell host-python-numpy not to look for (or rely on) external code.
>
> Because the host numpy environment shouldn't be doing any numerical
> heavy lifting, we opt for the latter here.
>
> Signed-off-by: Graeme Smecher <gsmecher@threespeedlogic.com>
Good commit log, but there is a missing piece to it, see below...
James: as far as I understand, this was the situation before the bump to
1.25.0 and the switch to meson, because we did not have the BLAS=None
and LAPACK=None options before that either, but had no such failure
report.
So I guess the defaults changed with the switch to meson, which might
explain why we did not notice before?
If so, we need to add that info to the commit log, to decide whether
this needs backporting to 2023.11, and what to do about 2023.02 LTS that
does not have 1.25.
Regards,
Yann E. MORIN.
> ---
> package/python-numpy/python-numpy.mk | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/package/python-numpy/python-numpy.mk b/package/python-numpy/python-numpy.mk
> index b52e0da17d..cc5dfc9224 100644
> --- a/package/python-numpy/python-numpy.mk
> +++ b/package/python-numpy/python-numpy.mk
> @@ -36,6 +36,10 @@ else
> PYTHON_NUMPY_CONF_OPTS += -Dblas=""
> endif
>
> +# Rather than add a host-blas or host-lapack dependencies, just use unoptimized,
> +# in-tree code.
> +HOST_PYTHON_NUMPY_CONF_OPTS = -Dblas="" -Dlapack=""
> +
> # Fixup the npymath.ini prefix path with actual target staging area where
> # numpy core was built. Without this, target builds using numpy distutils
> # extensions like python-scipy, python-numba cannot find -lnpymath since
> --
> 2.39.2
>
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency.
2024-01-29 20:40 ` [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency gsmecher
2024-01-29 20:58 ` Yann E. MORIN
@ 2024-02-05 18:04 ` Yann E. MORIN
2024-02-29 15:38 ` Peter Korsgaard
1 sibling, 1 reply; 6+ messages in thread
From: Yann E. MORIN @ 2024-02-05 18:04 UTC (permalink / raw)
To: gsmecher; +Cc: buildroot
Graeme, All,
On 2024-01-29 12:40 -0800, gsmecher@threespeedlogic.com spake thusly:
> From: Graeme Smecher <gsmecher@threespeedlogic.com>
>
> Before this patch, host-python-numpy attempted (and failed) to detect
> OpenBLAS as follows:
>
> buildroot$ make
> >>> host-python-numpy 1.25.0 Configuring
> [...]
> Run-time dependency openblas found: NO (tried pkgconfig and cmake)
> WARNING: CMake Toolchain: Failed to determine CMake compilers state
> Run-time dependency openblas found: YES 0.3.21
> Dependency openblas found: YES 0.3.21 (cached)
> Program _build_utils/process_src_template.py found: YES (/path/to/buildroot/output/host/bin/python3 /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py)
> Program _build_utils/tempita.py found: YES (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py)
>
> output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44: ERROR: Could not get cmake variable and no default provided for <CMakeDependency OpenBLAS: True None>
> [...]
So I dug in the Numpy git history, and before we bumped to 1.25, it was
using distutils, which would only warn and disable BLAS support if not
found, but the meson switch made the default to fail.
> Two obvious ways to correct this:
>
> - Add some kind of host-lapack / host-openblas dependency to
> host-python-numpy, or
> - Tell host-python-numpy not to look for (or rely on) external code.
>
> Because the host numpy environment shouldn't be doing any numerical
> heavy lifting, we opt for the latter here.
>
> Signed-off-by: Graeme Smecher <gsmecher@threespeedlogic.com>
Applied to master, with an extended commit log about the history,
thanks.
Regards,
Yann E. MORIN.
> ---
> package/python-numpy/python-numpy.mk | 4 ++++
> 1 file changed, 4 insertions(+)
>
> diff --git a/package/python-numpy/python-numpy.mk b/package/python-numpy/python-numpy.mk
> index b52e0da17d..cc5dfc9224 100644
> --- a/package/python-numpy/python-numpy.mk
> +++ b/package/python-numpy/python-numpy.mk
> @@ -36,6 +36,10 @@ else
> PYTHON_NUMPY_CONF_OPTS += -Dblas=""
> endif
>
> +# Rather than add a host-blas or host-lapack dependencies, just use unoptimized,
> +# in-tree code.
> +HOST_PYTHON_NUMPY_CONF_OPTS = -Dblas="" -Dlapack=""
> +
> # Fixup the npymath.ini prefix path with actual target staging area where
> # numpy core was built. Without this, target builds using numpy distutils
> # extensions like python-scipy, python-numba cannot find -lnpymath since
> --
> 2.39.2
>
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency.
2024-02-05 18:04 ` Yann E. MORIN
@ 2024-02-29 15:38 ` Peter Korsgaard
0 siblings, 0 replies; 6+ messages in thread
From: Peter Korsgaard @ 2024-02-29 15:38 UTC (permalink / raw)
To: Yann E. MORIN; +Cc: gsmecher, buildroot
>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
> Graeme, All,
> On 2024-01-29 12:40 -0800, gsmecher@threespeedlogic.com spake thusly:
>> From: Graeme Smecher <gsmecher@threespeedlogic.com>
>>
>> Before this patch, host-python-numpy attempted (and failed) to detect
>> OpenBLAS as follows:
>>
>> buildroot$ make
>> >>> host-python-numpy 1.25.0 Configuring
>> [...]
>> Run-time dependency openblas found: NO (tried pkgconfig and cmake)
>> WARNING: CMake Toolchain: Failed to determine CMake compilers state
>> Run-time dependency openblas found: YES 0.3.21
>> Dependency openblas found: YES 0.3.21 (cached)
>> Program _build_utils/process_src_template.py found: YES
>> (/path/to/buildroot/output/host/bin/python3
>> /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py)
>> Program _build_utils/tempita.py found: YES
>> (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py)
>>
>> output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44:
>> ERROR: Could not get cmake variable and no default provided for
>> <CMakeDependency OpenBLAS: True None>
>> [...]
> So I dug in the Numpy git history, and before we bumped to 1.25, it was
> using distutils, which would only warn and disable BLAS support if not
> found, but the meson switch made the default to fail.
>> Two obvious ways to correct this:
>>
>> - Add some kind of host-lapack / host-openblas dependency to
>> host-python-numpy, or
>> - Tell host-python-numpy not to look for (or rely on) external code.
>>
>> Because the host numpy environment shouldn't be doing any numerical
>> heavy lifting, we opt for the latter here.
>>
>> Signed-off-by: Graeme Smecher <gsmecher@threespeedlogic.com>
> Applied to master, with an extended commit log about the history,
> thanks.
Committed to 2023.11.x, thanks.
--
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-02-29 15:52 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-01-29 19:54 [Buildroot] host-python-numpy - confused about host openblas dependency Graeme Smecher
2024-01-29 20:22 ` Yann E. MORIN
2024-01-29 20:40 ` [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency gsmecher
2024-01-29 20:58 ` Yann E. MORIN
2024-02-05 18:04 ` Yann E. MORIN
2024-02-29 15:38 ` Peter Korsgaard
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox