* [Buildroot] host-python-numpy - confused about host openblas dependency @ 2024-01-29 19:54 Graeme Smecher 2024-01-29 20:22 ` Yann E. MORIN 0 siblings, 1 reply; 6+ messages in thread From: Graeme Smecher @ 2024-01-29 19:54 UTC (permalink / raw) To: buildroot Hi all, Several packages depend on host-python-numpy: buildroot$ grep host-python-numpy -rl package/ package/python-pythran/python-pythran.mk package/python-matplotlib/python-matplotlib.mk package/python-matplotlib/Config.in package/piglit/piglit.mk package/piglit/Config.in package/gnuradio/gnuradio.mk package/gnuradio/Config.in package/python-scipy/python-scipy.mk package/python-scipy/Config.in package/uhd/uhd.mk package/uhd/Config.in However, this package appears to rely on a host-openblas dependency that's implied but never really spelled out. My builds fail as follows: buldroot$ make >>> host-python-numpy 1.25.0 Configuring rm -rf /path/to/buildroot/output/build/host-python-numpy-1.25.0//build mkdir -p /path/to/buildroot/output/build/host-python-numpy-1.25.0//build GIT_DIR=. PATH="/path/to/buildroot/output/host/bin:/path/to/buildroot/output/host/sbin:/home/gsmecher/.local/bin:/home/gsmecher/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin" PKG_CONFIG="/path/to/buildroot/output/host/bin/pkg-config" PKG_CONFIG_SYSROOT_DIR="/" PKG_CONFIG_ALLOW_SYSTEM_CFLAGS=1 PKG_CONFIG_ALLOW_SYSTEM_LIBS=1 PKG_CONFIG_LIBDIR="/path/to/buildroot/output/host/lib/pkgconfig:/path/to/buildroot/output/host/share/pkgconfig" AR="/usr/bin/ar" AS="/usr/bin/as" LD="/usr/bin/ld" NM="/usr/bin/nm" CC="/usr/bin/gcc" GCC="/usr/bin/gcc" CXX="/usr/bin/g++" CPP="/usr/bin/cpp" OBJCOPY="/usr/bin/objcopy" RANLIB="/usr/bin/ranlib" CPPFLAGS="-I/path/to/buildroot/output/host/include" CFLAGS="-O2 -I/path/to/buildroot/output/host/include" CXXFLAGS="-O2 -I/path/to/buildroot/output/host/include" LDFLAGS="-L/path/to/buildroot/output/host/lib -Wl,-rpath,/path/to/buildroot/output/host/lib" INTLTOOL_PERL=/usr/bin/perl PYTHONNOUSERSITE=y /path/to/buildroot/output/host/bin/me son setup --prefix=/path/to/buildroot/output/host --libdir=lib --sysconfdir=/path/to/buildroot/output/host/etc --localstatedir=/path/to/buildroot/output/host/var --default-library=shared --buildtype=release --wrap-mode=nodownload -Dstrip=true /path/to/buildroot/output/build/host-python-numpy-1.25.0/ /path/to/buildroot/output/build/host-python-numpy-1.25.0//build The Meson build system Version: 1.1.1 Source dir: /path/to/buildroot/output/build/host-python-numpy-1.25.0 Build dir: /path/to/buildroot/output/build/host-python-numpy-1.25.0/build Build type: native build Project name: NumPy Project version: 1.24.0.dev0 C compiler for the host machine: /usr/bin/gcc (gcc 12.2.0 "gcc (Debian 12.2.0-14) 12.2.0") C linker for the host machine: /usr/bin/gcc ld.bfd 2.40 C++ compiler for the host machine: /usr/bin/g++ (gcc 12.2.0 "g++ (Debian 12.2.0-14) 12.2.0") C++ linker for the host machine: /usr/bin/g++ ld.bfd 2.40 Cython compiler for the host machine: cython (cython 0.29.36) Host machine cpu family: x86_64 Host machine cpu: x86_64 Program python3 found: YES (/path/to/buildroot/output/host/bin/python3) Found pkg-config: /path/to/buildroot/output/host/bin/pkg-config (1.6.3) Has header "Python.h" with dependency python-3.11: YES Compiler for C supports arguments -fno-strict-aliasing: YES Library m found: YES Found CMake: /path/to/buildroot/output/host/bin/cmake (3.27.7) WARNING: CMake Toolchain: Failed to determine CMake compilers state WARNING: CMake Toolchain: Failed to determine CMake compilers state Run-time dependency openblas found: NO (tried pkgconfig and cmake) WARNING: CMake Toolchain: Failed to determine CMake compilers state Run-time dependency openblas found: YES 0.3.21 Dependency openblas found: YES 0.3.21 (cached) Program _build_utils/process_src_template.py found: YES (/path/to/buildroot/output/host/bin/python3 /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py) Program _build_utils/tempita.py found: YES (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py) output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44: ERROR: Could not get cmake variable and no default provided for <CMakeDependency OpenBLAS: True None> A full log can be found at /path/to/buildroot/output/build/host-python-numpy-1.25.0/build/meson-logs/meson-log.txt make: *** [package/pkg-generic.mk:273: /path/to/buildroot/output/build/host-python-numpy-1.25.0/.stamp_configured] Error 1 Any suggestions? There are enough packages depending on host-numpy that I expect it's something unusual on my end. If I force the build to proceed using system pkg-config I can coax it into proceeding - but this is clearly a hack. thanks, Graeme _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Buildroot] host-python-numpy - confused about host openblas dependency 2024-01-29 19:54 [Buildroot] host-python-numpy - confused about host openblas dependency Graeme Smecher @ 2024-01-29 20:22 ` Yann E. MORIN 2024-01-29 20:40 ` [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency gsmecher 0 siblings, 1 reply; 6+ messages in thread From: Yann E. MORIN @ 2024-01-29 20:22 UTC (permalink / raw) To: Graeme Smecher; +Cc: buildroot Graeme, All, On 2024-01-29 11:54 -0800, Graeme Smecher spake thusly: > Several packages depend on host-python-numpy: [--SNIP--] > However, this package appears to rely on a host-openblas dependency > that's implied but never really spelled out. [--SNIP--] > Any suggestions? See package/python-numpy/python-numpy.mk: it has proper lapack and blas handling for the target variant, but indeed the host variant is left in the dark. > There are enough packages depending on host-numpy that I expect it's > something unusual on my end. Or rather just happenstance that you found it first... > If I force the build to proceed using system pkg-config I can coax it > into proceeding - but this is clearly a hack. Indeed it is not the proper solutoin. The proper way is to edit package/python-numpy/python-numpy.mk and add: HOST_PYTHON_NUMPY_CONF_OPTS = -Dblas="" -Dlapack="" Hopefully they are not needed for the host variant, because we not have host-lapack or host-openblas. Care to send a patch, please? Regards, Yann E. MORIN. -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency. 2024-01-29 20:22 ` Yann E. MORIN @ 2024-01-29 20:40 ` gsmecher 2024-01-29 20:58 ` Yann E. MORIN 2024-02-05 18:04 ` Yann E. MORIN 0 siblings, 2 replies; 6+ messages in thread From: gsmecher @ 2024-01-29 20:40 UTC (permalink / raw) To: buildroot; +Cc: Graeme Smecher, yann.morin.1998 From: Graeme Smecher <gsmecher@threespeedlogic.com> Before this patch, host-python-numpy attempted (and failed) to detect OpenBLAS as follows: buildroot$ make >>> host-python-numpy 1.25.0 Configuring [...] Run-time dependency openblas found: NO (tried pkgconfig and cmake) WARNING: CMake Toolchain: Failed to determine CMake compilers state Run-time dependency openblas found: YES 0.3.21 Dependency openblas found: YES 0.3.21 (cached) Program _build_utils/process_src_template.py found: YES (/path/to/buildroot/output/host/bin/python3 /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py) Program _build_utils/tempita.py found: YES (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py) output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44: ERROR: Could not get cmake variable and no default provided for <CMakeDependency OpenBLAS: True None> [...] Two obvious ways to correct this: - Add some kind of host-lapack / host-openblas dependency to host-python-numpy, or - Tell host-python-numpy not to look for (or rely on) external code. Because the host numpy environment shouldn't be doing any numerical heavy lifting, we opt for the latter here. Signed-off-by: Graeme Smecher <gsmecher@threespeedlogic.com> --- package/python-numpy/python-numpy.mk | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/package/python-numpy/python-numpy.mk b/package/python-numpy/python-numpy.mk index b52e0da17d..cc5dfc9224 100644 --- a/package/python-numpy/python-numpy.mk +++ b/package/python-numpy/python-numpy.mk @@ -36,6 +36,10 @@ else PYTHON_NUMPY_CONF_OPTS += -Dblas="" endif +# Rather than add a host-blas or host-lapack dependencies, just use unoptimized, +# in-tree code. +HOST_PYTHON_NUMPY_CONF_OPTS = -Dblas="" -Dlapack="" + # Fixup the npymath.ini prefix path with actual target staging area where # numpy core was built. Without this, target builds using numpy distutils # extensions like python-scipy, python-numba cannot find -lnpymath since -- 2.39.2 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency. 2024-01-29 20:40 ` [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency gsmecher @ 2024-01-29 20:58 ` Yann E. MORIN 2024-02-05 18:04 ` Yann E. MORIN 1 sibling, 0 replies; 6+ messages in thread From: Yann E. MORIN @ 2024-01-29 20:58 UTC (permalink / raw) To: gsmecher; +Cc: James Hilliard, buildroot Graeme, All, +James On 2024-01-29 12:40 -0800, gsmecher@threespeedlogic.com spake thusly: > From: Graeme Smecher <gsmecher@threespeedlogic.com> > > Before this patch, host-python-numpy attempted (and failed) to detect > OpenBLAS as follows: > > buildroot$ make > >>> host-python-numpy 1.25.0 Configuring > [...] > Run-time dependency openblas found: NO (tried pkgconfig and cmake) > WARNING: CMake Toolchain: Failed to determine CMake compilers state > Run-time dependency openblas found: YES 0.3.21 > Dependency openblas found: YES 0.3.21 (cached) > Program _build_utils/process_src_template.py found: YES (/path/to/buildroot/output/host/bin/python3 /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py) > Program _build_utils/tempita.py found: YES (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py) > > output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44: ERROR: Could not get cmake variable and no default provided for <CMakeDependency OpenBLAS: True None> > [...] > > Two obvious ways to correct this: > > - Add some kind of host-lapack / host-openblas dependency to > host-python-numpy, or > - Tell host-python-numpy not to look for (or rely on) external code. > > Because the host numpy environment shouldn't be doing any numerical > heavy lifting, we opt for the latter here. > > Signed-off-by: Graeme Smecher <gsmecher@threespeedlogic.com> Good commit log, but there is a missing piece to it, see below... James: as far as I understand, this was the situation before the bump to 1.25.0 and the switch to meson, because we did not have the BLAS=None and LAPACK=None options before that either, but had no such failure report. So I guess the defaults changed with the switch to meson, which might explain why we did not notice before? If so, we need to add that info to the commit log, to decide whether this needs backporting to 2023.11, and what to do about 2023.02 LTS that does not have 1.25. Regards, Yann E. MORIN. > --- > package/python-numpy/python-numpy.mk | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/package/python-numpy/python-numpy.mk b/package/python-numpy/python-numpy.mk > index b52e0da17d..cc5dfc9224 100644 > --- a/package/python-numpy/python-numpy.mk > +++ b/package/python-numpy/python-numpy.mk > @@ -36,6 +36,10 @@ else > PYTHON_NUMPY_CONF_OPTS += -Dblas="" > endif > > +# Rather than add a host-blas or host-lapack dependencies, just use unoptimized, > +# in-tree code. > +HOST_PYTHON_NUMPY_CONF_OPTS = -Dblas="" -Dlapack="" > + > # Fixup the npymath.ini prefix path with actual target staging area where > # numpy core was built. Without this, target builds using numpy distutils > # extensions like python-scipy, python-numba cannot find -lnpymath since > -- > 2.39.2 > -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency. 2024-01-29 20:40 ` [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency gsmecher 2024-01-29 20:58 ` Yann E. MORIN @ 2024-02-05 18:04 ` Yann E. MORIN 2024-02-29 15:38 ` Peter Korsgaard 1 sibling, 1 reply; 6+ messages in thread From: Yann E. MORIN @ 2024-02-05 18:04 UTC (permalink / raw) To: gsmecher; +Cc: buildroot Graeme, All, On 2024-01-29 12:40 -0800, gsmecher@threespeedlogic.com spake thusly: > From: Graeme Smecher <gsmecher@threespeedlogic.com> > > Before this patch, host-python-numpy attempted (and failed) to detect > OpenBLAS as follows: > > buildroot$ make > >>> host-python-numpy 1.25.0 Configuring > [...] > Run-time dependency openblas found: NO (tried pkgconfig and cmake) > WARNING: CMake Toolchain: Failed to determine CMake compilers state > Run-time dependency openblas found: YES 0.3.21 > Dependency openblas found: YES 0.3.21 (cached) > Program _build_utils/process_src_template.py found: YES (/path/to/buildroot/output/host/bin/python3 /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py) > Program _build_utils/tempita.py found: YES (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py) > > output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44: ERROR: Could not get cmake variable and no default provided for <CMakeDependency OpenBLAS: True None> > [...] So I dug in the Numpy git history, and before we bumped to 1.25, it was using distutils, which would only warn and disable BLAS support if not found, but the meson switch made the default to fail. > Two obvious ways to correct this: > > - Add some kind of host-lapack / host-openblas dependency to > host-python-numpy, or > - Tell host-python-numpy not to look for (or rely on) external code. > > Because the host numpy environment shouldn't be doing any numerical > heavy lifting, we opt for the latter here. > > Signed-off-by: Graeme Smecher <gsmecher@threespeedlogic.com> Applied to master, with an extended commit log about the history, thanks. Regards, Yann E. MORIN. > --- > package/python-numpy/python-numpy.mk | 4 ++++ > 1 file changed, 4 insertions(+) > > diff --git a/package/python-numpy/python-numpy.mk b/package/python-numpy/python-numpy.mk > index b52e0da17d..cc5dfc9224 100644 > --- a/package/python-numpy/python-numpy.mk > +++ b/package/python-numpy/python-numpy.mk > @@ -36,6 +36,10 @@ else > PYTHON_NUMPY_CONF_OPTS += -Dblas="" > endif > > +# Rather than add a host-blas or host-lapack dependencies, just use unoptimized, > +# in-tree code. > +HOST_PYTHON_NUMPY_CONF_OPTS = -Dblas="" -Dlapack="" > + > # Fixup the npymath.ini prefix path with actual target staging area where > # numpy core was built. Without this, target builds using numpy distutils > # extensions like python-scipy, python-numba cannot find -lnpymath since > -- > 2.39.2 > > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency. 2024-02-05 18:04 ` Yann E. MORIN @ 2024-02-29 15:38 ` Peter Korsgaard 0 siblings, 0 replies; 6+ messages in thread From: Peter Korsgaard @ 2024-02-29 15:38 UTC (permalink / raw) To: Yann E. MORIN; +Cc: gsmecher, buildroot >>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes: > Graeme, All, > On 2024-01-29 12:40 -0800, gsmecher@threespeedlogic.com spake thusly: >> From: Graeme Smecher <gsmecher@threespeedlogic.com> >> >> Before this patch, host-python-numpy attempted (and failed) to detect >> OpenBLAS as follows: >> >> buildroot$ make >> >>> host-python-numpy 1.25.0 Configuring >> [...] >> Run-time dependency openblas found: NO (tried pkgconfig and cmake) >> WARNING: CMake Toolchain: Failed to determine CMake compilers state >> Run-time dependency openblas found: YES 0.3.21 >> Dependency openblas found: YES 0.3.21 (cached) >> Program _build_utils/process_src_template.py found: YES >> (/path/to/buildroot/output/host/bin/python3 >> /path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/process_src_template.py) >> Program _build_utils/tempita.py found: YES >> (/path/to/buildroot/output/build/host-python-numpy-1.25.0/numpy/_build_utils/tempita.py) >> >> output/build/host-python-numpy-1.25.0/numpy/meson.build:201:44: >> ERROR: Could not get cmake variable and no default provided for >> <CMakeDependency OpenBLAS: True None> >> [...] > So I dug in the Numpy git history, and before we bumped to 1.25, it was > using distutils, which would only warn and disable BLAS support if not > found, but the meson switch made the default to fail. >> Two obvious ways to correct this: >> >> - Add some kind of host-lapack / host-openblas dependency to >> host-python-numpy, or >> - Tell host-python-numpy not to look for (or rely on) external code. >> >> Because the host numpy environment shouldn't be doing any numerical >> heavy lifting, we opt for the latter here. >> >> Signed-off-by: Graeme Smecher <gsmecher@threespeedlogic.com> > Applied to master, with an extended commit log about the history, > thanks. Committed to 2023.11.x, thanks. -- Bye, Peter Korsgaard _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-02-29 15:52 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2024-01-29 19:54 [Buildroot] host-python-numpy - confused about host openblas dependency Graeme Smecher 2024-01-29 20:22 ` Yann E. MORIN 2024-01-29 20:40 ` [Buildroot] [PATCH] host-python-numpy: correct implied host-lapack/host-blas dependency gsmecher 2024-01-29 20:58 ` Yann E. MORIN 2024-02-05 18:04 ` Yann E. MORIN 2024-02-29 15:38 ` Peter Korsgaard
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox