* [OE-core][PATCH] python3-numpy: upgrade 1.26.4 -> 2.1.2
@ 2024-10-29 19:37 Trevor Gamblin
2024-10-30 14:22 ` Mathieu Dubois-Briand
0 siblings, 1 reply; 8+ messages in thread
From: Trevor Gamblin @ 2024-10-29 19:37 UTC (permalink / raw)
To: openembedded-core
- This release contains numerous API changes and bug fixes.
Changelog: https://github.com/numpy/numpy/releases
- Notably, the build backend has switched to mesonpy since 1.26.4.
- Patch 0001-numpy-core-Define-RISCV-32-support.patch had to be updated
to change the paths for the target files. It has now been merged
upstream, but isn't included in any release yet. PR for reference:
https://github.com/numpy/numpy/pull/17780
- Builds OK and seems to be compatible with current piglit and pandas
(meta-python), which was not the case for the previous upgrade attempt
to numpy 2.0.1.
- For numpy, add 'pkgconfig' to inherit to avoid errors like this at
do_compile:
| Found Pkg-config: NO
| Run-time dependency python found: YES 3.12
| Has header "Python.h" with dependency python: NO
|
| ../numpy-2.0.0/meson.build:44:2: ERROR: Problem encountered: Cannot compile `Python.h`. Perhaps you need to install python-dev|python-devel
|
| A full log can be found at /home/tgamblin/workspace/yocto/poky/build/tmp/work/core2-64-poky-linux/python3-numpy/2.0.0/build/meson-logs/meson-log.txt
- Also modify the FILES:${PN}-staticdev line to fix a QA issue.
License-Update: Change copyright year to 2024
Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
---
...1-numpy-core-Define-RISCV-32-support.patch | 28 +++++++++++--------
...numpy_1.26.4.bb => python3-numpy_2.1.2.bb} | 10 ++++---
2 files changed, 22 insertions(+), 16 deletions(-)
rename meta/recipes-devtools/python/{python3-numpy_1.26.4.bb => python3-numpy_2.1.2.bb} (84%)
diff --git a/meta/recipes-devtools/python/python3-numpy/0001-numpy-core-Define-RISCV-32-support.patch b/meta/recipes-devtools/python/python3-numpy/0001-numpy-core-Define-RISCV-32-support.patch
index 676bdbb3afd..8dfb20edbb9 100644
--- a/meta/recipes-devtools/python/python3-numpy/0001-numpy-core-Define-RISCV-32-support.patch
+++ b/meta/recipes-devtools/python/python3-numpy/0001-numpy-core-Define-RISCV-32-support.patch
@@ -1,4 +1,4 @@
-From eb6d6579150bf4684603ce377c51e90ad3bb8109 Mon Sep 17 00:00:00 2001
+From df38843aaa7184115d262f14eae23ff38a20372a Mon Sep 17 00:00:00 2001
From: Khem Raj <raj.khem@gmail.com>
Date: Sun, 15 Nov 2020 15:32:39 -0800
Subject: [PATCH] numpy/core: Define RISCV-32 support
@@ -7,15 +7,19 @@ Helps compile on riscv32
Upstream-Status: Submitted [https://github.com/numpy/numpy/pull/17780]
Signed-off-by: Khem Raj <raj.khem@gmail.com>
+
+Update the patch with the new paths for numpy release 2.0.0.
+
+Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
---
- numpy/core/include/numpy/npy_cpu.h | 3 +++
- numpy/core/include/numpy/npy_endian.h | 1 +
+ numpy/_core/include/numpy/npy_cpu.h | 3 +++
+ numpy/_core/include/numpy/npy_endian.h | 1 +
2 files changed, 4 insertions(+)
-diff --git a/numpy/core/include/numpy/npy_cpu.h b/numpy/core/include/numpy/npy_cpu.h
-index 78d229e..04be511 100644
---- a/numpy/core/include/numpy/npy_cpu.h
-+++ b/numpy/core/include/numpy/npy_cpu.h
+diff --git a/numpy/_core/include/numpy/npy_cpu.h b/numpy/_core/include/numpy/npy_cpu.h
+index a19f8e6bbd..d824d4efb6 100644
+--- a/numpy/_core/include/numpy/npy_cpu.h
++++ b/numpy/_core/include/numpy/npy_cpu.h
@@ -19,6 +19,7 @@
* NPY_CPU_ARCEB
* NPY_CPU_RISCV64
@@ -33,10 +37,10 @@ index 78d229e..04be511 100644
#elif defined(__loongarch__)
#define NPY_CPU_LOONGARCH
#elif defined(__EMSCRIPTEN__)
-diff --git a/numpy/core/include/numpy/npy_endian.h b/numpy/core/include/numpy/npy_endian.h
-index 5e58a7f..0926212 100644
---- a/numpy/core/include/numpy/npy_endian.h
-+++ b/numpy/core/include/numpy/npy_endian.h
+diff --git a/numpy/_core/include/numpy/npy_endian.h b/numpy/_core/include/numpy/npy_endian.h
+index 5e58a7f52c..09262120bf 100644
+--- a/numpy/_core/include/numpy/npy_endian.h
++++ b/numpy/_core/include/numpy/npy_endian.h
@@ -49,6 +49,7 @@
|| defined(NPY_CPU_PPC64LE) \
|| defined(NPY_CPU_ARCEL) \
@@ -46,5 +50,5 @@ index 5e58a7f..0926212 100644
|| defined(NPY_CPU_WASM)
#define NPY_BYTE_ORDER NPY_LITTLE_ENDIAN
--
-2.20.1
+2.45.2
diff --git a/meta/recipes-devtools/python/python3-numpy_1.26.4.bb b/meta/recipes-devtools/python/python3-numpy_2.1.2.bb
similarity index 84%
rename from meta/recipes-devtools/python/python3-numpy_1.26.4.bb
rename to meta/recipes-devtools/python/python3-numpy_2.1.2.bb
index 914245507c7..83e921f753c 100644
--- a/meta/recipes-devtools/python/python3-numpy_1.26.4.bb
+++ b/meta/recipes-devtools/python/python3-numpy_2.1.2.bb
@@ -3,7 +3,7 @@ HOMEPAGE = "https://numpy.org/"
DESCRIPTION = "NumPy is the fundamental package needed for scientific computing with Python."
SECTION = "devel/python"
LICENSE = "BSD-3-Clause & BSD-2-Clause & PSF-2.0 & Apache-2.0 & MIT"
-LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=a752eb20459cf74a9d84ee4825e8317c"
+LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=1de863c37a83e71b1e97b64d036ea78b"
SRCNAME = "numpy"
@@ -13,14 +13,14 @@ SRC_URI = "${GITHUB_BASE_URI}/download/v${PV}/${SRCNAME}-${PV}.tar.gz \
file://fix_reproducibility.patch \
file://run-ptest \
"
-SRC_URI[sha256sum] = "2a02aba9ed12e4ac4eb3ea9421c420301a0c6460d9830d74a9df87efa4912010"
+SRC_URI[sha256sum] = "13532a088217fa624c99b843eeb54640de23b3414b14aa66d023805eb731066c"
GITHUB_BASE_URI = "https://github.com/numpy/numpy/releases"
UPSTREAM_CHECK_REGEX = "releases/tag/v?(?P<pver>\d+(\.\d+)+)$"
DEPENDS += "python3-cython-native"
-inherit ptest setuptools3 github-releases
+inherit ptest python_mesonpy pkgconfig github-releases
S = "${WORKDIR}/numpy-${PV}"
@@ -30,7 +30,9 @@ do_compile:prepend() {
export NPY_DISABLE_SVML=1
}
-FILES:${PN}-staticdev += "${PYTHON_SITEPACKAGES_DIR}/numpy/core/lib/*.a ${PYTHON_SITEPACKAGES_DIR}/numpy/random/lib/*.a"
+FILES:${PN}-staticdev += "${PYTHON_SITEPACKAGES_DIR}/numpy/_core/lib/*.a \
+ ${PYTHON_SITEPACKAGES_DIR}/numpy/random/lib/*.a \
+"
# install what is needed for numpy.test()
RDEPENDS:${PN} = "python3-unittest \
--
2.39.5
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [OE-core][PATCH] python3-numpy: upgrade 1.26.4 -> 2.1.2
2024-10-29 19:37 [OE-core][PATCH] python3-numpy: upgrade 1.26.4 -> 2.1.2 Trevor Gamblin
@ 2024-10-30 14:22 ` Mathieu Dubois-Briand
2024-11-01 10:08 ` Alexander Kanavin
0 siblings, 1 reply; 8+ messages in thread
From: Mathieu Dubois-Briand @ 2024-10-30 14:22 UTC (permalink / raw)
To: Trevor Gamblin; +Cc: openembedded-core
On Tue, Oct 29, 2024 at 03:37:45PM -0400, Trevor Gamblin wrote:
> - This release contains numerous API changes and bug fixes.
> Changelog: https://github.com/numpy/numpy/releases
>
> - Notably, the build backend has switched to mesonpy since 1.26.4.
>
> - Patch 0001-numpy-core-Define-RISCV-32-support.patch had to be updated
> to change the paths for the target files. It has now been merged
> upstream, but isn't included in any release yet. PR for reference:
>
> https://github.com/numpy/numpy/pull/17780
>
> - Builds OK and seems to be compatible with current piglit and pandas
> (meta-python), which was not the case for the previous upgrade attempt
> to numpy 2.0.1.
>
> - For numpy, add 'pkgconfig' to inherit to avoid errors like this at
> do_compile:
>
> | Found Pkg-config: NO
> | Run-time dependency python found: YES 3.12
> | Has header "Python.h" with dependency python: NO
> |
> | ../numpy-2.0.0/meson.build:44:2: ERROR: Problem encountered: Cannot compile `Python.h`. Perhaps you need to install python-dev|python-devel
> |
> | A full log can be found at /home/tgamblin/workspace/yocto/poky/build/tmp/work/core2-64-poky-linux/python3-numpy/2.0.0/build/meson-logs/meson-log.txt
>
> - Also modify the FILES:${PN}-staticdev line to fix a QA issue.
>
> License-Update: Change copyright year to 2024
>
> Signed-off-by: Trevor Gamblin <tgamblin@baylibre.com>
Hi Trevor,
It looks like this is breaking reproducible autobuilder:
AssertionError: The following deb packages are different and not in
exclusion list:
/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/python3-numpy-dbg_2.1.2-r0_amd64.deb
/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/python3-numpy-src_2.1.2-r0_amd64.deb
/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/deb/./core2-64/python3-numpy_2.1.2-r0_amd64.deb
The following ipk packages are different and not in exclusion list:
/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/python3-numpy-dbg_2.1.2-r0_core2-64.ipk
/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/python3-numpy-src_2.1.2-r0_core2-64.ipk
/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/ipk/./core2-64/python3-numpy_2.1.2-r0_core2-64.ipk
The following rpm packages are different and not in exclusion list:
/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/python3-numpy-2.1.2-r0.core2_64.rpm
/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/python3-numpy-dbg-2.1.2-r0.core2_64.rpm
/srv/pokybuild/yocto-worker/reproducible/build/build-st/reproducibleB-extended/tmp/deploy/rpm/./core2_64/python3-numpy-src-2.1.2-r0.core2_64.rpm
https://valkyrie.yoctoproject.org/#/builders/37/builds/369/steps/12/logs/stdio
Differences can be seen here:
- https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20241030-a5e77d8y/
- https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20241030-a5e77d8y/packages/diff-html/
Can you have a look please ?
--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [OE-core][PATCH] python3-numpy: upgrade 1.26.4 -> 2.1.2
2024-10-30 14:22 ` Mathieu Dubois-Briand
@ 2024-11-01 10:08 ` Alexander Kanavin
2024-11-01 13:17 ` Trevor Gamblin
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Alexander Kanavin @ 2024-11-01 10:08 UTC (permalink / raw)
To: mathieu.dubois-briand, Trevor Gamblin, openembedded-core; +Cc: Richard Purdie
On Wed, 30 Oct 2024 at 15:23, Mathieu Dubois-Briand via
lists.openembedded.org
<mathieu.dubois-briand=bootlin.com@lists.openembedded.org> wrote:
> Differences can be seen here:
> - https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20241030-a5e77d8y/
> - https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20241030-a5e77d8y/packages/diff-html/
>
> Can you have a look please ?
I believe there are two separate reproducibility issues there:
1. usr/lib/python3.13/site-packages/numpy/__config__.py contains
numerous references to compiler and linker flags which contains build
paths. This is not uncommon, and we generally handle it by patching
these out after the fact with sed in do_install:append() or similar.
Plenty of examples all over oe-core.
2. The second issue is slightly trickier. There's a number of
generated c files produced by cython and they contain lines like:
static const char __pyx_k_srv_storage_alex_yocto_build_64[] =
"/srv/storage/alex/yocto/build-64/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/numpy-2.1.2/numpy/random/bit_generator.pyx";
PyObject *__pyx_kp_s_srv_storage_alex_yocto_build_64;
etc.
This needs to be fixed by tracking down where in cython this stuff is
generated, and replacing real paths (or parts of them) with something
reproducible, like 'reproducible_build_path' or similar. Then this
should be submitted or raised with upstream.
3. We should also drop
# Python pyx -> c -> so build leaves absolute build paths in the code
INSANE_SKIP:${PN} += "buildpaths"
INSANE_SKIP:${PN}-src += "buildpaths"
from meta/classes-recipe/python_mesonpy.bbclass when point two is addressed.
Hope this helps,
Alex
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [OE-core][PATCH] python3-numpy: upgrade 1.26.4 -> 2.1.2
2024-11-01 10:08 ` Alexander Kanavin
@ 2024-11-01 13:17 ` Trevor Gamblin
2024-11-01 21:47 ` Ross Burton
2024-11-05 16:17 ` Ross Burton
2 siblings, 0 replies; 8+ messages in thread
From: Trevor Gamblin @ 2024-11-01 13:17 UTC (permalink / raw)
To: Alexander Kanavin, mathieu.dubois-briand, openembedded-core
Cc: Richard Purdie
On 2024-11-01 06:08, Alexander Kanavin wrote:
> On Wed, 30 Oct 2024 at 15:23, Mathieu Dubois-Briand via
> lists.openembedded.org
> <mathieu.dubois-briand=bootlin.com@lists.openembedded.org> wrote:
>
>> Differences can be seen here:
>> - https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20241030-a5e77d8y/
>> - https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20241030-a5e77d8y/packages/diff-html/
>>
>> Can you have a look please ?
> I believe there are two separate reproducibility issues there:
>
> 1. usr/lib/python3.13/site-packages/numpy/__config__.py contains
> numerous references to compiler and linker flags which contains build
> paths. This is not uncommon, and we generally handle it by patching
> these out after the fact with sed in do_install:append() or similar.
> Plenty of examples all over oe-core.
>
> 2. The second issue is slightly trickier. There's a number of
> generated c files produced by cython and they contain lines like:
>
> static const char __pyx_k_srv_storage_alex_yocto_build_64[] =
> "/srv/storage/alex/yocto/build-64/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/numpy-2.1.2/numpy/random/bit_generator.pyx";
> PyObject *__pyx_kp_s_srv_storage_alex_yocto_build_64;
> etc.
>
> This needs to be fixed by tracking down where in cython this stuff is
> generated, and replacing real paths (or parts of them) with something
> reproducible, like 'reproducible_build_path' or similar. Then this
> should be submitted or raised with upstream.
Yeah, I've gotten this far, but I think the real issue is that these
should be configurable via some (possibly-nonexistent) bitbake variable
in the recipe. I haven't yet determined how to add this stuff yet, though.
In the python3-pandas recipe in meta-python, there's a
fix_cythonized_sources() function that seems to be of the right spirit:
https://git.openembedded.org/meta-openembedded/tree/meta-python/recipes-devtools/python/python3-pandas_2.2.2.bb#n26
I don't think this is quite the right fix, though - it seems like these
adjustments should be happening during do_configure if possible, so that
the host paths aren't being propagated through all of the different
copies of files like __config__.py...
There are quite a lot of hits under the numpy workdir for "Generated by
Cython":
tgamblin@megalith ~/workspace/yocto/poky (master)$ grep -r "Generated by
Cython" build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/temp/run.fix_cythonized_sources.581946:
for f in `grep -l -r -e '\/* Generated by Cython.*/$'
/home/tgamblin/workspace/yocto/poky/build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2`;
do
grep:
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Utils.cpython-313-x86_64-linux-gnu.so:
binary file matches
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Utils.py:GENERATED_BY_MARKER
= "/* Generated by Cython %s */" % cython_version
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Utils.py:
file_content.startswith(b"/* Generated by Cython ") or
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/Annotate.py:
<!-- Generated by Cython {watermark} -->
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/Annotate.py:
<p><span style="border-bottom: solid 1px grey;">Generated by Cython
{watermark}</span>{more_info}</p>
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/Main.py:
if u'<!-- Generated by Cython' in html_file.read(100):
grep:
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/__pycache__/Annotate.cpython-313.pyc:
binary file matches
grep:
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/__pycache__/Main.cpython-313.pyc:
binary file matches
grep:
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/__pycache__/Utils.cpython-313.pyc:
binary file matches
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_mt19937.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_common.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_common.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_generator.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_bounded_integers.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_bounded_integers.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_pcg64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_sfc64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_sfc64.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/bit_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/bit_generator.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/mtrand.cpython-313-x86_64-linux-gnu.so.p/numpy/random/mtrand.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_philox.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_philox.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_mt19937.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_common.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_common.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_generator.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_bounded_integers.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_bounded_integers.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_pcg64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_sfc64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_sfc64.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/bit_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/bit_generator.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/mtrand.cpython-313-x86_64-linux-gnu.so.p/numpy/random/mtrand.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_philox.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_philox.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_mt19937.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_common.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_common.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_generator.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_bounded_integers.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_bounded_integers.pyx.c:/*
Generated by Cython 3.0.11/
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_pcg64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_sfc64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_sfc64.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/bit_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/bit_generator.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/mtrand.cpython-313-x86_64-linux-gnu.so.p/numpy/random/mtrand.pyx.c:/*
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_philox.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_philox.pyx.c:/*
Generated by Cython 3.0.11 */
This makes me think that the Cython recipe is also missing a few cases
in cython_fix_sources() (namely, ../Cython/Compiler/Annotate.py,
../Cython/Compiler/Main.py, and ../Cython/Utils.py, but locally adding
those and rebuild doesn't seem to fix the numpy issue.
Still looking.
>
> 3. We should also drop
> # Python pyx -> c -> so build leaves absolute build paths in the code
> INSANE_SKIP:${PN} += "buildpaths"
> INSANE_SKIP:${PN}-src += "buildpaths"
>
> from meta/classes-recipe/python_mesonpy.bbclass when point two is addressed.
Completely agree, it's on my radar. Apparently I need to learn a lot
more about how all of this is actually happening underneath...
>
> Hope this helps,
> Alex
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [OE-core][PATCH] python3-numpy: upgrade 1.26.4 -> 2.1.2
2024-11-01 10:08 ` Alexander Kanavin
2024-11-01 13:17 ` Trevor Gamblin
@ 2024-11-01 21:47 ` Ross Burton
2024-11-04 14:20 ` Ross Burton
2024-11-05 16:17 ` Ross Burton
2 siblings, 1 reply; 8+ messages in thread
From: Ross Burton @ 2024-11-01 21:47 UTC (permalink / raw)
To: Alexander Kanavin
Cc: Mathieu Dubois-Briand, Trevor Gamblin, openembedded-core,
Richard Purdie
On 1 Nov 2024, at 10:08, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote:
> 2. The second issue is slightly trickier. There's a number of
> generated c files produced by cython and they contain lines like:
>
> static const char __pyx_k_srv_storage_alex_yocto_build_64[] =
> "/srv/storage/alex/yocto/build-64/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/numpy-2.1.2/numpy/random/bit_generator.pyx";
> PyObject *__pyx_kp_s_srv_storage_alex_yocto_build_64;
> etc.
>
> This needs to be fixed by tracking down where in cython this stuff is
> generated, and replacing real paths (or parts of them) with something
> reproducible, like 'reproducible_build_path' or similar. Then this
> should be submitted or raised with upstream.
There’s an upstream ticket already: https://github.com/cython/cython/issues/5949.
> 3. We should also drop
> # Python pyx -> c -> so build leaves absolute build paths in the code
> INSANE_SKIP:${PN} += "buildpaths"
> INSANE_SKIP:${PN}-src += "buildpaths"
>
> from meta/classes-recipe/python_mesonpy.bbclass when point two is addressed.
This should be removed from that class anyway, this is a cython workaround so doesn’t have any place in the mesonpy class.
There’s also that .c files generated by cython embed the python include dir (thus, build path) inside a “Cython Metadata” blob. A number of recipes in meta-python strip this out so I’m wondering if we need a cython.bbclass which does something like:
do_compile[postfuncs] = “strip_cython_metadata”
strip_cython_metadata() {
find ${B} -name \*.c -print0 | xargs -0 sed -i -e "/BEGIN: Cython Metadata/,/END: Cython Metadata/d”
}
Alternatively, patch cython to not generate that metadata in the first place. Apparently there are tools that read it, so leaving it in the tree but stripping it before install seems like the safest option.
Ross
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [OE-core][PATCH] python3-numpy: upgrade 1.26.4 -> 2.1.2
2024-11-01 21:47 ` Ross Burton
@ 2024-11-04 14:20 ` Ross Burton
2024-11-04 14:24 ` Trevor Gamblin
0 siblings, 1 reply; 8+ messages in thread
From: Ross Burton @ 2024-11-04 14:20 UTC (permalink / raw)
To: Alexander Kanavin
Cc: Mathieu Dubois-Briand, Trevor Gamblin, openembedded-core,
Richard Purdie
On 1 Nov 2024, at 21:47, Ross Burton <Ross.Burton@arm.com> wrote:
>
> On 1 Nov 2024, at 10:08, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote:
>> 2. The second issue is slightly trickier. There's a number of
>> generated c files produced by cython and they contain lines like:
>>
>> static const char __pyx_k_srv_storage_alex_yocto_build_64[] =
>> "/srv/storage/alex/yocto/build-64/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/numpy-2.1.2/numpy/random/bit_generator.pyx";
>> PyObject *__pyx_kp_s_srv_storage_alex_yocto_build_64;
>> etc.
>>
>> This needs to be fixed by tracking down where in cython this stuff is
>> generated, and replacing real paths (or parts of them) with something
>> reproducible, like 'reproducible_build_path' or similar. Then this
>> should be submitted or raised with upstream.
I’ve got a prototype patch that fixes this in cython itself, meaning a pretty large chunk of horrible code in recipes (that silenced the buildpath warnings, but didnt actually fix reproduciblity) can be removed. I’ll hopefully get it good enough to post today.
Ross
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core][PATCH] python3-numpy: upgrade 1.26.4 -> 2.1.2
2024-11-04 14:20 ` Ross Burton
@ 2024-11-04 14:24 ` Trevor Gamblin
0 siblings, 0 replies; 8+ messages in thread
From: Trevor Gamblin @ 2024-11-04 14:24 UTC (permalink / raw)
To: Ross Burton, Alexander Kanavin
Cc: Mathieu Dubois-Briand, openembedded-core, Richard Purdie
On 2024-11-04 09:20, Ross Burton wrote:
> On 1 Nov 2024, at 21:47, Ross Burton <Ross.Burton@arm.com> wrote:
>> On 1 Nov 2024, at 10:08, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote:
>>> 2. The second issue is slightly trickier. There's a number of
>>> generated c files produced by cython and they contain lines like:
>>>
>>> static const char __pyx_k_srv_storage_alex_yocto_build_64[] =
>>> "/srv/storage/alex/yocto/build-64/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/numpy-2.1.2/numpy/random/bit_generator.pyx";
>>> PyObject *__pyx_kp_s_srv_storage_alex_yocto_build_64;
>>> etc.
>>>
>>> This needs to be fixed by tracking down where in cython this stuff is
>>> generated, and replacing real paths (or parts of them) with something
>>> reproducible, like 'reproducible_build_path' or similar. Then this
>>> should be submitted or raised with upstream.
> I’ve got a prototype patch that fixes this in cython itself, meaning a pretty large chunk of horrible code in recipes (that silenced the buildpath warnings, but didnt actually fix reproduciblity) can be removed. I’ll hopefully get it good enough to post today.
Let me know if you want help testing - I started looking at the PR you
mentioned over the weekend but didn't get anywhere yet.
>
> Ross
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [OE-core][PATCH] python3-numpy: upgrade 1.26.4 -> 2.1.2
2024-11-01 10:08 ` Alexander Kanavin
2024-11-01 13:17 ` Trevor Gamblin
2024-11-01 21:47 ` Ross Burton
@ 2024-11-05 16:17 ` Ross Burton
2 siblings, 0 replies; 8+ messages in thread
From: Ross Burton @ 2024-11-05 16:17 UTC (permalink / raw)
To: Alexander Kanavin, Trevor Gamblin
Cc: Mathieu Dubois-Briand, openembedded-core, Richard Purdie
On 1 Nov 2024, at 10:08, Alexander Kanavin via lists.openembedded.org <alex.kanavin=gmail.com@lists.openembedded.org> wrote:
> 2. The second issue is slightly trickier. There's a number of
> generated c files produced by cython and they contain lines like:
>
> static const char __pyx_k_srv_storage_alex_yocto_build_64[] =
> "/srv/storage/alex/yocto/build-64/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/numpy-2.1.2/numpy/random/bit_generator.pyx";
> PyObject *__pyx_kp_s_srv_storage_alex_yocto_build_64;
> etc.
>
> This needs to be fixed by tracking down where in cython this stuff is
> generated, and replacing real paths (or parts of them) with something
> reproducible, like 'reproducible_build_path' or similar. Then this
> should be submitted or raised with upstream.
I’ve just posted a patchset that fixes this. The patch isn’t yet upstreamed but I’m working on that, and I think integrating it to test it would be useful data for upstream.
> 3. We should also drop
> # Python pyx -> c -> so build leaves absolute build paths in the code
> INSANE_SKIP:${PN} += "buildpaths"
> INSANE_SKIP:${PN}-src += "buildpaths"
>
> from meta/classes-recipe/python_mesonpy.bbclass when point two is addressed.
My patchset also drops that.
Ross
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2024-11-05 16:17 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-10-29 19:37 [OE-core][PATCH] python3-numpy: upgrade 1.26.4 -> 2.1.2 Trevor Gamblin
2024-10-30 14:22 ` Mathieu Dubois-Briand
2024-11-01 10:08 ` Alexander Kanavin
2024-11-01 13:17 ` Trevor Gamblin
2024-11-01 21:47 ` Ross Burton
2024-11-04 14:20 ` Ross Burton
2024-11-04 14:24 ` Trevor Gamblin
2024-11-05 16:17 ` Ross Burton
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.