Re: [OE-core][PATCH] python3-numpy: upgrade 1.26.4 -> 2.1.2

[Trevor Gamblin <tgamblin@baylibre.com>]

On 2024-11-01 06:08, Alexander Kanavin wrote:
> On Wed, 30 Oct 2024 at 15:23, Mathieu Dubois-Briand via
> lists.openembedded.org
> <mathieu.dubois-briand=bootlin.com@lists.openembedded.org> wrote:
>
>> Differences can be seen here:
>> - https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20241030-a5e77d8y/
>> - https://valkyrie.yocto.io/pub/repro-fail/oe-reproducible-20241030-a5e77d8y/packages/diff-html/
>>
>> Can you have a look please ?
> I believe there are two separate reproducibility issues there:
>
> 1. usr/lib/python3.13/site-packages/numpy/__config__.py contains
> numerous references to compiler and linker flags which contains build
> paths. This is not uncommon, and we generally handle it by patching
> these out after the fact with sed in do_install:append() or similar.
> Plenty of examples all over oe-core.
>
> 2. The second issue is slightly trickier. There's a number of
> generated c files produced by cython and they contain lines like:
>
> static const char __pyx_k_srv_storage_alex_yocto_build_64[] =
> "/srv/storage/alex/yocto/build-64/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/numpy-2.1.2/numpy/random/bit_generator.pyx";
>   PyObject *__pyx_kp_s_srv_storage_alex_yocto_build_64;
> etc.
>
> This needs to be fixed by tracking down where in cython this stuff is
> generated, and replacing real paths (or parts of them) with something
> reproducible, like 'reproducible_build_path' or similar. Then this
> should be submitted or raised with upstream.

Yeah, I've gotten this far, but I think the real issue is that these 
should be configurable via some (possibly-nonexistent) bitbake variable 
in the recipe. I haven't yet determined how to add this stuff yet, though.

In the python3-pandas recipe in meta-python, there's a 
fix_cythonized_sources() function that seems to be of the right spirit:

https://git.openembedded.org/meta-openembedded/tree/meta-python/recipes-devtools/python/python3-pandas_2.2.2.bb#n26

I don't think this is quite the right fix, though - it seems like these 
adjustments should be happening during do_configure if possible, so that 
the host paths aren't being propagated through all of the different 
copies of files like __config__.py...

There are quite a lot of hits under the numpy workdir for "Generated by 
Cython":

tgamblin@megalith ~/workspace/yocto/poky (master)$ grep -r "Generated by 
Cython" build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/temp/run.fix_cythonized_sources.581946: 
for f in `grep -l -r -e '\/* Generated by Cython.*/$' 
/home/tgamblin/workspace/yocto/poky/build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2`; 
do
grep: 
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Utils.cpython-313-x86_64-linux-gnu.so: 
binary file matches
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Utils.py:GENERATED_BY_MARKER 
= "/* Generated by Cython %s */" % cython_version
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Utils.py: 
file_content.startswith(b"/* Generated by Cython ") or
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/Annotate.py: 
<!-- Generated by Cython {watermark} -->
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/Annotate.py: 
<p><span style="border-bottom: solid 1px grey;">Generated by Cython 
{watermark}</span>{more_info}</p>
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/Main.py: 
if u'<!-- Generated by Cython' in html_file.read(100):
grep: 
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/__pycache__/Annotate.cpython-313.pyc: 
binary file matches
grep: 
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/Compiler/__pycache__/Main.cpython-313.pyc: 
binary file matches
grep: 
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/recipe-sysroot-native/usr/lib/python3.13/site-packages/Cython/__pycache__/Utils.cpython-313.pyc: 
binary file matches
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_mt19937.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_common.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_common.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_generator.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_bounded_integers.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_bounded_integers.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_pcg64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_sfc64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_sfc64.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/bit_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/bit_generator.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/mtrand.cpython-313-x86_64-linux-gnu.so.p/numpy/random/mtrand.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/build/numpy/random/_philox.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_philox.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_mt19937.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_common.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_common.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_generator.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_bounded_integers.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_bounded_integers.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_pcg64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_sfc64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_sfc64.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/bit_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/bit_generator.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/mtrand.cpython-313-x86_64-linux-gnu.so.p/numpy/random/mtrand.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/package/usr/src/debug/python3-numpy/2.1.2/numpy/random/_philox.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_philox.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_mt19937.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_mt19937.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_common.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_common.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_generator.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_bounded_integers.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_bounded_integers.pyx.c:/* 
Generated by Cython 3.0.11/
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_pcg64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_pcg64.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_sfc64.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_sfc64.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/bit_generator.cpython-313-x86_64-linux-gnu.so.p/numpy/random/bit_generator.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/mtrand.cpython-313-x86_64-linux-gnu.so.p/numpy/random/mtrand.pyx.c:/* 
Generated by Cython 3.0.11 */
build/tmp/work/core2-64-poky-linux/python3-numpy/2.1.2/packages-split/python3-numpy-src/usr/src/debug/python3-numpy/2.1.2/numpy/random/_philox.cpython-313-x86_64-linux-gnu.so.p/numpy/random/_philox.pyx.c:/* 
Generated by Cython 3.0.11 */

This makes me think that the Cython recipe is also missing a few cases 
in cython_fix_sources() (namely, ../Cython/Compiler/Annotate.py, 
../Cython/Compiler/Main.py, and ../Cython/Utils.py, but locally adding 
those and rebuild doesn't seem to fix the numpy issue.

Still looking.

>
> 3. We should also drop
> # Python pyx -> c -> so build leaves absolute build paths in the code
> INSANE_SKIP:${PN} += "buildpaths"
> INSANE_SKIP:${PN}-src += "buildpaths"
>
> from meta/classes-recipe/python_mesonpy.bbclass when point two is addressed.
Completely agree, it's on my radar. Apparently I need to learn a lot 
more about how all of this is actually happening underneath...
>
> Hope this helps,
> Alex