[Buildroot] [Bug 14886] New: Python ctypes find_library() not working correctly.

[Buildroot] [Bug 14886] New: Python ctypes find_library() not working correctly.

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14886

            Bug ID: 14886
           Summary: Python ctypes find_library() not working correctly.
           Product: buildroot
           Version: 2022.02.3
          Hardware: PC
                OS: All
            Status: NEW
          Severity: normal
          Priority: P5
         Component: Other
          Assignee: unassigned@buildroot.uclibc.org
          Reporter: pmorici@dev295.com
                CC: buildroot@uclibc.org
  Target Milestone: ---

The find_library() function in the ctypes Python standard library is not
working properly.

  https://docs.python.org/3/library/ctypes.html#finding-shared-libraries

You can reproduce the issue by trying the following in the python shell...

>>> from ctypes.util import find_library
>>> find_library("c")

This should return the string "libc.so.0" but returns nothing, this is the same
for any library name I tried even though you can see the libraries are
installed and "ldconfig -v" lists the libraries.

This has the effect of causing libraries like pyusb to not work properly
because they reply on ctypes & find_library() to find libusb.

Digging a bit deeper on how find_library() works it calls either ldconfig -p,
ld, or gcc.  buildroot doesn't have ld or gcc installed so I assume that means
it is using "ldconfig -p".  ldconfig is part of the uclibc library and uclibc
has caching turned off in its configuration so ldconfig -p just prints a
message says, "Library cache disabled" instead of printing a list of libraries
and their absolute paths.

https://github.com/python/cpython/blob/main/Lib/ctypes/util.py

There is an option in the uclibc menuconfig to enable the cache
(LDSO_CACHE_SUPPORT) but when I did that it didn't seem to change anything. 
Any ideas how to fix this?

I'm using the pc_x86_64_efi_defconfig with Python enabled.  No other changes.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14886] Python ctypes find_library() not working correctly.

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14886

--- Comment #1 from Peter Scheie <pscheie@gmail.com> ---
I have seen this bug, too, in building an image for the Beaglebone Black.  It
appears in the iio package in the python binding: In iio.py, CDLL calls
find_library with the argument "iio" to get the name of the libiio.so.0
library, but fails.  Some folks on #beagle said using find_library() this way
was a bad idea because it produces unpredictable results (which I can attest
to).  Instead, it was recommended that CDLL be called with libiio.so.0
directly, removing find_libary().  This works and I am using a patch to
implement it.  BR currently uses version 0.19 of libiio but even in the most
current version, 0.24, it still uses find_library().

I tried find_library() on a few other libs and got the same None result.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14886] Python ctypes find_library() not working correctly.

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14886

--- Comment #2 from Pete Morici <pmorici@dev295.com> ---
The problem is a bunch of dependencies rely on the find_library function so I
don't really have a choice short of patching every library.

I think the right way to fix this would be to enable the ldconfig cache
mechanism in uclibc or mark libraries that depend on ctypes as dependent on
glibc tthat does support the caching mechanism required for find_library

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14886] Python ctypes find_library() not working correctly.

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14886

--- Comment #3 from Pete Morici <pmorici@dev295.com> ---
    I was able to solve this by enabling the LDSO_CACHE_SUPPORT option in
uclibc via a configuration fragment and then patching the file
Lib/ctypes/util.py in the python3 package to change the mapping of "x86_64-64"
from "libc6,x86_64" to "libc0".  I wasn't able to find a way to generate the
cache file during the build process so you have to run ldconfig on first boot,
you could use and init script, before using Python to generate /etc/ls.so.cache
and then ctypes will work correctly.

    The crux of the issue is that, on Linux, Python's ctypes module just isn't
compatible with anything except glibc because is explicitly looks for "libc6"
and other glibc specific items in the output of ldconfig -p which is uses to
resolve paths to libraries loaded with ctypes.  If you are using uclibc, of
musl you are going to run into this.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14886] Python ctypes find_library() not working correctly.

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14886

--- Comment #4 from Thomas Petazzoni <thomas.petazzoni@bootlin.com> ---
This issue is indeed tricky. I think the right solution would be to patch
Python to reimplement the find_library() function in a way that is compatible
with Buildroot:

 (1) Walk the library directories to find the matching library.

 (2) Retrieve its SONAME. This might be a bit annoying to do in pure Python
without invoking a separate Python library to read ELF files.

A bit annoying to do, but probably the solution that is going to require the
least amount of dependencies in Buildroot.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14886] Python ctypes find_library() not working correctly.

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14886

--- Comment #5 from Pete Morici <pmorici@dev295.com> ---
Created attachment 9341
  --> https://bugs.busybox.net/attachment.cgi?id=9341&action=edit
Python patch to support ctypes when using uclibc

Also set LDSO_CACHE_SUPPORT=y un uclibc configuration.  Then run ldconfig at
boot prior to using Python / ctypes if ld.so.cache doesn't exist.

if [[ ! -e "/etc/ld.so.cache" ]]; then ldconfig; fi

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14886] Python ctypes find_library() not working correctly.

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14886

--- Comment #6 from Pete Morici <pmorici@dev295.com> ---
My understanding from reading some of the other posts is that the reason the
ld.so.cache is disabled is because you can't reliably run it in the fakeroot
environment without a bunch of patches.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14886] Python ctypes find_library() not working correctly.

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14886

--- Comment #7 from Marcus Hoffmann <marcus.hoffmann@othermo.de> ---
Created attachment 9566
  --> https://bugs.busybox.net/attachment.cgi?id=9566&action=edit
libiio patch to fix python bindings

I've just run into this problem with libiio python bindings as well. (On a
glibc system though), the attached (libiio) patch fixes it, though I'm not sure
if that's a desired solution.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [Bug 14886] Python ctypes find_library() not working correctly.

[bugzilla]

https://bugs.busybox.net/show_bug.cgi?id=14886

Yann E. MORIN <yann.morin.1998@free.fr> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Resolution|---                         |MOVED
                 CC|                            |yann.morin.1998@free.fr
             Status|NEW                         |RESOLVED

--- Comment #8 from Yann E. MORIN <yann.morin.1998@free.fr> ---
Thank you for your report.

The issue tracker for the Buildroot project has been moved to
the Gitlab.com issue tracker:
    https://gitlab.com/buildroot.org/buildroot/-/issues

We are taking this opportunity to close old issues in this old
tracker. If you believe your issue is still relevant, please
open one in the new issue tracker.

Thank you!

-- 
You are receiving this mail because:
You are on the CC list for the bug.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot