Re: [OE-core] [PATCH] python3-cryptography: workaround broken native functionality

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Tue, 2022-09-13 at 13:29 +0300, Mikko Rapeli wrote:
> Hi,
> 
> On Tue, 13 Sept 2022 at 13:01, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > 
> > On Tue, 2022-09-13 at 12:34 +0300, Mikko Rapeli wrote:
> > > The python3-cryptography-native builds work but are functionally broken
> > > on Ubuntu 18.04 build host since the update from 3.3.2 in
> > > meta-openembedded/meta-python. If recipe needs and DEPENDS on
> > > python3-cryptography-native for signing use cases, loading
> > > the python modules fails:
> > > 
> > > $ python3 -c  "from OpenSSL import crypto"
> > > Traceback (most recent call last):
> > >   File "<string>", line 1, in <module>
> > >   File "/home/builder/poky/build_kirkstone/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/OpenSSL/__init__.py", line 8, in <module>
> > >     from OpenSSL import crypto, SSL
> > >   File "/home/builder/poky/build_kirkstone/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/OpenSSL/crypto.py", line 11, in <module>
> > >     from OpenSSL._util import (
> > >   File "/home/builder/poky/build_kirkstone/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/OpenSSL/_util.py", line 5, in <module>
> > >     from cryptography.hazmat.bindings.openssl.binding import Binding
> > >   File "/home/builder/poky/build_kirkstone/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 228, in <module>
> > >     Binding.init_static_locks()
> > >   File "/home/builder/poky/build_kirkstone/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 188, in init_static_locks
> > >     cls._ensure_ffi_initialized()
> > >   File "/home/builder/poky/build_kirkstone/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 176, in _ensure_ffi_initialized
> > >     _openssl_assert(
> > >   File "/home/builder/poky/build_kirkstone/tmp/work/core2-64-poky-linux/busybox/1.35.0-r0/recipe-sysroot-native/usr/lib/python3.10/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 90, in _openssl_assert
> > >     raise InternalError(
> > > cryptography.exceptions.InternalError: Unknown OpenSSL error. This error is commonly encountered when another library is not cleaning up the OpenSSL error stack. If you are using cryptography with another library that uses OpenSSL try disabling it before reporting a bug. Otherwise please file an issue at https://github.com/pyca/cryptography/issues with information on how to reproduce this. ([_OpenSSLErrorWithText(code=310378599, lib=37, reason=103, reason_text=b'error:12800067:DSO support routines::could not load the shared library'), _OpenSSLErrorWithText(code=310378599, lib=37, reason=103, reason_text=b'error:12800067:DSO support routines::could not load the shared library'), _OpenSSLErrorWithText(code=126615813, lib=15, reason=786693, reason_text=b'error:078C0105:common libcrypto routines::init fail')])
> > > 
> > > This hacky patch enables enough functionality in
> > > python3-cryptography-native to work so that basic secure boot
> > > signing use cases work again.
> > > 
> > > Signed-off-by: Mikko Rapeli <mikko.rapeli@linaro.org>
> > > ---
> > >  ...3-cryptography_hack_to_remove_legacy.patch | 54 +++++++++++++++++++
> > >  .../python/python3-cryptography_37.0.4.bb     |  5 ++
> > >  2 files changed, 59 insertions(+)
> > >  create mode 100644 meta/recipes-devtools/python/python3-cryptography/python3-cryptography_hack_to_remove_legacy.patch
> > 
> > I'm very nervous about taking a patch like this as it would be near
> > impossible to tell when we still need it or not and it has zero chance
> > of making it upstream.
> > 
> > Do we know how the openssl library is breaking internally? Is this some
> > kind of glibc or loader mismatch? Is it mixing up our sysroot ssl
> > library with the host one somehow?
> 
> I could not see what exactly was wrong.
> 
> python3 is taken correctly from recipe-sysroot-native path, same for
> all shared libraries like openssl, cffi etc.
> I went through strace output of the test case and could not see what
> exactly is wrong there. All binaries are openat()'ed from
> the native sysroot, part from libc, pthreads and a few others which
> AFAIK are normal.

Are you using uninative? I'd have expected glibc and pthreads to come
from there rather than the host.

>  The openssl.cnf file
> is not found in native sysroot, which is another small bug, but that
> did not seem to fix this (I just hacked it to work, some
> absolute build openssl-native env path leaks into the openssl-native binaries).
> 
> The old version 3.3.2 version of python3-cryptography from
> meta-openembedded/meta-python works without any problems.
> It's just the new versions 35, 36 and 37 which have this issue.
> 
> On my Ubuntu 18.04 machine, python3-cryptography-native 35 and later
> don't work at all without this workaround.
> Would be nice to know if others can reproduce this on other host distributions.
> 
> I was testing with busybox changes:
> 
> meta/recipes-core/busybox/busybox_1.35.0.bb
> @@ -54,3 +54,7 @@ SRC_URI =
> "https://busybox.net/downloads/busybox-${PV}.tar.bz2;name=tarball \
>  SRC_URI:append:libc-musl = " file://musl.cfg "
> 
>  SRC_URI[tarball.sha256sum] =
> "faeeb244c35a348a334f4a59e44626ee870fb07b6884d68c10ae8bc19f83a694"
> +
> +inherit python3native
> +
> +DEPENDS += "python3-pyopenssl-native"
> 
> And then in bitbake -c devshell busybox:
> 
> # python3 -c  "from OpenSSL import crypto"
> 
> I guess there is no way to add a test like that for python3-cryptography-native?

You could probably put that in do_configure to test it?

Cheers,

Richard