[Angstrom-devel] Python library hashlib is missig hash-code.

[Angstrom-devel] Python library hashlib is missig hash-code.

[Koen Kooi]

My theory is that this is caused by the openssl /usr changes, can anyone confirm or deny that?

Begin doorgestuurd bericht:

> Van: Mattias Millbro <mmillbro@kth.se>
> Onderwerp: [Angstrom-devel] Python library hashlib is missig hash-code.
> Datum: 15 april 2012 14:38:56 GMT+02:00
> Aan: angstrom-distro-devel@linuxtogo.org
> Antwoord aan: Discussion of the angstrom distribution development <angstrom-distro-devel@linuxtogo.org>
> 
> Hi everybody,
> 
> as a new member of this mailing list I'm not familiar to the proper procedure of this kids of questions/bug reports. But I'll give it a try and please let me know if any necessary information is missing.
> 
> I'm running Angstrom on a Beaglebone using the Angstrom-systemd-image-eglibc-ipk-v2012.01-core-beaglebone.rootfs.tar.bz2 image from http://www.angstrom-distribution.org/demo/beaglebone/. I installed python and pyuthon-modules using opkg.
> 
> I recently made a completely fresh install and discovered that the python library hashlib does no longer work properly. It does not seem to find code for sha hashes even tho python.org states
> 
> "Included are the FIPS secure hash algorithms SHA1, SHA224, SHA256, SHA384, and SHA512 (defined in FIPS 180-2) as well as RSA’s MD5 algorithm (defined in Internet RFC 1321).".
> 
> All related packages seems to be up to date and I have spent several days googling and installing/reinstalling various packages without any success.
> 
> This is what happens when importing hashlib:
> 
> root@beaglebone:~# python
> Python 2.7.2 (default, Apr 13 2012, 21:18:55)
> [GCC 4.5.4 20120305 (prerelease)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import hashlib
> ERROR:root:code for hash sha1 was not found.
> Traceback (most recent call last):
> File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
> globals()[__func_name] = __get_hash(__func_name)
> File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
> raise ValueError('unsupported hash type %s' % name)
> ValueError: unsupported hash type sha1
> ERROR:root:code for hash sha224 was not found.
> Traceback (most recent call last):
> File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
> globals()[__func_name] = __get_hash(__func_name)
> File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
> raise ValueError('unsupported hash type %s' % name)
> ValueError: unsupported hash type sha224
> ERROR:root:code for hash sha256 was not found.
> Traceback (most recent call last):
> File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
> globals()[__func_name] = __get_hash(__func_name)
> File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
> raise ValueError('unsupported hash type %s' % name)
> ValueError: unsupported hash type sha256
> ERROR:root:code for hash sha384 was not found.
> Traceback (most recent call last):
> File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
> globals()[__func_name] = __get_hash(__func_name)
> File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
> raise ValueError('unsupported hash type %s' % name)
> ValueError: unsupported hash type sha384
> ERROR:root:code for hash sha512 was not found.
> Traceback (most recent call last):
> File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
> globals()[__func_name] = __get_hash(__func_name)
> File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
> raise ValueError('unsupported hash type %s' % name)
> ValueError: unsupported hash type sha512
> >>>
> 
> Here is some additional information that might be useful:
> 
> root@beaglebone:~# opkg list-installed | grep python
> libpython2.7-1.0 - 2.7.2-r2.11
> python-2to3 - 2.7.2-r2.11
> python-audio - 2.7.2-r2.11
> python-bsddb - 2.7.2-r2.11
> python-codecs - 2.7.2-r2.11
> python-compile - 2.7.2-r2.11
> python-compiler - 2.7.2-r2.11
> python-compression - 2.7.2-r2.11
> python-core - 2.7.2-r2.11
> python-crypt - 2.7.2-r2.11
> python-ctypes - 2.7.2-r2.11
> python-curses - 2.7.2-r2.11
> python-datetime - 2.7.2-r2.11
> python-db - 2.7.2-r2.11
> python-debugger - 2.7.2-r2.11
> python-difflib - 2.7.2-r2.11
> python-distutils - 2.7.2-r2.11
> python-doctest - 2.7.2-r2.11
> python-elementtree - 2.7.2-r2.11
> python-email - 2.7.2-r2.11
> python-fcntl - 2.7.2-r2.11
> python-gdbm - 2.7.2-r2.11
> python-hotshot - 2.7.2-r2.11
> python-html - 2.7.2-r2.11
> python-idle - 2.7.2-r2.11
> python-image - 2.7.2-r2.11
> python-io - 2.7.2-r2.11
> python-json - 2.7.2-r2.11
> python-lang - 2.7.2-r2.11
> python-logging - 2.7.2-r2.11
> python-mailbox - 2.7.2-r2.11
> python-math - 2.7.2-r2.11
> python-mime - 2.7.2-r2.11
> python-mmap - 2.7.2-r2.11
> python-modules - 2.7.2-r2.11
> python-multiprocessing - 2.7.2-r2.11
> python-netclient - 2.7.2-r2.11
> python-netserver - 2.7.2-r2.11
> python-numbers - 2.7.2-r2.11
> python-pickle - 2.7.2-r2.11
> python-pkgutil - 2.7.2-r2.11
> python-pprint - 2.7.2-r2.11
> python-profile - 2.7.2-r2.11
> python-pydoc - 2.7.2-r2.11
> python-re - 2.7.2-r2.11
> python-readline - 2.7.2-r2.11
> python-resource - 2.7.2-r2.11
> python-robotparser - 2.7.2-r2.11
> python-shell - 2.7.2-r2.11
> python-smtpd - 2.7.2-r2.11
> python-sqlite3 - 2.7.2-r2.11
> python-sqlite3-tests - 2.7.2-r2.11
> python-stringold - 2.7.2-r2.11
> python-subprocess - 2.7.2-r2.11
> python-syslog - 2.7.2-r2.11
> python-terminal - 2.7.2-r2.11
> python-tests - 2.7.2-r2.11
> python-textutils - 2.7.2-r2.11
> python-threading - 2.7.2-r2.11
> python-tkinter - 2.7.2-r2.11
> python-unittest - 2.7.2-r2.11
> python-unixadmin - 2.7.2-r2.11
> python-xml - 2.7.2-r2.11
> python-xmlrpc - 2.7.2-r2.11
> python-zlib - 2.7.2-r2.11
> 
> root@beaglebone:~# opkg list-installed | grep openssl
> openssl - 1.0.0h-r15.1
> 
> I'm grateful for any kind of help solving this issue.
> 
> Best Regards
> / Mattias
> 
> _______________________________________________
> Angstrom-distro-devel mailing list
> Angstrom-distro-devel@linuxtogo.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/angstrom-distro-devel

Re: [Angstrom-devel] Python library hashlib is missig hash-code.

[Koen Kooi]

Op 15 apr. 2012, om 16:35 heeft Koen Kooi het volgende geschreven:

> My theory is that this is caused by the openssl /usr changes, can anyone confirm or deny that?

I dug into this. The breakage was recorded by buildhistory:

==========================================================================
commit fef22c30d3834abdfe8b5c12cd2dd17e3e031973
Author: Koen Kooi <koen@dominion.thruhere.net>
Date:   Fri Feb 24 12:39:55 2012 +0100

    Build Angstrom v2012.02-core of angstrom v2012.02-core for machine beagleboard on dominion

diff --git a/packages/armv7a-angstrom-linux-gnueabi/python/python-crypt/latest b/packages/armv7a-angstrom-linux-gnueabi/python/python-crypt/latest
index 3ec1030..829e7ce 100644
--- a/packages/armv7a-angstrom-linux-gnueabi/python/python-crypt/latest
+++ b/packages/armv7a-angstrom-linux-gnueabi/python/python-crypt/latest
@@ -1,7 +1,7 @@
 PV = 2.7.2
 PR = r0.6
-RDEPENDS = eglibc (>= 2.12) libcrypto (>= 1.0.0g) libpython2 (>= 2.7.2) python-core
+RDEPENDS = eglibc (>= 2.12) libpython2 (>= 2.7.2) python-core
 RRECOMMENDS = openssl
-PKGSIZE = 25436
+PKGSIZE = 14252
 FILES = /usr/lib/python2.7/hashlib.* /usr/lib/python2.7/md5.* /usr/lib/python2.7/sha.* /usr/lib/python2.7/lib-dynload/crypt.so /usr/lib/python2.7/lib-dynload/_hashlib.so /usr/lib/python2.7/lib-dynload/_sha256.so /usr/lib/python2.7/lib-dynload/_sha512.so
-FILELIST = /usr/lib/python2.7/hashlib.py /usr/lib/python2.7/hashlib.pyo /usr/lib/python2.7/lib-dynload/_hashlib.so /usr/lib/python2.7/lib-dynload/crypt.so /usr/lib/python2.7/md5.py /usr/lib/python2.7/md5.pyo /usr/lib/python2.7/sha.py /usr/lib/python2.7/sha.pyo
+FILELIST = /usr/lib/python2.7/hashlib.py /usr/lib/python2.7/hashlib.pyo /usr/lib/python2.7/lib-dynload/crypt.so /usr/lib/python2.7/md5.py /usr/lib/python2.7/md5.pyo /usr/lib/python2.7/sha.py /usr/lib/python2.7/sha.pyo
==========================================================================


The preceding commit affecting python crypt:

==========================================================================
commit f572f6547bc4b1e2a64e4cae8201b70c0e1b4162
Author: Koen Kooi <koen@dominion.thruhere.net>
Date:   Wed Feb 22 14:53:00 2012 +0100

    Build Angstrom v2012.02-core of angstrom v2012.02-core for machine beaglebone on dominion

diff --git a/packages/armv7a-angstrom-linux-gnueabi/python/python-crypt/latest b/packages/armv7a-angstrom-linux-gnueabi/python/python-crypt/latest
index 5f21c50..3ec1030 100644
--- a/packages/armv7a-angstrom-linux-gnueabi/python/python-crypt/latest
+++ b/packages/armv7a-angstrom-linux-gnueabi/python/python-crypt/latest
@@ -3,5 +3,5 @@ PR = r0.6
 RDEPENDS = eglibc (>= 2.12) libcrypto (>= 1.0.0g) libpython2 (>= 2.7.2) python-core
 RRECOMMENDS = openssl
 PKGSIZE = 25436
-FILES = /usr/lib/python2.7/hashlib.* /usr/lib/python2.7/md5.* /usr/lib/python2.7/sha.* /usr/lib/python2.7/lib-dynload/crypt.so /usr/lib/python2.7/lib-dynload/_hashlib.so /usr/lib/python2.7/lib-dynload/_sha256.so /usr/lib/python2.7/lib-dynload/_sha512.so 
-FILELIST = /usr/lib/python2.7/md5.py /usr/lib/python2.7/md5.pyo /usr/lib/python2.7/sha.pyo /usr/lib/python2.7/hashlib.pyo /usr/lib/python2.7/sha.py /usr/lib/python2.7/hashlib.py /usr/lib/python2.7/lib-dynload/_hashlib.so /usr/lib/python2.7/lib-dynload/crypt.so
+FILES = /usr/lib/python2.7/hashlib.* /usr/lib/python2.7/md5.* /usr/lib/python2.7/sha.* /usr/lib/python2.7/lib-dynload/crypt.so /usr/lib/python2.7/lib-dynload/_hashlib.so /usr/lib/python2.7/lib-dynload/_sha256.so /usr/lib/python2.7/lib-dynload/_sha512.so
+FILELIST = /usr/lib/python2.7/hashlib.py /usr/lib/python2.7/hashlib.pyo /usr/lib/python2.7/lib-dynload/_hashlib.so /usr/lib/python2.7/lib-dynload/crypt.so /usr/lib/python2.7/md5.py /usr/lib/python2.7/md5.pyo /usr/lib/python2.7/sha.py /usr/lib/python2.7/sha.pyo
==========================================================================

And look at that, an openssl patch in between those dates:

==========================================================================
From 01ea85f7f6c53c66c76d6f832518b28bf06ec072 Mon Sep 17 00:00:00 2001
From: Andrei Gherzan <andrei@gherzan.ro>
Date: Tue, 21 Feb 2012 23:03:55 +0000
Subject: openssl: Move libcrypto to base_libdir

This fix is for dhclient. It needs libcrypto at runtime and if
libcrypto is in libdir, it's path can be inaccessible on systems
where /usr is on nfs for example or dhclient is needed before
/usr is mounted.

Signed-off-by: Andrei Gherzan <andrei@gherzan.ro>
==========================================================================

I'm going to band-aid it so we can get a working python hashlib module in the 'denzil' branch, hopefully someone will come up with a proper fix soon.

regards,

Koen


> 
> Begin doorgestuurd bericht:
> 
>> Van: Mattias Millbro <mmillbro@kth.se>
>> Onderwerp: [Angstrom-devel] Python library hashlib is missig hash-code.
>> Datum: 15 april 2012 14:38:56 GMT+02:00
>> Aan: angstrom-distro-devel@linuxtogo.org
>> Antwoord aan: Discussion of the angstrom distribution development <angstrom-distro-devel@linuxtogo.org>
>> 
>> Hi everybody,
>> 
>> as a new member of this mailing list I'm not familiar to the proper procedure of this kids of questions/bug reports. But I'll give it a try and please let me know if any necessary information is missing.
>> 
>> I'm running Angstrom on a Beaglebone using the Angstrom-systemd-image-eglibc-ipk-v2012.01-core-beaglebone.rootfs.tar.bz2 image from http://www.angstrom-distribution.org/demo/beaglebone/. I installed python and pyuthon-modules using opkg.
>> 
>> I recently made a completely fresh install and discovered that the python library hashlib does no longer work properly. It does not seem to find code for sha hashes even tho python.org states
>> 
>> "Included are the FIPS secure hash algorithms SHA1, SHA224, SHA256, SHA384, and SHA512 (defined in FIPS 180-2) as well as RSA’s MD5 algorithm (defined in Internet RFC 1321).".
>> 
>> All related packages seems to be up to date and I have spent several days googling and installing/reinstalling various packages without any success.
>> 
>> This is what happens when importing hashlib:
>> 
>> root@beaglebone:~# python
>> Python 2.7.2 (default, Apr 13 2012, 21:18:55)
>> [GCC 4.5.4 20120305 (prerelease)] on linux2
>> Type "help", "copyright", "credits" or "license" for more information.
>>>>> import hashlib
>> ERROR:root:code for hash sha1 was not found.
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
>> globals()[__func_name] = __get_hash(__func_name)
>> File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
>> raise ValueError('unsupported hash type %s' % name)
>> ValueError: unsupported hash type sha1
>> ERROR:root:code for hash sha224 was not found.
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
>> globals()[__func_name] = __get_hash(__func_name)
>> File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
>> raise ValueError('unsupported hash type %s' % name)
>> ValueError: unsupported hash type sha224
>> ERROR:root:code for hash sha256 was not found.
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
>> globals()[__func_name] = __get_hash(__func_name)
>> File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
>> raise ValueError('unsupported hash type %s' % name)
>> ValueError: unsupported hash type sha256
>> ERROR:root:code for hash sha384 was not found.
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
>> globals()[__func_name] = __get_hash(__func_name)
>> File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
>> raise ValueError('unsupported hash type %s' % name)
>> ValueError: unsupported hash type sha384
>> ERROR:root:code for hash sha512 was not found.
>> Traceback (most recent call last):
>> File "/usr/lib/python2.7/hashlib.py", line 139, in <module>
>> globals()[__func_name] = __get_hash(__func_name)
>> File "/usr/lib/python2.7/hashlib.py", line 91, in __get_builtin_constructor
>> raise ValueError('unsupported hash type %s' % name)
>> ValueError: unsupported hash type sha512
>>>>> 
>> 
>> Here is some additional information that might be useful:
>> 
>> root@beaglebone:~# opkg list-installed | grep python
>> libpython2.7-1.0 - 2.7.2-r2.11
>> python-2to3 - 2.7.2-r2.11
>> python-audio - 2.7.2-r2.11
>> python-bsddb - 2.7.2-r2.11
>> python-codecs - 2.7.2-r2.11
>> python-compile - 2.7.2-r2.11
>> python-compiler - 2.7.2-r2.11
>> python-compression - 2.7.2-r2.11
>> python-core - 2.7.2-r2.11
>> python-crypt - 2.7.2-r2.11
>> python-ctypes - 2.7.2-r2.11
>> python-curses - 2.7.2-r2.11
>> python-datetime - 2.7.2-r2.11
>> python-db - 2.7.2-r2.11
>> python-debugger - 2.7.2-r2.11
>> python-difflib - 2.7.2-r2.11
>> python-distutils - 2.7.2-r2.11
>> python-doctest - 2.7.2-r2.11
>> python-elementtree - 2.7.2-r2.11
>> python-email - 2.7.2-r2.11
>> python-fcntl - 2.7.2-r2.11
>> python-gdbm - 2.7.2-r2.11
>> python-hotshot - 2.7.2-r2.11
>> python-html - 2.7.2-r2.11
>> python-idle - 2.7.2-r2.11
>> python-image - 2.7.2-r2.11
>> python-io - 2.7.2-r2.11
>> python-json - 2.7.2-r2.11
>> python-lang - 2.7.2-r2.11
>> python-logging - 2.7.2-r2.11
>> python-mailbox - 2.7.2-r2.11
>> python-math - 2.7.2-r2.11
>> python-mime - 2.7.2-r2.11
>> python-mmap - 2.7.2-r2.11
>> python-modules - 2.7.2-r2.11
>> python-multiprocessing - 2.7.2-r2.11
>> python-netclient - 2.7.2-r2.11
>> python-netserver - 2.7.2-r2.11
>> python-numbers - 2.7.2-r2.11
>> python-pickle - 2.7.2-r2.11
>> python-pkgutil - 2.7.2-r2.11
>> python-pprint - 2.7.2-r2.11
>> python-profile - 2.7.2-r2.11
>> python-pydoc - 2.7.2-r2.11
>> python-re - 2.7.2-r2.11
>> python-readline - 2.7.2-r2.11
>> python-resource - 2.7.2-r2.11
>> python-robotparser - 2.7.2-r2.11
>> python-shell - 2.7.2-r2.11
>> python-smtpd - 2.7.2-r2.11
>> python-sqlite3 - 2.7.2-r2.11
>> python-sqlite3-tests - 2.7.2-r2.11
>> python-stringold - 2.7.2-r2.11
>> python-subprocess - 2.7.2-r2.11
>> python-syslog - 2.7.2-r2.11
>> python-terminal - 2.7.2-r2.11
>> python-tests - 2.7.2-r2.11
>> python-textutils - 2.7.2-r2.11
>> python-threading - 2.7.2-r2.11
>> python-tkinter - 2.7.2-r2.11
>> python-unittest - 2.7.2-r2.11
>> python-unixadmin - 2.7.2-r2.11
>> python-xml - 2.7.2-r2.11
>> python-xmlrpc - 2.7.2-r2.11
>> python-zlib - 2.7.2-r2.11
>> 
>> root@beaglebone:~# opkg list-installed | grep openssl
>> openssl - 1.0.0h-r15.1
>> 
>> I'm grateful for any kind of help solving this issue.
>> 
>> Best Regards
>> / Mattias
>> 
>> _______________________________________________
>> Angstrom-distro-devel mailing list
>> Angstrom-distro-devel@linuxtogo.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/angstrom-distro-devel
> 

Re: [Angstrom-devel] Python library hashlib is missig hash-code.

[Koen Kooi]

Op 23 apr. 2012, om 10:15 heeft Koen Kooi het volgende geschreven:

> Op 15 apr. 2012, om 16:35 heeft Koen Kooi het volgende geschreven:
> 
>> My theory is that this is caused by the openssl /usr changes, can anyone confirm or deny that?
> 
> I dug into this. The breakage was recorded by buildhistory:

[..]

> I'm going to band-aid it so we can get a working python hashlib module in the 'denzil' branch, hopefully someone will come up with a proper fix soon.

After spending some more time on this and getting a ton of help from RP:

Python can't link to anything in ${base_libdir} due to a combination of python having a stupid buildsystem and the patches we apply to make it less stupid. The only fix I found to get hashlib and ssl to work in python is to revert the patch to openssl that moves libcrypto around.

I really need python unbroken in the release branch, so I see a few scenarios:

1) revert openssl patch, seperate /usr doesn't work properly anyway in this release
2) fix python
3) revert openssl patch, statically link libcrypto into dhclient.
4) bbappend openssl in meta-angstrom to revert the openssl patch

There are probably more scenarios, but I'm going to have lunch before thinking some more about it. My question to all of you:

	How do you want to fix this for denzil and how do you want to fix this for master?

And to be clear: I'm mean getting things into the denzil branch of OE-core, not getting it into the release, it's too late for that.

regards,

koen

Re: [Angstrom-devel] Python library hashlib is missig hash-code.

[Richard Purdie]

On Mon, 2012-04-23 at 15:08 +0200, Koen Kooi wrote:
> Op 23 apr. 2012, om 10:15 heeft Koen Kooi het volgende geschreven:
> 
> > Op 15 apr. 2012, om 16:35 heeft Koen Kooi het volgende geschreven:
> > 
> >> My theory is that this is caused by the openssl /usr changes, can
> anyone confirm or deny that?
> > 
> > I dug into this. The breakage was recorded by buildhistory:
> 
> [..]
> 
> > I'm going to band-aid it so we can get a working python hashlib
> module in the 'denzil' branch, hopefully someone will come up with a
> proper fix soon.
> 
> After spending some more time on this and getting a ton of help from
> RP:
> 
> Python can't link to anything in ${base_libdir} due to a combination
> of python having a stupid buildsystem and the patches we apply to make
> it less stupid. The only fix I found to get hashlib and ssl to work in
> python is to revert the patch to openssl that moves libcrypto around.
> 
> I really need python unbroken in the release branch, so I see a few
> scenarios:
> 
> 1) revert openssl patch, seperate /usr doesn't work properly anyway in
> this release
> 2) fix python
> 3) revert openssl patch, statically link libcrypto into dhclient.
> 4) bbappend openssl in meta-angstrom to revert the openssl patch
> 
> There are probably more scenarios, but I'm going to have lunch before
> thinking some more about it. My question to all of you:
> 
> 	How do you want to fix this for denzil and how do you want to fix
> this for master?
> 
> And to be clear: I'm mean getting things into the denzil branch of
> OE-core, not getting it into the release, it's too late for that.

I've managed to squeeze a fix for this in, I posted the patches last
night. It wasn't just the ssl piece that was broken, there were a number
of problems in the python recipe causing various modules to fail to
build (such as curses and readline). Some of these issues had been
present for much longer than the openssl change.

Its worth noting the openssl change was not the problem, it was python
passing native paths to a cross-gcc which then understandably got
confused.

At this point the release is pretty much done, assuming a favourable QA
report.

Cheers,

Richard