Missing certificates

Missing certificates

[Gary Thomas]

I was trying to run a simple fetch from python using
         url = 'https://raw.github.com/Itseez/opencv/master/samples/c/fruits.jpg'
         filedata = urllib2.urlopen(url).read()

This failed:
   Traceback (most recent call last):
   File "./edge.py", line 36, in <module>
     filedata = urllib2.urlopen(url).read()
   File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
     return opener.open(url, data, timeout)
   File "/usr/lib/python2.7/urllib2.py", line 431, in open
     response = self._open(req, data)
   File "/usr/lib/python2.7/urllib2.py", line 449, in _open
     '_open', req)
   File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
     result = func(*args)
   File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
     context=self._context)
   File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
     raise URLError(err)
urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>

I can see that it was looking for some certificates in /usr/lib/ssl/certs
but that directory is missing.

Anyone know what I might be missing (or have misconfigured)?

Thanks

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Missing certificates

[Aníbal Limón]

Hi Gary,

What version of python do you use?.

Since 2.7.9 cert checking is enabled by default causing this kind of 
errors. [1]

[1] https://www.python.org/dev/peps/pep-0476/

Kind regards,
     alimon

On 24/07/15 13:02, Gary Thomas wrote:
> I was trying to run a simple fetch from python using
>         url = 
> 'https://raw.github.com/Itseez/opencv/master/samples/c/fruits.jpg'
>         filedata = urllib2.urlopen(url).read()
>
> This failed:
>   Traceback (most recent call last):
>   File "./edge.py", line 36, in <module>
>     filedata = urllib2.urlopen(url).read()
>   File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
>     return opener.open(url, data, timeout)
>   File "/usr/lib/python2.7/urllib2.py", line 431, in open
>     response = self._open(req, data)
>   File "/usr/lib/python2.7/urllib2.py", line 449, in _open
>     '_open', req)
>   File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
>     result = func(*args)
>   File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
>     context=self._context)
>   File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
>     raise URLError(err)
> urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] 
> certificate verify failed (_ssl.c:581)>
>
> I can see that it was looking for some certificates in /usr/lib/ssl/certs
> but that directory is missing.
>
> Anyone know what I might be missing (or have misconfigured)?
>
> Thanks
>

Re: Missing certificates

[Gary Thomas]

On 2015-07-24 13:30, Aníbal Limón wrote:
> Hi Gary,
>
> What version of python do you use?.
>
> Since 2.7.9 cert checking is enabled by default causing this kind of errors. [1]
>
> [1] https://www.python.org/dev/peps/pep-0476/
>
> Kind regards,
>      alimon

I'm using the stock python 2.7.9 from Poky/Yocto master:901be2cb69892595443ed41ab4be285932db15eb

Is there an answer for this that's a bit less intrusive?
Perhaps there could be a DISTRO or even IMAGE feature to
enable/disable this checking?

The pep you referenced mostly talks about why this was changed
and how to disable it - manually within the python code itself.
What I don't see is where/how/what to change/import to actually
let the full certificate checking happen.

>
> On 24/07/15 13:02, Gary Thomas wrote:
>> I was trying to run a simple fetch from python using
>>         url = 'https://raw.github.com/Itseez/opencv/master/samples/c/fruits.jpg'
>>         filedata = urllib2.urlopen(url).read()
>>
>> This failed:
>>   Traceback (most recent call last):
>>   File "./edge.py", line 36, in <module>
>>     filedata = urllib2.urlopen(url).read()
>>   File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
>>     return opener.open(url, data, timeout)
>>   File "/usr/lib/python2.7/urllib2.py", line 431, in open
>>     response = self._open(req, data)
>>   File "/usr/lib/python2.7/urllib2.py", line 449, in _open
>>     '_open', req)
>>   File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
>>     result = func(*args)
>>   File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
>>     context=self._context)
>>   File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
>>     raise URLError(err)
>> urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
>>
>> I can see that it was looking for some certificates in /usr/lib/ssl/certs
>> but that directory is missing.
>>
>> Anyone know what I might be missing (or have misconfigured)?
>>
>> Thanks
>>

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Missing certificates

[Aníbal Limón]

On 24/07/15 14:49, Gary Thomas wrote:
> On 2015-07-24 13:30, Aníbal Limón wrote:
>> Hi Gary,
>>
>> What version of python do you use?.
>>
>> Since 2.7.9 cert checking is enabled by default causing this kind of 
>> errors. [1]
>>
>> [1] https://www.python.org/dev/peps/pep-0476/
>>
>> Kind regards,
>>      alimon
>
> I'm using the stock python 2.7.9 from Poky/Yocto 
> master:901be2cb69892595443ed41ab4be285932db15eb
>
> Is there an answer for this that's a bit less intrusive?
> Perhaps there could be a DISTRO or even IMAGE feature to
> enable/disable this checking?

I don't think that Python guys include a configuration flags to disable 
this behavior because it's
the default now due to security issues.

>
> The pep you referenced mostly talks about why this was changed
> and how to disable it - manually within the python code itself.
> What I don't see is where/how/what to change/import to actually
> let the full certificate checking happen.

You can use this code for disable per urlopen call or globally [1].

Regards,
     alimon

[1] https://www.python.org/dev/peps/pep-0476/#opting-out

>
>>
>> On 24/07/15 13:02, Gary Thomas wrote:
>>> I was trying to run a simple fetch from python using
>>>         url = 
>>> 'https://raw.github.com/Itseez/opencv/master/samples/c/fruits.jpg'
>>>         filedata = urllib2.urlopen(url).read()
>>>
>>> This failed:
>>>   Traceback (most recent call last):
>>>   File "./edge.py", line 36, in <module>
>>>     filedata = urllib2.urlopen(url).read()
>>>   File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
>>>     return opener.open(url, data, timeout)
>>>   File "/usr/lib/python2.7/urllib2.py", line 431, in open
>>>     response = self._open(req, data)
>>>   File "/usr/lib/python2.7/urllib2.py", line 449, in _open
>>>     '_open', req)
>>>   File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
>>>     result = func(*args)
>>>   File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
>>>     context=self._context)
>>>   File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
>>>     raise URLError(err)
>>> urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] 
>>> certificate verify failed (_ssl.c:581)>
>>>
>>> I can see that it was looking for some certificates in 
>>> /usr/lib/ssl/certs
>>> but that directory is missing.
>>>
>>> Anyone know what I might be missing (or have misconfigured)?
>>>
>>> Thanks
>>>
>

Re: Missing certificates

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1249 bytes --]

On Fri, Jul 24, 2015 at 12:49 PM, Gary Thomas <gary@mlbassoc.com> wrote:

> On 2015-07-24 13:30, Aníbal Limón wrote:
>
>> Hi Gary,
>>
>> What version of python do you use?.
>>
>> Since 2.7.9 cert checking is enabled by default causing this kind of
>> errors. [1]
>>
>> [1] https://www.python.org/dev/peps/pep-0476/
>>
>> Kind regards,
>>      alimon
>>
>
> I'm using the stock python 2.7.9 from Poky/Yocto
> master:901be2cb69892595443ed41ab4be285932db15eb
>
> Is there an answer for this that's a bit less intrusive?
> Perhaps there could be a DISTRO or even IMAGE feature to
> enable/disable this checking?
>
> The pep you referenced mostly talks about why this was changed
> and how to disable it - manually within the python code itself.
> What I don't see is where/how/what to change/import to actually
> let the full certificate checking happen.


I think the better bet is to fix it so it actually finds the certs from
ca-certificates rather than bypassing certificate checking, personally, but
I can see how that would be a useful workaround. :)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1870 bytes --]

Re: Missing certificates

[Gary Thomas]

On 2015-07-24 14:09, Christopher Larson wrote:
>
> On Fri, Jul 24, 2015 at 12:49 PM, Gary Thomas <gary@mlbassoc.com <mailto:gary@mlbassoc.com>> wrote:
>
>     On 2015-07-24 13:30, Aníbal Limón wrote:
>
>         Hi Gary,
>
>         What version of python do you use?.
>
>         Since 2.7.9 cert checking is enabled by default causing this kind of errors. [1]
>
>         [1] https://www.python.org/dev/peps/pep-0476/
>
>         Kind regards,
>               alimon
>
>
>     I'm using the stock python 2.7.9 from Poky/Yocto master:901be2cb69892595443ed41ab4be285932db15eb
>
>     Is there an answer for this that's a bit less intrusive?
>     Perhaps there could be a DISTRO or even IMAGE feature to
>     enable/disable this checking?
>
>     The pep you referenced mostly talks about why this was changed
>     and how to disable it - manually within the python code itself.
>     What I don't see is where/how/what to change/import to actually
>     let the full certificate checking happen.
>
>
> I think the better bet is to fix it so it actually finds the certs from ca-certificates rather than bypassing certificate checking, personally, but I can see how that would be a
> useful workaround. :)

I tried this same code on my Ubuntu 15.04 desktop and it looks
like they've disabled it in the main python http[s] code - there
were no certificates examined during the transaction as far as
I could tell (strace is my friend)

I'll see if I can figure out how to stitch this together with our
[Poky/Yocto/OE-core] setup for OpenSSL and ca-certificates.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: Missing certificates

[Christopher Larson]

[-- Attachment #1: Type: text/plain, Size: 1704 bytes --]

On Fri, Jul 24, 2015 at 11:02 AM, Gary Thomas <gary@mlbassoc.com> wrote:

> I was trying to run a simple fetch from python using
>         url = '
> https://raw.github.com/Itseez/opencv/master/samples/c/fruits.jpg'
>         filedata = urllib2.urlopen(url).read()
>
> This failed:
>   Traceback (most recent call last):
>   File "./edge.py", line 36, in <module>
>     filedata = urllib2.urlopen(url).read()
>   File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
>     return opener.open(url, data, timeout)
>   File "/usr/lib/python2.7/urllib2.py", line 431, in open
>     response = self._open(req, data)
>   File "/usr/lib/python2.7/urllib2.py", line 449, in _open
>     '_open', req)
>   File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
>     result = func(*args)
>   File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
>     context=self._context)
>   File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
>     raise URLError(err)
> urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED]
> certificate verify failed (_ssl.c:581)>
>
> I can see that it was looking for some certificates in /usr/lib/ssl/certs
> but that directory is missing.
>
> Anyone know what I might be missing (or have misconfigured)?
>

At least in the past, we’ve been highly inconsistent in certificate store
configuration in various recipes, some pointing at a dir, some a file, and
the paths vary. I don’t know if anyone ever fixed that, however — this is
from memory.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2374 bytes --]

Re: Missing certificates

[Gary Thomas]

On 2015-07-24 12:02, Gary Thomas wrote:
> I was trying to run a simple fetch from python using
>          url = 'https://raw.github.com/Itseez/opencv/master/samples/c/fruits.jpg'
>          filedata = urllib2.urlopen(url).read()
>
> This failed:
>    Traceback (most recent call last):
>    File "./edge.py", line 36, in <module>
>      filedata = urllib2.urlopen(url).read()
>    File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
>      return opener.open(url, data, timeout)
>    File "/usr/lib/python2.7/urllib2.py", line 431, in open
>      response = self._open(req, data)
>    File "/usr/lib/python2.7/urllib2.py", line 449, in _open
>      '_open', req)
>    File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
>      result = func(*args)
>    File "/usr/lib/python2.7/urllib2.py", line 1240, in https_open
>      context=self._context)
>    File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
>      raise URLError(err)
> urllib2.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:581)>
>
> I can see that it was looking for some certificates in /usr/lib/ssl/certs
> but that directory is missing.
>
> Anyone know what I might be missing (or have misconfigured)?
>
> Thanks
>

I've found a discussion about this problem on the OpenEmbedded
development list:
   http://lists.openembedded.org/pipermail/openembedded-devel/2015-July/102160.html

So the problem that this has uncovered is twofold:
1) Python (and OpenSSL) are not using the certificates that are installed by the ca-certificates package
    OpenSSL expects the certificates in /usr/lib/ssl/certs and ca-certificates uses /etc/ssl/certs
2) The certificates from ca-certificates are not immediately usable by OpenSSL since they are not
    hashed.  This is done by the 'c_rehash' program but has been explicitly disabled by a patch.
    Further exploration implies that this was disabled because not all targets will have c_rehash
    available and since the hashing is expected to be done on the target when the certificates
    are loaded/updated.  Finally, c_rehash, may or may not exist in the OpenSSL packages, depending
    on whether or not perl is available on the target (it's a perl script)

How best to solve this?  As is, python htts:// support is broken in OE-core, so I think an
off-the-shelf solution is warranted.

Perhaps the PACKAGECONFIG for openssl should default to supporting perl on the target, and hence
the c_rehash utility would be available?  Certainly the choice of where the certificates live, etc,
should be standardized.

Maybe the c_rehash can be run at package build time for ca-certificates?  This would make things work,
at least for the real CA certificates.

Ideas?

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

missing certificates

[Edward Vidal]

[-- Attachment #1: Type: text/plain, Size: 924 bytes --]

Hello All,In July 2015, several messages were posted with regard to missing certificatesfrom Gary Thomas, Christopher Larson, and Aníbal Limón.I am working with a Raspberry Pi 2 B and having a problem with fetching a bitbucket repository using HG (Mercurial)  which is certificate issue.  Is this a python or O/S issue.  I have been checking the site mentioned by 
Aníbal Limón https://www.python.org/dev/peps/pep-0476/
I noticed that very bottom there is a link https://hg.python.org/peps/file/tip/pep-0476.txt date Wed, 02 Sep 2015 15:00:42 -0400 (20 hours ago).After reading this several times I am still not sure how to proceed.This problem occurs when I am trying to install "yosys"  which during the build process tries  "hg clone https://bitbucket.org/alanmi/abc abc"
How best to get this working on my core-image-sato.Thanks in advance.

 Edward Vidal Jr. e-mail develone@sbcglobal.net 915-595-1613

[-- Attachment #2: Type: text/html, Size: 2337 bytes --]