[Buildroot] [PATCH 1/1] python3: add patch to fix PEP 3147 issue with automake built packages

[Buildroot] [PATCH 1/1] python3: add patch to fix PEP 3147 issue with automake built packages

[Christophe Vu-Brugier]

Packages built with automake use a `py-compile` helper to byte-compile
Python source files. This script uses the "py_compile" module from the
standard library. In turn, the compile() function in the "py_compile"
module invokes the cache_from_source() function provided by importlib.

This patch changes cache_from_source() and source_from_cache() in
importlib to get rid of the "__pycache__" directory.

This fixes the following import error in kmod when the module is built
for Python 3:

  >>> from kmod import Kmod
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  ImportError: cannot import name 'Kmod'

Signed-off-by: Christophe Vu-Brugier <cvubrugier@fastmail.fm>
---
 package/python3/020-importlib-no-pep3147.patch | 72 ++++++++++++++++++++++++++
 1 file changed, 72 insertions(+)
 create mode 100644 package/python3/020-importlib-no-pep3147.patch

diff --git a/package/python3/020-importlib-no-pep3147.patch b/package/python3/020-importlib-no-pep3147.patch
new file mode 100644
index 0000000..3e797d3
--- /dev/null
+++ b/package/python3/020-importlib-no-pep3147.patch
@@ -0,0 +1,72 @@
+Add importlib fix for PEP 3147 issue
+
+Python 3 follows PEP 3147 and stores byte-compiled files in a per
+interpreter "__pycache__" directory. The drawback of PEP 3147 is that
+it requires the .py file to be installed. That's why Buildroot
+disables PEP 3147 to accommodate tiny embedded systems.
+
+Buildroot contains patches to disable PEP 3147 in setuptools and
+distutils but packages built with automake use a `py-compile` helper
+to byte-compile Python source files. This script uses the "py_compile"
+module from the standard library. In turn, the compile() function in
+the "py_compile" module invokes the cache_from_source() function
+provided by importlib.
+
+This patch changes cache_from_source() and source_from_cache() in
+importlib to get rid of the "__pycache__" directory.
+
+  http://www.gnu.org/software/automake/manual/html_node/Python.html
+
+Signed-off-by: Christophe Vu-Brugier <cvubrugier@fastmail.fm>
+
+diff -purN a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
+--- a/Lib/importlib/_bootstrap.py	2015-02-25 12:27:44.000000000 +0100
++++ b/Lib/importlib/_bootstrap.py	2015-08-27 17:33:26.167191059 +0200
+@@ -444,8 +444,6 @@ def cache_from_source(path, debug_overri
+     If debug_override is not None, then it must be a boolean and is used in
+     place of sys.flags.optimize.
+ 
+-    If sys.implementation.cache_tag is None then NotImplementedError is raised.
+-
+     """
+     debug = not sys.flags.optimize if debug_override is None else debug_override
+     if debug:
+@@ -454,33 +452,19 @@ def cache_from_source(path, debug_overri
+         suffixes = OPTIMIZED_BYTECODE_SUFFIXES
+     head, tail = _path_split(path)
+     base, sep, rest = tail.rpartition('.')
+-    tag = sys.implementation.cache_tag
+-    if tag is None:
+-        raise NotImplementedError('sys.implementation.cache_tag is None')
+-    filename = ''.join([(base if base else rest), sep, tag, suffixes[0]])
+-    return _path_join(head, _PYCACHE, filename)
++    filename = ''.join([(base if base else rest), suffixes[0]])
++    return _path_join(head, filename)
+ 
+ 
+ def source_from_cache(path):
+     """Given the path to a .pyc./.pyo file, return the path to its .py file.
+ 
+     The .pyc/.pyo file does not need to exist; this simply returns the path to
+-    the .py file calculated to correspond to the .pyc/.pyo file.  If path does
+-    not conform to PEP 3147 format, ValueError will be raised. If
+-    sys.implementation.cache_tag is None then NotImplementedError is raised.
++    the .py file calculated to correspond to the .pyc/.pyo file.
+ 
+     """
+-    if sys.implementation.cache_tag is None:
+-        raise NotImplementedError('sys.implementation.cache_tag is None')
+-    head, pycache_filename = _path_split(path)
+-    head, pycache = _path_split(head)
+-    if pycache != _PYCACHE:
+-        raise ValueError('{} not bottom-level directory in '
+-                         '{!r}'.format(_PYCACHE, path))
+-    if pycache_filename.count('.') != 2:
+-        raise ValueError('expected only 2 dots in '
+-                         '{!r}'.format(pycache_filename))
+-    base_filename = pycache_filename.partition('.')[0]
++    head, filename = _path_split(path)
++    base_filename = filename.partition('.')[0]
+     return _path_join(head, base_filename + SOURCE_SUFFIXES[0])
+ 
+ 
-- 
2.5.0

[Buildroot] [PATCH 1/1] python3: add patch to fix PEP 3147 issue with automake built packages

[Thomas Petazzoni]

Dear Christophe Vu-Brugier,

On Fri, 28 Aug 2015 16:58:41 +0200, Christophe Vu-Brugier wrote:
> Packages built with automake use a `py-compile` helper to byte-compile
> Python source files. This script uses the "py_compile" module from the
> standard library. In turn, the compile() function in the "py_compile"
> module invokes the cache_from_source() function provided by importlib.
> 
> This patch changes cache_from_source() and source_from_cache() in
> importlib to get rid of the "__pycache__" directory.
> 
> This fixes the following import error in kmod when the module is built
> for Python 3:
> 
>   >>> from kmod import Kmod
>   Traceback (most recent call last):
>     File "<stdin>", line 1, in <module>
>   ImportError: cannot import name 'Kmod'
> 
> Signed-off-by: Christophe Vu-Brugier <cvubrugier@fastmail.fm>

Thanks for this patch. However, I'm wondering if it doesn't make the
existing patch 016-distutils-no-pep3147.patch unnecessary. Patch 016
fixes the distutils/setuptools case by preventing the logic from
calling importlib.util.cache_from_source(). But since you're fixing
this function in your patch, maybe there's no need for the 016 patch
anymore ? What about patch 004 ?

Could you do some quick testing without patches 004 and 016, and see if
1/ the Python standard library is byte-compiled without __pycache__
folders and 2/ same for Python external modules built with distutils
and setuptools.

I'm pretty sure with your patch we can remove patch 016 (so your patch
could replace it). I'm not sure about patch 004, though.

Thanks a lot!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH 1/1] python3: add patch to fix PEP 3147 issue with automake built packages

[Christophe Vu-Brugier]

Hi Thomas,

On Fri, 28 Aug 2015 19:36:43 +0200, Thomas Petazzoni wrote :
> On Fri, 28 Aug 2015 16:58:41 +0200, Christophe Vu-Brugier wrote:
> > Packages built with automake use a `py-compile` helper to
> > byte-compile Python source files. This script uses the "py_compile"
> > module from the standard library. In turn, the compile() function
> > in the "py_compile" module invokes the cache_from_source() function
> > provided by importlib.
> > 
> > This patch changes cache_from_source() and source_from_cache() in
> > importlib to get rid of the "__pycache__" directory.
> > 
> > This fixes the following import error in kmod when the module is
> > built for Python 3:
> > 
> >   >>> from kmod import Kmod
> >   Traceback (most recent call last):
> >     File "<stdin>", line 1, in <module>
> >   ImportError: cannot import name 'Kmod'
> > 
> > Signed-off-by: Christophe Vu-Brugier <cvubrugier@fastmail.fm>
> 
> Thanks for this patch. However, I'm wondering if it doesn't make the
> existing patch 016-distutils-no-pep3147.patch unnecessary. Patch 016
> fixes the distutils/setuptools case by preventing the logic from
> calling importlib.util.cache_from_source(). But since you're fixing
> this function in your patch, maybe there's no need for the 016 patch
> anymore ? What about patch 004 ?
> 
> Could you do some quick testing without patches 004 and 016, and see
> if 1/ the Python standard library is byte-compiled without __pycache__
> folders and 2/ same for Python external modules built with distutils
> and setuptools.
> 
> I'm pretty sure with your patch we can remove patch 016 (so your patch
> could replace it). I'm not sure about patch 004, though.

Indeed, patch 016 for distutils / setuptools is no longer necessary.
Moreover, patch 004 for the standard library can be dropped as well.
Afer removing patches 004 and 016, I see no __pycache__ directory in the
Python path and the interpreter behaves as expected (I tested by loading
modules and running samples for readline, urwid, six, kmod or
pyparsing).

I will respin my patch to drop 004 and 016 and update 005 which is
impacted by the removal of 004.

Thank you !

-- 
Christophe Vu-Brugier

[Buildroot] [PATCH 1/1] python3: add patch to fix PEP 3147 issue with automake built packages

[Thomas Petazzoni]

Christophe,

On Sat, 29 Aug 2015 14:06:35 +0200, Christophe Vu-Brugier wrote:

> > I'm pretty sure with your patch we can remove patch 016 (so your patch
> > could replace it). I'm not sure about patch 004, though.
> 
> Indeed, patch 016 for distutils / setuptools is no longer necessary.
> Moreover, patch 004 for the standard library can be dropped as well.
> Afer removing patches 004 and 016, I see no __pycache__ directory in the
> Python path and the interpreter behaves as expected (I tested by loading
> modules and running samples for readline, urwid, six, kmod or
> pyparsing).
> 
> I will respin my patch to drop 004 and 016 and update 005 which is
> impacted by the removal of 004.

Excellent, thanks for the additional investigation! It's definitely
great to replace 3 patches by just one.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com