[Buildroot] [PATCH v2] python3: add patch to fix PEP 3147 issue with automake built packages

[Buildroot] [PATCH v2] python3: add patch to fix PEP 3147 issue with automake built packages

[Christophe Vu-Brugier]

Packages built with automake use a `py-compile` helper to byte-compile
Python source files. This script uses the "py_compile" module from the
standard library. In turn, the compile() function in the "py_compile"
module invokes the cache_from_source() function provided by importlib.

This commit adds a new patch named "020-importlib-no-pep3147.patch"
that changes cache_from_source() and source_from_cache() in importlib
to get rid of the "__pycache__" directory.

This commit fixes the following import error in kmod when the module
is built for Python 3:

  >>> from kmod import Kmod
  Traceback (most recent call last):
    File "<stdin>", line 1, in <module>
  ImportError: cannot import name 'Kmod'

Moreover, this commit removes two patches that are no longer necessary
since modifying cache_from_source() and source_from_cache() disables
PEP 3147 for the standard library and distutils / setuptools.

 * 004-old-stdlib-cache.patch
 * 016-distutils-no-pep3147.patch

Signed-off-by: Christophe Vu-Brugier <cvubrugier@fastmail.fm>

---
Changes since v1:
 * remove patches 004 and 016 as suggested by Thomas Petazzoni

---
 package/python3/004-old-stdlib-cache.patch     | 78 --------------------------
 package/python3/005-pyc-pyo-conditional.patch  |  8 +--
 package/python3/016-distutils-no-pep3147.patch | 36 ------------
 package/python3/020-importlib-no-pep3147.patch | 69 +++++++++++++++++++++++
 4 files changed, 73 insertions(+), 118 deletions(-)
 delete mode 100644 package/python3/004-old-stdlib-cache.patch
 delete mode 100644 package/python3/016-distutils-no-pep3147.patch
 create mode 100644 package/python3/020-importlib-no-pep3147.patch

diff --git a/package/python3/004-old-stdlib-cache.patch b/package/python3/004-old-stdlib-cache.patch
deleted file mode 100644
index 9ed7e59..0000000
--- a/package/python3/004-old-stdlib-cache.patch
+++ /dev/null
@@ -1,78 +0,0 @@
-python3: Fix pyc-only related runtime exceptions
-
-Python3 changes the pyc lookup strategy, ignoring the
-__pycache__ directory if the .py file is missing. Change
-install location to enable use of .pyc without their parent .py
-
-See http://www.python.org/dev/peps/pep-3147
-
-Signed-off-by: Daniel Nelson <daniel@sigpwr.com>
-
-Index: b/configure.ac
-===================================================================
---- a/configure.ac
-+++ b/configure.ac
-@@ -352,6 +352,23 @@
- AC_SUBST(FRAMEWORKUNIXTOOLSPREFIX)
- AC_SUBST(FRAMEWORKINSTALLAPPSPREFIX)
- 
-+STDLIB_CACHE_FLAGS=
-+AC_MSG_CHECKING(for --enable-old-stdlib-cache)
-+AC_ARG_ENABLE(old-stdlib-cache,
-+    AS_HELP_STRING([--enable-old-stdlib-cache], [enable pre-pep3147 stdlib cache]),
-+[
-+    if test "$enableval" = "yes"
-+    then
-+        STDLIB_CACHE_FLAGS="-b"
-+    else
-+        STDLIB_CACHE_FLAGS="" 
-+    fi
-+],
-+[
-+    STDLIB_CACHE_FLAGS="" 
-+])
-+AC_SUBST(STDLIB_CACHE_FLAGS)
-+
- ##AC_ARG_WITH(dyld,
- ##            AS_HELP_STRING([--with-dyld],
- ##                           [Use (OpenStep|Rhapsody) dynamic linker]))
-Index: b/Makefile.pre.in
-===================================================================
---- a/Makefile.pre.in
-+++ b/Makefile.pre.in
-@@ -162,6 +162,9 @@
- # Options to enable prebinding (for fast startup prior to Mac OS X 10.3)
- OTHER_LIBTOOL_OPT=@OTHER_LIBTOOL_OPT@
- 
-+# Option to enable old-style precompiled stdlib
-+STDLIB_CACHE_FLAGS=@STDLIB_CACHE_FLAGS@
-+
- # Environment to run shared python without installed libraries
- RUNSHARED=       @RUNSHARED@
- 
-@@ -1247,21 +1250,21 @@
- 	fi
- 	-PYTHONPATH=$(DESTDIR)$(LIBDEST)  $(RUNSHARED) \
- 		$(PYTHON_FOR_BUILD) -Wi $(DESTDIR)$(LIBDEST)/compileall.py \
--		-d $(LIBDEST) -f \
-+		-d $(LIBDEST) -f $(STDLIB_CACHE_FLAGS) \
- 		-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
- 		$(DESTDIR)$(LIBDEST)
- 	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
- 		$(PYTHON_FOR_BUILD) -Wi -O $(DESTDIR)$(LIBDEST)/compileall.py \
--		-d $(LIBDEST) -f \
-+		-d $(LIBDEST) -f $(STDLIB_CACHE_FLAGS) \
- 		-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
- 		$(DESTDIR)$(LIBDEST)
- 	-PYTHONPATH=$(DESTDIR)$(LIBDEST)  $(RUNSHARED) \
- 		$(PYTHON_FOR_BUILD) -Wi $(DESTDIR)$(LIBDEST)/compileall.py \
--		-d $(LIBDEST)/site-packages -f \
-+		-d $(LIBDEST)/site-packages -f $(STDLIB_CACHE_FLAGS) \
- 		-x badsyntax $(DESTDIR)$(LIBDEST)/site-packages
- 	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
- 		$(PYTHON_FOR_BUILD) -Wi -O $(DESTDIR)$(LIBDEST)/compileall.py \
--		-d $(LIBDEST)/site-packages -f \
-+		-d $(LIBDEST)/site-packages -f $(STDLIB_CACHE_FLAGS) \
- 		-x badsyntax $(DESTDIR)$(LIBDEST)/site-packages
- 	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
- 		$(PYTHON_FOR_BUILD) -m lib2to3.pgen2.driver $(DESTDIR)$(LIBDEST)/lib2to3/Grammar.txt
diff --git a/package/python3/005-pyc-pyo-conditional.patch b/package/python3/005-pyc-pyo-conditional.patch
index c3b159d..0d60ff6 100644
--- a/package/python3/005-pyc-pyo-conditional.patch
+++ b/package/python3/005-pyc-pyo-conditional.patch
@@ -9,27 +9,27 @@ Index: b/Makefile.pre.in
 +ifeq (@PYC_BUILD@,yes)
  	-PYTHONPATH=$(DESTDIR)$(LIBDEST)  $(RUNSHARED) \
  		$(PYTHON_FOR_BUILD) -Wi $(DESTDIR)$(LIBDEST)/compileall.py \
- 		-d $(LIBDEST) -f $(STDLIB_CACHE_FLAGS) \
+ 		-d $(LIBDEST) -f \
  		-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
  		$(DESTDIR)$(LIBDEST)
 +endif
 +ifeq (@PYO_BUILD@,yes)
  	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
  		$(PYTHON_FOR_BUILD) -Wi -O $(DESTDIR)$(LIBDEST)/compileall.py \
- 		-d $(LIBDEST) -f $(STDLIB_CACHE_FLAGS) \
+ 		-d $(LIBDEST) -f \
  		-x 'bad_coding|badsyntax|site-packages|lib2to3/tests/data' \
  		$(DESTDIR)$(LIBDEST)
 +endif
 +ifeq (@PYC_BUILD@,yes)
  	-PYTHONPATH=$(DESTDIR)$(LIBDEST)  $(RUNSHARED) \
  		$(PYTHON_FOR_BUILD) -Wi $(DESTDIR)$(LIBDEST)/compileall.py \
- 		-d $(LIBDEST)/site-packages -f $(STDLIB_CACHE_FLAGS) \
+ 		-d $(LIBDEST)/site-packages -f \
  		-x badsyntax $(DESTDIR)$(LIBDEST)/site-packages
 +endif
 +ifeq (@PYO_BUILD@,yes)
  	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
  		$(PYTHON_FOR_BUILD) -Wi -O $(DESTDIR)$(LIBDEST)/compileall.py \
- 		-d $(LIBDEST)/site-packages -f $(STDLIB_CACHE_FLAGS) \
+ 		-d $(LIBDEST)/site-packages -f \
  		-x badsyntax $(DESTDIR)$(LIBDEST)/site-packages
 +endif
  	-PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
diff --git a/package/python3/016-distutils-no-pep3147.patch b/package/python3/016-distutils-no-pep3147.patch
deleted file mode 100644
index 0fdfeb2..0000000
--- a/package/python3/016-distutils-no-pep3147.patch
+++ /dev/null
@@ -1,36 +0,0 @@
-Add distutils fix for PEP 3147 issue
-
-Python 3 has a new standard for installing .pyc file, called PEP
-3147. Unfortunately, this standard requires both the .py and .pyc
-files to be installed for a Python module to be found. This is quite
-annoying on space-constrained embedded systems, since the .py file is
-technically not required for execution.
-
-For the Python standard library, our Python 3 package already contains
-a patch named python3-004-old-stdlib-cache.patch, which allows to
-disable the PEP 3147 installation.
-
-But that lives the distutils/setuptools package an unsolved
-problem. This patch therefore adds a new patch to Python, which makes
-distutils package use the traditional installation path when byte
-compiling, rather than the PEP 3147 installation path. Since
-setuptools relies on distutils internally, it also fixes setuptools
-based packages.
-
-Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
-
-Index: b/Lib/distutils/util.py
-===================================================================
---- a/Lib/distutils/util.py
-+++ b/Lib/distutils/util.py
-@@ -437,7 +437,9 @@
-             # Terminology from the py_compile module:
-             #   cfile - byte-compiled file
-             #   dfile - purported source filename (same as 'file' by default)
--            if optimize >= 0:
-+            if "_python_sysroot" in os.environ:
-+                cfile = file + (__debug__ and "c" or "o")
-+            elif optimize >= 0:
-                 cfile = importlib.util.cache_from_source(
-                     file, debug_override=not optimize)
-             else:
diff --git a/package/python3/020-importlib-no-pep3147.patch b/package/python3/020-importlib-no-pep3147.patch
new file mode 100644
index 0000000..779b5e2
--- /dev/null
+++ b/package/python3/020-importlib-no-pep3147.patch
@@ -0,0 +1,69 @@
+Add importlib fix for PEP 3147 issue
+
+Python 3 has a new standard for installing .pyc file, called PEP
+3147. Unfortunately, this standard requires both the .py and .pyc
+files to be installed for a Python module to be found. This is quite
+annoying on space-constrained embedded systems, since the .py file is
+technically not required for execution.
+
+This patch changes cache_from_source() and source_from_cache() in
+importlib to get rid of the "__pycache__" directory.
+This effectively disables PEP 3147 for:
+
+* The python standard library
+* Packages built with distutils or setuptools
+* Packages built with automake that use the `py-compile` helper
+
+Signed-off-by: Christophe Vu-Brugier <cvubrugier@fastmail.fm>
+
+diff -purN a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py
+--- a/Lib/importlib/_bootstrap.py	2015-02-25 12:27:44.000000000 +0100
++++ b/Lib/importlib/_bootstrap.py	2015-08-27 17:33:26.167191059 +0200
+@@ -444,8 +444,6 @@ def cache_from_source(path, debug_overri
+     If debug_override is not None, then it must be a boolean and is used in
+     place of sys.flags.optimize.
+ 
+-    If sys.implementation.cache_tag is None then NotImplementedError is raised.
+-
+     """
+     debug = not sys.flags.optimize if debug_override is None else debug_override
+     if debug:
+@@ -454,33 +452,19 @@ def cache_from_source(path, debug_overri
+         suffixes = OPTIMIZED_BYTECODE_SUFFIXES
+     head, tail = _path_split(path)
+     base, sep, rest = tail.rpartition('.')
+-    tag = sys.implementation.cache_tag
+-    if tag is None:
+-        raise NotImplementedError('sys.implementation.cache_tag is None')
+-    filename = ''.join([(base if base else rest), sep, tag, suffixes[0]])
+-    return _path_join(head, _PYCACHE, filename)
++    filename = ''.join([(base if base else rest), suffixes[0]])
++    return _path_join(head, filename)
+ 
+ 
+ def source_from_cache(path):
+     """Given the path to a .pyc./.pyo file, return the path to its .py file.
+ 
+     The .pyc/.pyo file does not need to exist; this simply returns the path to
+-    the .py file calculated to correspond to the .pyc/.pyo file.  If path does
+-    not conform to PEP 3147 format, ValueError will be raised. If
+-    sys.implementation.cache_tag is None then NotImplementedError is raised.
++    the .py file calculated to correspond to the .pyc/.pyo file.
+ 
+     """
+-    if sys.implementation.cache_tag is None:
+-        raise NotImplementedError('sys.implementation.cache_tag is None')
+-    head, pycache_filename = _path_split(path)
+-    head, pycache = _path_split(head)
+-    if pycache != _PYCACHE:
+-        raise ValueError('{} not bottom-level directory in '
+-                         '{!r}'.format(_PYCACHE, path))
+-    if pycache_filename.count('.') != 2:
+-        raise ValueError('expected only 2 dots in '
+-                         '{!r}'.format(pycache_filename))
+-    base_filename = pycache_filename.partition('.')[0]
++    head, filename = _path_split(path)
++    base_filename = filename.partition('.')[0]
+     return _path_join(head, base_filename + SOURCE_SUFFIXES[0])
+ 
+ 
-- 
2.5.0

[Buildroot] [PATCH v2] python3: add patch to fix PEP 3147 issue with automake built packages

[Thomas Petazzoni]

Dear Christophe Vu-Brugier,

On Mon, 31 Aug 2015 14:21:44 +0200, Christophe Vu-Brugier wrote:
> Packages built with automake use a `py-compile` helper to byte-compile
> Python source files. This script uses the "py_compile" module from the
> standard library. In turn, the compile() function in the "py_compile"
> module invokes the cache_from_source() function provided by importlib.
> 
> This commit adds a new patch named "020-importlib-no-pep3147.patch"
> that changes cache_from_source() and source_from_cache() in importlib
> to get rid of the "__pycache__" directory.
> 
> This commit fixes the following import error in kmod when the module
> is built for Python 3:
> 
>   >>> from kmod import Kmod
>   Traceback (most recent call last):
>     File "<stdin>", line 1, in <module>
>   ImportError: cannot import name 'Kmod'
> 
> Moreover, this commit removes two patches that are no longer necessary
> since modifying cache_from_source() and source_from_cache() disables
> PEP 3147 for the standard library and distutils / setuptools.
> 
>  * 004-old-stdlib-cache.patch
>  * 016-distutils-no-pep3147.patch
> 
> Signed-off-by: Christophe Vu-Brugier <cvubrugier@fastmail.fm>
> 
> ---
> Changes since v1:
>  * remove patches 004 and 016 as suggested by Thomas Petazzoni

Applied, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v2] python3: add patch to fix PEP 3147 issue with automake built packages

[Thomas Petazzoni]

Hello Christophe,

In fact, it seems that this patch might be causing some Python 3 build
issues. See
http://autobuild.buildroot.org/results/06e/06e7c654a10a7618c0c64c57bfd9f27e770138fc/build-end.log.

Can you have a look?

Thanks!

Thomas

On Wed, 4 Nov 2015 22:55:41 +0100, Thomas Petazzoni wrote:
> Dear Christophe Vu-Brugier,
> 
> On Mon, 31 Aug 2015 14:21:44 +0200, Christophe Vu-Brugier wrote:
> > Packages built with automake use a `py-compile` helper to byte-compile
> > Python source files. This script uses the "py_compile" module from the
> > standard library. In turn, the compile() function in the "py_compile"
> > module invokes the cache_from_source() function provided by importlib.
> > 
> > This commit adds a new patch named "020-importlib-no-pep3147.patch"
> > that changes cache_from_source() and source_from_cache() in importlib
> > to get rid of the "__pycache__" directory.
> > 
> > This commit fixes the following import error in kmod when the module
> > is built for Python 3:
> > 
> >   >>> from kmod import Kmod
> >   Traceback (most recent call last):
> >     File "<stdin>", line 1, in <module>
> >   ImportError: cannot import name 'Kmod'
> > 
> > Moreover, this commit removes two patches that are no longer necessary
> > since modifying cache_from_source() and source_from_cache() disables
> > PEP 3147 for the standard library and distutils / setuptools.
> > 
> >  * 004-old-stdlib-cache.patch
> >  * 016-distutils-no-pep3147.patch
> > 
> > Signed-off-by: Christophe Vu-Brugier <cvubrugier@fastmail.fm>
> > 
> > ---
> > Changes since v1:
> >  * remove patches 004 and 016 as suggested by Thomas Petazzoni
> 
> Applied, thanks.
> 
> Thomas



-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v2] python3: add patch to fix PEP 3147 issue with automake built packages

[Christophe Vu-Brugier]

Hi Thomas,

On Thu, 5 Nov 2015 00:04:20 +0100, Thomas Petazzoni wrote :
> In fact, it seems that this patch might be causing some Python 3 build
> issues. See
> http://autobuild.buildroot.org/results/06e/06e7c654a10a7618c0c64c57bfd9f27e770138fc/build-end.log.
> 
> Can you have a look?

Sorry for the late reply.

Here is a short explanation for the error. The Makefile defines a rule
to regenerate "Python/importlib.h" if "Lib/importlib/_bootstrap.py" is
modified using a C helper named `_freeze_importlib`. Since
"_boostrap.py" is modified by my patch, the rule is invoked. The
problem is that the helper is built for the target so it cannot be
executed on the host.

If I understand correctly "importlib.h" is a frozen module: it contains
bytecode generated from "_bootstrap.py". It is only used by a dummy
"frozen.c" program.

I see two options:

1/ Patch the Makefile to invoke a version of `_freeze_importlib` built
   for the host.
2/ Simply do not regenerate "importlib.h" because it is not used.

I would favor the second option.

With best regards,

-- 
Christophe Vu-Brugier

[Buildroot] [PATCH v2] python3: add patch to fix PEP 3147 issue with automake built packages

[Arnout Vandecappelle]

On 12-11-15 16:31, Christophe Vu-Brugier wrote:
> Hi Thomas,
> 
> On Thu, 5 Nov 2015 00:04:20 +0100, Thomas Petazzoni wrote :
>> In fact, it seems that this patch might be causing some Python 3 build
>> issues. See
>> http://autobuild.buildroot.org/results/06e/06e7c654a10a7618c0c64c57bfd9f27e770138fc/build-end.log.
>>
>> Can you have a look?
> 
> Sorry for the late reply.
> 
> Here is a short explanation for the error. The Makefile defines a rule
> to regenerate "Python/importlib.h" if "Lib/importlib/_bootstrap.py" is
> modified using a C helper named `_freeze_importlib`. Since
> "_boostrap.py" is modified by my patch, the rule is invoked. The
> problem is that the helper is built for the target so it cannot be
> executed on the host.
> 
> If I understand correctly "importlib.h" is a frozen module: it contains
> bytecode generated from "_bootstrap.py". It is only used by a dummy
> "frozen.c" program.

 Isn't importlib.h installed to STAGING_DIR where it could be picked up by other
modules?

> 
> I see two options:
> 
> 1/ Patch the Makefile to invoke a version of `_freeze_importlib` built
>    for the host.
> 2/ Simply do not regenerate "importlib.h" because it is not used.

 Most likely, the generated importlib.h doesn't change due to your patch. So the
simplest solution is to do a 'touch $(@D)/Python/importlib.h' in a post-patch
hook (with a large comment explaining why it is necessary and allowed).


 Regards,
 Arnout

> 
> I would favor the second option.
> 
> With best regards,
> 


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v2] python3: add patch to fix PEP 3147 issue with automake built packages

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:

Hi,

 >> 2/ Simply do not regenerate "importlib.h" because it is not used.

 >  Most likely, the generated importlib.h doesn't change due to your patch. So the
 > simplest solution is to do a 'touch $(@D)/Python/importlib.h' in a post-patch
 > hook (with a large comment explaining why it is necessary and allowed).

Yes, sounds sensible. Christophe, care to send a patch?

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH v2] python3: add patch to fix PEP 3147 issue with automake built packages

[Christophe Vu-Brugier]

Hi,

On Fri, 13 Nov 2015 22:45:05 +0100, Peter Korsgaard wrote :
> >>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:  
> 
> Hi,
> 
>  >> 2/ Simply do not regenerate "importlib.h" because it is not
>  >> used.  
> 
>  >  Most likely, the generated importlib.h doesn't change due to your
>  > patch. So the simplest solution is to do a 'touch
>  > $(@D)/Python/importlib.h' in a post-patch hook (with a large
>  > comment explaining why it is necessary and allowed).  

Well importlib.h is changed because it contains the bytecode
of _bootstrap.py which is patched. However, importlib.h is not
installed in staging and Debian does not ship the header even in
development packages so it's useless.


> Yes, sounds sensible. Christophe, care to send a patch?

Patch sent with subject "[PATCH v3] python3: add patch to fix PEP 3147
issue with automake built packages".

Thanks and best regards,

-- 
Christophe Vu-Brugier