[RFC PATCH] package.bbclass: omit .pyc and .pyo file

[RFC PATCH] package.bbclass: omit .pyc and .pyo file

[Robert Yang]

We should not ship .pyc or .pyo file, but there are a few packages
ship .pyc, should we:
1) Ignore them in package.bbclass as this patch showes ?
Or
2) Add a qa check  then fix it by hand one by one ?

Here is the list of oe-core's world build:
python-smartpm-1.4.1
nativesdk-python-smartpm-1.4.1
python3-distribute-0.6.32
python-pycurl-7.19.5
python-pyrex-0.9.9
python-numpy-1.7.0
python-distribute-0.6.32
python-async-0.6.1
python-docutils-0.12
python-pycairo-1.10.0
python-scons-2.3.2
python-imaging-1.1.7
python-gitdb-0.5.4

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 meta/classes/package.bbclass |    6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
index fc501fc..6960221 100644
--- a/meta/classes/package.bbclass
+++ b/meta/classes/package.bbclass
@@ -1022,6 +1022,9 @@ python populate_packages () {
             files.append(file)
 
         for file in files:
+            # Skip .pyc and .pyo file.
+            if file.endswith('.pyc') or file.endswith('.pyo'):
+                continue
             if not cpath.islink(file):
                 if cpath.isdir(file):
                     newfiles =  [ os.path.join(file,x) for x in os.listdir(file) ]
@@ -1083,6 +1086,9 @@ python populate_packages () {
         if not dir:
             dir = os.sep
         for f in (files + dirs):
+            # Skip .pyc and .pyo file.
+            if f.endswith('.pyc') or f.endswith('.pyo'):
+                continue
             path = os.path.join(dir, f)
             if ('.' + path) not in seen:
                 unshipped.append(path)
-- 
1.7.9.5

Re: [RFC PATCH] package.bbclass: omit .pyc and .pyo file

[ChenQi]

On 01/07/2015 09:07 AM, Robert Yang wrote:
> We should not ship .pyc or .pyo file, but there are a few packages
> ship .pyc, should we:
> 1) Ignore them in package.bbclass as this patch showes ?
> Or
> 2) Add a qa check  then fix it by hand one by one ?

3) Fix it in a bbclass that python related recipes would generally inherit.

Regards,
Chen Qi

> Here is the list of oe-core's world build:
> python-smartpm-1.4.1
> nativesdk-python-smartpm-1.4.1
> python3-distribute-0.6.32
> python-pycurl-7.19.5
> python-pyrex-0.9.9
> python-numpy-1.7.0
> python-distribute-0.6.32
> python-async-0.6.1
> python-docutils-0.12
> python-pycairo-1.10.0
> python-scons-2.3.2
> python-imaging-1.1.7
> python-gitdb-0.5.4
>
> Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
> ---
>   meta/classes/package.bbclass |    6 ++++++
>   1 file changed, 6 insertions(+)
>
> diff --git a/meta/classes/package.bbclass b/meta/classes/package.bbclass
> index fc501fc..6960221 100644
> --- a/meta/classes/package.bbclass
> +++ b/meta/classes/package.bbclass
> @@ -1022,6 +1022,9 @@ python populate_packages () {
>               files.append(file)
>   
>           for file in files:
> +            # Skip .pyc and .pyo file.
> +            if file.endswith('.pyc') or file.endswith('.pyo'):
> +                continue
>               if not cpath.islink(file):
>                   if cpath.isdir(file):
>                       newfiles =  [ os.path.join(file,x) for x in os.listdir(file) ]
> @@ -1083,6 +1086,9 @@ python populate_packages () {
>           if not dir:
>               dir = os.sep
>           for f in (files + dirs):
> +            # Skip .pyc and .pyo file.
> +            if f.endswith('.pyc') or f.endswith('.pyo'):
> +                continue
>               path = os.path.join(dir, f)
>               if ('.' + path) not in seen:
>                   unshipped.append(path)

Re: [RFC PATCH] package.bbclass: omit .pyc and .pyo file

[Richard Purdie]

On Tue, 2015-01-06 at 17:07 -0800, Robert Yang wrote:
> We should not ship .pyc or .pyo file, but there are a few packages
> ship .pyc, should we:

Why should we not ship them? Doesn't python create these at runtime if
they're not present? What happens on a read only filesystem?

I'm sure we've had issues raised by someone with a read only filesystem
before FWIW.

I agree there is probably an issue here but deleting them may not be the
best option. I'm open to ideas though.

Cheers,

Richard

Re: [RFC PATCH] package.bbclass: omit .pyc and .pyo file

[Mike Looijmans]

On 01/07/2015 09:07 AM, Richard Purdie wrote:
> On Tue, 2015-01-06 at 17:07 -0800, Robert Yang wrote:
>> We should not ship .pyc or .pyo file, but there are a few packages
>> ship .pyc, should we:
>
> Why should we not ship them? Doesn't python create these at runtime if
> they're not present? What happens on a read only filesystem?

You definitely SHOULD ship the .pyc files. If they don't exist, the 
interpreter is forced to re-compile the .py source, and will attempt to write 
the result to the filesystem. It won't cause harm, it won't fail, but it's 
very inefficient. It's better to let the host do the py->pyc conversion anyway.

The opposite works just fine: You can omit the .py files and ship only .pyc 
files. We do that on settopboxes that use Python for the GUI, this saves 
several megabytes of flash space.
To accomplish that, we put the .py files into a $PN-src package.

There has been general agreement that .pyo files are utterly pointless.

> I'm sure we've had issues raised by someone with a read only filesystem
> before FWIW.
>
> I agree there is probably an issue here but deleting them may not be the
> best option. I'm open to ideas though.

My idea would be to standardize on shipping ONLY compiled files, and put the 
source .py files into a separate package named $PN-src by default. There is no 
need to install megabytes of python source files that neither users nor the 
interpreter will ever read.



Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/

Re: [RFC PATCH] package.bbclass: omit .pyc and .pyo file

[Robert Yang]

On 01/07/2015 05:23 PM, Mike Looijmans wrote:
> On 01/07/2015 09:07 AM, Richard Purdie wrote:
>> On Tue, 2015-01-06 at 17:07 -0800, Robert Yang wrote:
>>> We should not ship .pyc or .pyo file, but there are a few packages
>>> ship .pyc, should we:
>>
>> Why should we not ship them? Doesn't python create these at runtime if
>> they're not present? What happens on a read only filesystem?
>
> You definitely SHOULD ship the .pyc files. If they don't exist, the interpreter
> is forced to re-compile the .py source, and will attempt to write the result to
> the filesystem. It won't cause harm, it won't fail, but it's very inefficient.
> It's better to let the host do the py->pyc conversion anyway.

AFAIK, the .pyc is not version compatible, the .pyc created with the
build host's python may not work with the target python.

// Robert

>
> The opposite works just fine: You can omit the .py files and ship only .pyc
> files. We do that on settopboxes that use Python for the GUI, this saves several
> megabytes of flash space.
> To accomplish that, we put the .py files into a $PN-src package.
>
> There has been general agreement that .pyo files are utterly pointless.
>
>> I'm sure we've had issues raised by someone with a read only filesystem
>> before FWIW.
>>
>> I agree there is probably an issue here but deleting them may not be the
>> best option. I'm open to ideas though.
>
> My idea would be to standardize on shipping ONLY compiled files, and put the
> source .py files into a separate package named $PN-src by default. There is no
> need to install megabytes of python source files that neither users nor the
> interpreter will ever read.
>
>
>
> Met vriendelijke groet / kind regards,
>
> Mike Looijmans
> System Expert
>
>
> TOPIC Embedded Systems
> Eindhovenseweg 32-C, NL-5683 KH Best
> Postbus 440, NL-5680 AK Best
> Telefoon: (+31) (0) 499 33 69 79
> Telefax:  (+31) (0) 499 33 69 70
> E-mail: mike.looijmans@topic.nl
> Website: www.topic.nl
>
> Please consider the environment before printing this e-mail
>
> Topic zoekt gedreven (embedded) software specialisten!
> http://topic.nl/vacatures/topic-zoekt-software-engineers/
>

Re: [RFC PATCH] package.bbclass: omit .pyc and .pyo file

[Richard Purdie]

On Wed, 2015-01-07 at 17:32 +0800, Robert Yang wrote:
> 
> On 01/07/2015 05:23 PM, Mike Looijmans wrote:
> > On 01/07/2015 09:07 AM, Richard Purdie wrote:
> >> On Tue, 2015-01-06 at 17:07 -0800, Robert Yang wrote:
> >>> We should not ship .pyc or .pyo file, but there are a few packages
> >>> ship .pyc, should we:
> >>
> >> Why should we not ship them? Doesn't python create these at runtime if
> >> they're not present? What happens on a read only filesystem?
> >
> > You definitely SHOULD ship the .pyc files. If they don't exist, the interpreter
> > is forced to re-compile the .py source, and will attempt to write the result to
> > the filesystem. It won't cause harm, it won't fail, but it's very inefficient.
> > It's better to let the host do the py->pyc conversion anyway.
> 
> AFAIK, the .pyc is not version compatible, the .pyc created with the
> build host's python may not work with the target python.

I thought that was true for pyo but not pyc? Do you have a pointer to
documentation on that?

Cheers,

Richard

Re: [RFC PATCH] package.bbclass: omit .pyc and .pyo file

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 1625 bytes --]

On 7 January 2015 at 09:23, Mike Looijmans <mike.looijmans@topic.nl> wrote:

> You definitely SHOULD ship the .pyc files. If they don't exist, the
> interpreter is forced to re-compile the .py source, and will attempt to
> write the result to the filesystem. It won't cause harm, it won't fail, but
> it's very inefficient. It's better to let the host do the py->pyc
> conversion anyway.
>
> The opposite works just fine: You can omit the .py files and ship only
> .pyc files. We do that on settopboxes that use Python for the GUI, this
> saves several megabytes of flash space.
> To accomplish that, we put the .py files into a $PN-src package.
>

I agree with Mike, there is a use-case for just shipping .pyc.  Whether
oe-core should do something to assist with this or not is debatable.

There's an open bug report about this:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=6434.

https://docs.python.org/2/tutorial/modules.html#compiled-python-files has
the details we're after.  Summary:

.pyc is Python bytecode, which is an implementation detail of CPython.  As
such it's version-dependent but explicitly architecture-independent.
.pyo generated with -O is simply .pyc but with asserts removed.
.pyo generated with -O -O has asserts and docstrings removed.

I think it's fair to say we should just ignore .pyo - no point generating
and shipping it.  It does seem that if we want to ship usable .pyc we need
to ensure that they are compiled with python-native. I guess this could be
done by calling python-native's compileall module to recompile the sources
at package time.

Ross

[-- Attachment #2: Type: text/html, Size: 2616 bytes --]

Re: [RFC PATCH] package.bbclass: omit .pyc and .pyo file

[Mike Looijmans]

On 01/07/2015 12:16 PM, Burton, Ross wrote:
>
> On 7 January 2015 at 09:23, Mike Looijmans <mike.looijmans@topic.nl
> <mailto:mike.looijmans@topic.nl>> wrote:
>
>     You definitely SHOULD ship the .pyc files. If they don't exist, the
>     interpreter is forced to re-compile the .py source, and will attempt to
>     write the result to the filesystem. It won't cause harm, it won't fail,
>     but it's very inefficient. It's better to let the host do the py->pyc
>     conversion anyway.
>
>     The opposite works just fine: You can omit the .py files and ship only
>     .pyc files. We do that on settopboxes that use Python for the GUI, this
>     saves several megabytes of flash space.
>     To accomplish that, we put the .py files into a $PN-src package.
>
>
> I agree with Mike, there is a use-case for just shipping .pyc.  Whether
> oe-core should do something to assist with this or not is debatable.

We currently have this in a python_2.7.%.bbappend:

PACKAGES =+ "${PN}-src"
RDEPENDS_{PN}-src = "${PN}"
FILES_${PN}-src += "${libdir}/python${PYTHON_MAJMIN}/*.py"
FILES_${PN}-src += "${libdir}/python${PYTHON_MAJMIN}/*/*.py"
FILES_${PN}-src += "${libdir}/python${PYTHON_MAJMIN}/*/*/*.py"

This moves all sources into a single package. I experimented with creating src 
packages for each python sub-package, but all that accomplished was adding a 
lot of packages to the feed that no one would ever install.

I patched the larger Python recipes (e.g. twisted) in similar ways to reduce 
the image size (and network traffic).

A generic "please remove or split all .py files" flag or class would be a 
useful addition.

> There's an open bug report about this:
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=6434.
>
> https://docs.python.org/2/tutorial/modules.html#compiled-python-files has the
> details we're after.  Summary:
>
> .pyc is Python bytecode, which is an implementation detail of CPython.  As
> such it's version-dependent but explicitly architecture-independent.
> .pyo generated with -O is simply .pyc but with asserts removed.
> .pyo generated with -O -O has asserts and docstrings removed.

For horrible historic reasons OpenPLi (and its many forks) still use a patch 
that makes .pyo files the default instead of .pyc. It's been on my todo list 
for over a year now to get rid of it, but too many plugins and 3rd party stuff 
still count on this being the case.

> I think it's fair to say we should just ignore .pyo - no point generating and
> shipping it.  It does seem that if we want to ship usable .pyc we need to
> ensure that they are compiled with python-native. I guess this could be done
> by calling python-native's compileall module to recompile the sources at
> package time.

I'd expect any halfway decent recipe to have done that already in the 
do_compile step. Most, if not all, Python recipes already do so. If anything, 
the core could provide a class that provides a simple do_compile that calls 
compileall on the source dir.

Moving this to packaging will lead to unexpected failures, there are some 
situations where the source .py file must be installed and the .pyc will be 
ignored. A simple example is when the .py script is the application entry 
point, those aren't compiled into pyc at runtime, and adding a pyc would be a 
waste. Only the package's makefile (or whatever) can be expected to know that.

Mike.


Met vriendelijke groet / kind regards,

Mike Looijmans
System Expert


TOPIC Embedded Systems
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: (+31) (0) 499 33 69 79
Telefax:  (+31) (0) 499 33 69 70
E-mail: mike.looijmans@topic.nl
Website: www.topic.nl

Please consider the environment before printing this e-mail

Topic zoekt gedreven (embedded) software specialisten!
http://topic.nl/vacatures/topic-zoekt-software-engineers/