Re: [OE-core] python3: Allow to specify which pyc files to keep

[Mike Looijmans <mike.looijmans@topic.nl>]

On 10-04-2025 16:31, Alexander Kanavin via lists.openembedded.org wrote:
> On Thu, 10 Apr 2025 at 15:05, Lukas Woodtli via lists.openembedded.org
> <lw=hqv.ch@lists.openembedded.org> wrote:
>> The reason to use these features of the Python interpreter is that we develop for
>> resource restricted devices. The target device should not need to compile the Python
>> files to byte-code at runtime (which happens regardless of the optimization level).
>> We can do this work at build time. Moreover, Embedded devices usually
>> have restricted storage size. So using the optimization level 2 for pyc files is sensible.
>> Most embedded devices don't need the docstrings in the Python files (simmilar to
>> not needing the man pages). So we can get rid of them (that is what optimization
>> level 2 does, apart from disabling asserts).
> Most embedded devices also need to be robust. Lest you forget, python
> and its extensions are written in C, and asserts prevent what is
> over-generously termed 'undefined behavior', with a loud crash. Which
> is a feature some people find useful. Can you demonstrate speedups?
>
> Docstrings argument as well needs to be substantiated with numbers.
> How much space is actually being saved?
>
> Is there anything else to these optimizations? Is it documented
> anywhere? How can we know if it changes over time, with new python
> releases? I feel like this needs to be considered far more carefully
> than you have so far.

Getting a deja-vu feel here... We've used a similar approach in the past for 
OpenPLi, as we had to cram the full DVB settop box GUI firmware into 32MB NAND 
flash for some boxes.

The impact of the "optimization" on runtime performance was negligible, and 
we're talking about 200-400 MHz MIPS CPU systems in those days.

Python's optimization level 2 is sure to break things. Quite a large software 
base uses asserts and docstrings for their internal purposes. One might argue 
that that is "broken", but that's not something that can be fixed from this 
side...

We didn't experience any ill effects from optimization level 1, so the we just 
patched the Python stuff to just always use that, regardless of the file 
extension being pyc.

As for space savings, the best thing was to omit the "py" files. That results 
in more than halving the space required. To accomplish that, we split the 
packages and put all "py" source files into a "-src" package. So on the end 
product, only the pyc files got installed and developers could optionally 
install some -src packages if they wanted to hack on the box itself (the major 
advantage of using Python).

M.