Re: [OE-core] python3: Allow to specify which pyc files to keep

Lukas Woodtli
2025-04-17T09:19:06Z

The hardware you describe closely resembles our latest Linux-based
product. Unfortunately, we need to work with such outdated and
constrained technology.

Regarding the compilation and optimization of Python files:

We need the files to be pre-compiled at build-time. Otherwise, the
initial startup of the device takes too long due to the compilation
process and saving the compiled files to the flash. This slows the
system down to the point where our services are killed by the watchdog.

Asserts should not be active in the production build. They are not a
replacement for proper error handling. Disabling asserts in release
builds is considered best practice, regardless of the programming
environment. In my experience, this approach works also well with Python
projects.

Removing docstrings is more critical. Some packages require docstrings
to be available at runtime in the native Python interpreter on the build
hose. However, we have not encountered any issues removing them on the
target device. This results in a small saving in storage space, but we
take every bit.

We will also consider removing all .py files and relying solely on
optimized .pyc files.

I have implemented the necessary optimizations internally, as there does
not seem to be sufficient interest in supporting them upstream.

One could argue that Python is not the ideal technology for such a
limited device, but that's another story...

Regards,
Lukas

On 11.04.25 10:55, Mike Looijmans via lists.openembedded.org wrote:
> On 10-04-2025 16:31, Alexander Kanavin via lists.openembedded.org wrote:
>> On Thu, 10 Apr 2025 at 15:05, Lukas Woodtli via lists.openembedded.org
>> <lw=hqv.ch@lists.openembedded.org> wrote:
>>> The reason to use these features of the Python interpreter is that
>>> we develop for
>>> resource restricted devices. The target device should not need to
>>> compile the Python
>>> files to byte-code at runtime (which happens regardless of the
>>> optimization level).
>>> We can do this work at build time. Moreover, Embedded devices usually
>>> have restricted storage size. So using the optimization level 2 for
>>> pyc files is sensible.
>>> Most embedded devices don't need the docstrings in the Python files
>>> (simmilar to
>>> not needing the man pages). So we can get rid of them (that is what
>>> optimization
>>> level 2 does, apart from disabling asserts).
>> Most embedded devices also need to be robust. Lest you forget, python
>> and its extensions are written in C, and asserts prevent what is
>> over-generously termed 'undefined behavior', with a loud crash. Which
>> is a feature some people find useful. Can you demonstrate speedups?
>>
>> Docstrings argument as well needs to be substantiated with numbers.
>> How much space is actually being saved?
>>
>> Is there anything else to these optimizations? Is it documented
>> anywhere? How can we know if it changes over time, with new python
>> releases? I feel like this needs to be considered far more carefully
>> than you have so far.
>
> Getting a deja-vu feel here... We've used a similar approach in the
> past for OpenPLi, as we had to cram the full DVB settop box GUI
> firmware into 32MB NAND flash for some boxes.
>
> The impact of the "optimization" on runtime performance was
> negligible, and we're talking about 200-400 MHz MIPS CPU systems in
> those days.
>
> Python's optimization level 2 is sure to break things. Quite a large
> software base uses asserts and docstrings for their internal purposes.
> One might argue that that is "broken", but that's not something that
> can be fixed from this side...
>
> We didn't experience any ill effects from optimization level 1, so the
> we just patched the Python stuff to just always use that, regardless
> of the file extension being pyc.
>
> As for space savings, the best thing was to omit the "py" files. That
> results in more than halving the space required. To accomplish that,
> we split the packages and put all "py" source files into a "-src"
> package. So on the end product, only the pyc files got installed and
> developers could optionally install some -src packages if they wanted
> to hack on the box itself (the major advantage of using Python).
>
> M.
>
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#214720): https://lists.openembedded.org/g/openembedded-core/message/214720
> Mute This Topic: https://lists.openembedded.org/mt/112186823/9160547
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [lw@hqv.ch]
> -=-=-=-=-=-=-=-=-=-=-=-
>