Re: [OE-core] [RFC][PATCH 00/14] Support Python freethreading

["Böszörményi Zoltán" <zboszor@gmail.com>]

2026. 03. 26. 6:29 keltezéssel, Zoltan Boszormenyi via lists.openembedded.org írta:
> 2026. 03. 25. 15:17 keltezéssel, Bruce Ashfield írta:
>>
>>
>> On Wed, Mar 25, 2026 at 9:51 AM Böszörményi Zoltán <zboszor@gmail.com> wrote:
>>
>>     2026. 03. 25. 14:17 keltezéssel, Bruce Ashfield írta:
>>     >
>>     >
>>     > On Wed, Mar 25, 2026 at 9:13 AM Böszörményi Zoltán <zboszor@gmail.com> wrote:
>>     >
>>     >     2026. 03. 25. 14:01 keltezéssel, Bruce Ashfield írta:
>>     >     >
>>     >     >
>>     >     > On Wed, Mar 25, 2026 at 7:31 AM Zoltan Boszormenyi via
>>     lists.openembedded.org <http://lists.openembedded.org>
>>     >     <http://lists.openembedded.org>
>>     >     > <http://lists.openembedded.org> <zboszor=gmail.com@lists.openembedded.org>
>>     wrote:
>>     >     >
>>     >     >     This patchset allows turning on Python freethreading by using
>>     >     >     a new DISTRO_FEATURES setting:
>>     >     >
>>     >     >     DISTRO_FEATURES += "python_freethreading"
>>     >     >
>>     >     >     and optionally:
>>     >     >
>>     >     >     DISTRO_FEATURES_NATIVE += "python_freethreading"
>>     >     >
>>     >     >     The change to python3-dir.bbclass is crucial, as all python module
>>     >     >     build helper classes rely on it.
>>     >     >
>>     >     >     I also added native variants of some settings because another
>>     >     >     series against meta-openembedded is needed to complete this one.
>>     >     >     In that, the change to the fontforge recipe relies on the native
>>     >     >     setting for setting BUILD_LDFLAGS.
>>     >     >
>>     >     >
>>     >     > Looking at those other patches in the series, they don't seem to
>>     >     > be conditional on the distro feature.
>>     >     >
>>     >     > Are those changes valid for a build with and without freethreading ?
>>     >
>>     >     Yes!
>>     >
>>     >     The main change is in python3-dir.bbclass, and the recipe tweaks
>>     >     are just adaptations to the fact that only ${PYTHON_DIR} is needed,
>>     >     not ${PYTHON_DIR}${PYTHON_ABI} from that point.
>>     >
>>     >     I was thinking that the introductory mail stressed this enough.
>>     >
>>     >
>>     > Obviously not :)
>>
>>     Obviously. :)
>>
>>     > I didn't get that impression at all from reading the introductory
>>     > message or looking at the patches themselves.
>>     >
>>     > You have the overview knowledge of the feature, why it is
>>     > useful, how it impacts existing recipes, how the migration would
>>     > work, what is the compatibility, etc.
>>
>>     I think the Python 3.14 release notes has enough details.
>>     See also https://www.phoronix.com/news/Python-3.14
>>
>>
>> The information needs to be distilled into your commit, not left
>> to the patch reader to go off and read release notes. That's the
>> point I'm trying to make.
>>
>>
>>
>>     With the two series I sent, only a single recipe is left (cmpi-bindings)
>>     in openembedded-core and meta-openembedded that uses
>>     the PYTHON_ABI setting explicitly.
>>
>>     Migration (of python scripts) should not be a problem.
>>     If it worked with 3.14 as is, it works with freethreading enabled.
>>
>>     Compatibility is poor for a lot of modules, that clearly spell this out
>>     in their pyproject.toml or PKG-INFO:
>>
>>          "Programming Language :: Python :: Free Threading :: 1 - Unstable"
>>
>>     or something not quite production level support.
>>     It's still experimental for 3rd party modules, despite being officially
>>     supported for the interpreter and built-in modules in 3.14.
>>
>>     This is the reason this was an RFC series.
>>
>>     Anyway, the reason I was experimenting with this was that
>>     with heavily threaded python code, an import storm can cause
>>     a crash in 3.12.11 or the manually updated 3.13.12 in Yocto 5.3 in
>>     importlib._bootstrap according to the backtrace.
>>
>>     I would like to see if Python 3.14 (with and without freethreading)
>>     fixes such crashes.
>>
>>
>> Definitely worth putting in the RFC as well, the motivation for
>> the change matters!
>>
>>
>>     > The patch reader doesn't, so that all needs to be clearly
>>     > supplied.
>>
>>     Maybe next time for v2 after I have a complete image
>>     successfully built with freethreading enabled.
>>
>
> No such luck.
>
> lldb in clang won't build because its swig source wants to play games
> with PyObject->ob_refcnt directly instead of using the proper macros.
> That struct member does not exist when freethreading is enabled.
> In fact, the whole struct _object (the PyObject implementation)
> is different when freethreading is enabled.
>
> The llvm sources, even in the newly released 22.x and main
> do not have any reference to the Py_GIL_DISABLED macro,
> so as a whole, it does not support Python freethreading.
>
> I can revisit this series a few year later when llvm adds the
> necessary support, which can be backported to older clang
> versions in my meta-clang-revival.
>
> This concludes my little experiment.
>
> That being said, I think the cleanup in the recipes is a good direction.
> PYTHON_DIR should include PYTHON_ABI in python3-dir.bbclass
> because build helper classes rely on it. Then PYTHON_DIR must be
> used consistently in recipes that need it, even if PYTHON_ABI stays empty.

Which means I can re-send the whole series as a cleanup
or preparation series without the DISTRO_FEATURES enablement
part in python3-dir.bbclass.

That one can stay RFC as a standalone patch, separate from
the cleanup series.

I will re-test my build without enabling freethreading
before I would do that.

>
>>
>> Ack'd
>>
>> Bruce
>>
>>
>>     >
>>     > Bruce
>>     >
>>     >
>>     >
>>     >     FWIW, less than 30 recipes was spelling the latter out in
>>     >     openembedded-core and meta-openembedded combined.
>>     >
>>     >     > That's the type of information, along with the benefits of enabling
>>     >     > this would be information expected in a 0/N introductory patch.
>>     >     >
>>     >     > Cheers,
>>     >     >
>>     >     > Bruce
>>     >     >
>>     >     >
>>     >     >     All of the changes included here were successfully build tested,
>>     >     >     except for the gdb-cross-canadian change. That one failed, but
>>     >     >     it's included for completeness.
>>     >     >
>>     >     >     Quite a few of python modules were build tested for native and
>>     >     >     target builds, including python3-pytorch from my meta-python-ai
>>     >     >     with reduced RDEPENDS because e.g. python3-onnx there failed
>>     >     >     to build with freethreading enabled.
>>     >     >
>>     >     >     All in all, this series at least allows to get things going.
>>     >     >     Though maybe not for Yocto 6.0, as it's quite late.
>>     >     >
>>     >     >     Best regards,
>>     >     >     Zoltán Böszörményi
>>     >     >
>>     >     >
>>     >     >
>>     >     >
>>     >     >
>>     >     > --
>>     >     > - Thou shalt not follow the NULL pointer, for chaos and madness await thee
>>     at its end
>>     >     > - "Use the force Harry" - Gandalf, Star Trek II
>>     >     >
>>     >
>>     >
>>     >
>>     > --
>>     > - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its 
>> end
>>     > - "Use the force Harry" - Gandalf, Star Trek II
>>     >
>>
>>
>>
>> -- 
>> - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
>> - "Use the force Harry" - Gandalf, Star Trek II
>>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#233951): https://lists.openembedded.org/g/openembedded-core/message/233951
> Mute This Topic: https://lists.openembedded.org/mt/118499141/3617728
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [zboszor@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>