Re: [OE-core] [RFC][PATCH 00/14] Support Python freethreading

["Böszörményi Zoltán" <zboszor@gmail.com>]

2026. 03. 25. 15:17 keltezéssel, Bruce Ashfield írta:
>
>
> On Wed, Mar 25, 2026 at 9:51 AM Böszörményi Zoltán <zboszor@gmail.com> wrote:
>
>     2026. 03. 25. 14:17 keltezéssel, Bruce Ashfield írta:
>     >
>     >
>     > On Wed, Mar 25, 2026 at 9:13 AM Böszörményi Zoltán <zboszor@gmail.com> wrote:
>     >
>     >     2026. 03. 25. 14:01 keltezéssel, Bruce Ashfield írta:
>     >     >
>     >     >
>     >     > On Wed, Mar 25, 2026 at 7:31 AM Zoltan Boszormenyi via
>     lists.openembedded.org <http://lists.openembedded.org>
>     >     <http://lists.openembedded.org>
>     >     > <http://lists.openembedded.org> <zboszor=gmail.com@lists.openembedded.org>
>     wrote:
>     >     >
>     >     >     This patchset allows turning on Python freethreading by using
>     >     >     a new DISTRO_FEATURES setting:
>     >     >
>     >     >     DISTRO_FEATURES += "python_freethreading"
>     >     >
>     >     >     and optionally:
>     >     >
>     >     >     DISTRO_FEATURES_NATIVE += "python_freethreading"
>     >     >
>     >     >     The change to python3-dir.bbclass is crucial, as all python module
>     >     >     build helper classes rely on it.
>     >     >
>     >     >     I also added native variants of some settings because another
>     >     >     series against meta-openembedded is needed to complete this one.
>     >     >     In that, the change to the fontforge recipe relies on the native
>     >     >     setting for setting BUILD_LDFLAGS.
>     >     >
>     >     >
>     >     > Looking at those other patches in the series, they don't seem to
>     >     > be conditional on the distro feature.
>     >     >
>     >     > Are those changes valid for a build with and without freethreading ?
>     >
>     >     Yes!
>     >
>     >     The main change is in python3-dir.bbclass, and the recipe tweaks
>     >     are just adaptations to the fact that only ${PYTHON_DIR} is needed,
>     >     not ${PYTHON_DIR}${PYTHON_ABI} from that point.
>     >
>     >     I was thinking that the introductory mail stressed this enough.
>     >
>     >
>     > Obviously not :)
>
>     Obviously. :)
>
>     > I didn't get that impression at all from reading the introductory
>     > message or looking at the patches themselves.
>     >
>     > You have the overview knowledge of the feature, why it is
>     > useful, how it impacts existing recipes, how the migration would
>     > work, what is the compatibility, etc.
>
>     I think the Python 3.14 release notes has enough details.
>     See also https://www.phoronix.com/news/Python-3.14
>
>
> The information needs to be distilled into your commit, not left
> to the patch reader to go off and read release notes. That's the
> point I'm trying to make.
>
>
>
>     With the two series I sent, only a single recipe is left (cmpi-bindings)
>     in openembedded-core and meta-openembedded that uses
>     the PYTHON_ABI setting explicitly.
>
>     Migration (of python scripts) should not be a problem.
>     If it worked with 3.14 as is, it works with freethreading enabled.
>
>     Compatibility is poor for a lot of modules, that clearly spell this out
>     in their pyproject.toml or PKG-INFO:
>
>          "Programming Language :: Python :: Free Threading :: 1 - Unstable"
>
>     or something not quite production level support.
>     It's still experimental for 3rd party modules, despite being officially
>     supported for the interpreter and built-in modules in 3.14.
>
>     This is the reason this was an RFC series.
>
>     Anyway, the reason I was experimenting with this was that
>     with heavily threaded python code, an import storm can cause
>     a crash in 3.12.11 or the manually updated 3.13.12 in Yocto 5.3 in
>     importlib._bootstrap according to the backtrace.
>
>     I would like to see if Python 3.14 (with and without freethreading)
>     fixes such crashes.
>
>
> Definitely worth putting in the RFC as well, the motivation for
> the change matters!
>
>
>     > The patch reader doesn't, so that all needs to be clearly
>     > supplied.
>
>     Maybe next time for v2 after I have a complete image
>     successfully built with freethreading enabled.
>

No such luck.

lldb in clang won't build because its swig source wants to play games
with PyObject->ob_refcnt directly instead of using the proper macros.
That struct member does not exist when freethreading is enabled.
In fact, the whole struct _object (the PyObject implementation)
is different when freethreading is enabled.

The llvm sources, even in the newly released 22.x and main
do not have any reference to the Py_GIL_DISABLED macro,
so as a whole, it does not support Python freethreading.

I can revisit this series a few year later when llvm adds the
necessary support, which can be backported to older clang
versions in my meta-clang-revival.

This concludes my little experiment.

That being said, I think the cleanup in the recipes is a good direction.
PYTHON_DIR should include PYTHON_ABI in python3-dir.bbclass
because build helper classes rely on it. Then PYTHON_DIR must be
used consistently in recipes that need it, even if PYTHON_ABI stays empty.

>
> Ack'd
>
> Bruce
>
>
>     >
>     > Bruce
>     >
>     >
>     >
>     >     FWIW, less than 30 recipes was spelling the latter out in
>     >     openembedded-core and meta-openembedded combined.
>     >
>     >     > That's the type of information, along with the benefits of enabling
>     >     > this would be information expected in a 0/N introductory patch.
>     >     >
>     >     > Cheers,
>     >     >
>     >     > Bruce
>     >     >
>     >     >
>     >     >     All of the changes included here were successfully build tested,
>     >     >     except for the gdb-cross-canadian change. That one failed, but
>     >     >     it's included for completeness.
>     >     >
>     >     >     Quite a few of python modules were build tested for native and
>     >     >     target builds, including python3-pytorch from my meta-python-ai
>     >     >     with reduced RDEPENDS because e.g. python3-onnx there failed
>     >     >     to build with freethreading enabled.
>     >     >
>     >     >     All in all, this series at least allows to get things going.
>     >     >     Though maybe not for Yocto 6.0, as it's quite late.
>     >     >
>     >     >     Best regards,
>     >     >     Zoltán Böszörményi
>     >     >
>     >     >     -=-=-=-=-=-=-=-=-=-=-=-
>     >     >     Links: You receive all messages sent to this group.
>     >     >     View/Reply Online (#233874):
>     >     > https://lists.openembedded.org/g/openembedded-core/message/233874
>     >     >     Mute This Topic: https://lists.openembedded.org/mt/118499141/1050810
>     >     >     Group Owner: openembedded-core+owner@lists.openembedded.org
>     <mailto:openembedded-core%2Bowner@lists.openembedded.org>
>     >     <mailto:openembedded-core%2Bowner@lists.openembedded.org
>     <mailto:openembedded-core%252Bowner@lists.openembedded.org>>
>     >     >     <mailto:openembedded-core%2Bowner@lists.openembedded.org
>     <mailto:openembedded-core%252Bowner@lists.openembedded.org>
>     >     <mailto:openembedded-core%252Bowner@lists.openembedded.org
>     <mailto:openembedded-core%25252Bowner@lists.openembedded.org>>>
>     >     >     Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
>     >     >     [bruce.ashfield@gmail.com]
>     >     >     -=-=-=-=-=-=-=-=-=-=-=-
>     >     >
>     >     >
>     >     >
>     >     > --
>     >     > - Thou shalt not follow the NULL pointer, for chaos and madness await thee
>     at its end
>     >     > - "Use the force Harry" - Gandalf, Star Trek II
>     >     >
>     >
>     >
>     >
>     > --
>     > - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
>     > - "Use the force Harry" - Gandalf, Star Trek II
>     >
>
>
>
> -- 
> - Thou shalt not follow the NULL pointer, for chaos and madness await thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II
>