From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from aws-us-west-2-korg-lkml-1.web.codeaurora.org (localhost.localdomain [127.0.0.1]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DF4BC369A6 for ; Thu, 10 Apr 2025 11:37:00 +0000 (UTC) Subject: Re: python3: Allow to specify which pyc files to keep To: openembedded-core@lists.openembedded.org From: "Lukas Woodtli" X-Originating-Location: Zurich, CH (185.132.16.78) X-Originating-Platform: Linux Firefox 136 User-Agent: GROUPS.IO Web Poster MIME-Version: 1.0 Date: Thu, 10 Apr 2025 04:36:50 -0700 References: In-Reply-To: Message-ID: <13400.1744285010599994886@lists.openembedded.org> Content-Type: multipart/alternative; boundary="5qjtHgQtMZcQEAG7tt5l" List-Id: X-Webhook-Received: from li982-79.members.linode.com [45.33.32.79] by aws-us-west-2-korg-lkml-1.web.codeaurora.org with HTTPS for ; Thu, 10 Apr 2025 11:37:00 -0000 X-Groupsio-URL: https://lists.openembedded.org/g/openembedded-core/message/214651 --5qjtHgQtMZcQEAG7tt5l Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable >=20 > Should we rather just package the higher level .pyc into their own > packages? >=20 But this would include 3 additional packages. One for each optimization lev= el. Embedded devices might have limited storage resources. Therefore, it makes sense to just include the .pyc files with the necessary= optimization level to the image. And not all of them. Patch with (hopefully) proper formatting: >From d7159cd842fd2a10ba5aeff62877be2ebb2eab87 Mon Sep 17 00:00:00 2001 From: Lukas Woodtli Date: Mon, 7 Apr 2025 11:55:33 +0200 Subject: [PATCH] python3: Allow to specify which pyc files to keep The pyc files for a specific optimization level can be kept and are installed in the final image. Signed-off-by: Lukas Woodtli --- .../recipes-devtools/python/python3_3.13.2.bb | 32 +++++++++++++++---- 1 file changed, 25 insertions(+), 7 deletions(-) diff --git a/meta/recipes-devtools/python/python3_3.13.2.bb b/meta/recipes-= devtools/python/python3_3.13.2.bb index 7c36fd92ed..ac74432a2a 100644 --- a/meta/recipes-devtools/python/python3_3.13.2.bb +++ b/meta/recipes-devtools/python/python3_3.13.2.bb @@ -217,6 +217,12 @@ do_install:append:class-native() { mv ${D}/${bindir}/${PN}/python*config ${D}/${bindir}/ } +# We want bytecode precompiled .py files (.pyc's) by default +# but the user may set it on their own conf +INCLUDE_PYCS ?=3D "1" + +PYCS_OPT_LEVEL ?=3D "0" + do_install:append() { for c in ${D}/${libdir}/python${PYTHON_MAJMIN}/_sysconfigdata*.py; do python3 ${UNPACKDIR}/reformat_sysconfig.py $c @@ -249,10 +255,25 @@ do_install:append() { # so remove it too rm -f ${D}${libdir}/python${PYTHON_MAJMIN}/__pycache__/traceback.cpython* - =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0# Remove the opt-1.pyc and opt-= 2.pyc files. They effectively waste space on embedded - =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0# style targets as they're only= used when python is called with the -O or -OO options - =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0# which is rare. - =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0find ${D} -name *opt-*.pyc -del= ete + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if [ "${INCLUDE_PYCS}" -eq "1" = ]; then + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0#= Remove only the .pyc files of unused optimization level. + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0if [ "$= {PYCS_OPT_LEVEL}" -eq "0" ]; then + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0# keep only unoptimized .pyc files + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0find ${D} -name *.opt-1.pyc -exec rm -f {} \; + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0find ${D} -name *.opt-2.pyc -exec rm -f {} \; + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0elif [ = "${PYCS_OPT_LEVEL}" -eq "1" ]; then + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0# keep only .pyc files with optimization level 1 + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0find ${D} -name *.pyc -and -not -name *.opt-1.pyc -exec r= m -f {} \; + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0elif [ = "${PYCS_OPT_LEVEL}" -eq "2" ]; then + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0# keep only .pyc files with optimization level 2 + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0find ${D} -name *.pyc -and -not -name *.opt-2.pyc -exec r= m -f {} \; + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0else + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0bberror "Python optimization level ${PYCS_OPT_LEVEL} is n= ot supported" + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0fi + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0else + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0# remov= e all .pyc files + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0find ${= D} -name *.pyc -delete + =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0fi } do_install:append:class-nativesdk () { @@ -326,9 +347,6 @@ py_package_preprocess () { rm -rf ${PKGD}/${libdir}/python-sysconfigdata } -# We want bytecode precompiled .py files (.pyc's) by default -# but the user may set it on their own conf -INCLUDE_PYCS ?=3D "1" python(){ import collections, json -- 2.43.0 --5qjtHgQtMZcQEAG7tt5l Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable
 
Should we rather just package the higher level .pyc into their own pac= kages?
 
 
But this would include 3 additional packages. One for each optimizatio= n level.
Embedded devices might have limited storage resources.
Therefore, it makes sense to just include the .pyc files with the nece= ssary optimization level to the image.
And not all of them.
 
 
 
Patch with (hopefully) proper formatting:
 
From d7159cd842fd2a10ba5aeff62877be2ebb2eab87 = Mon Sep 17 00:00:00 2001
From: Lukas Woodtli <lukas.woodtli= @husqvarnagroup.com>
Date: Mon, 7 Apr 2025 11:55:33 +0200
Su= bject: [PATCH] python3: Allow to specify which pyc files to keep

The pyc files for a specific optimization level can be kept and are
installed in the final image.

Signed-off-by: Lukas Woodtli <= ;lukas.woodtli@husqvarnagroup.com>
---
.../recipes-devtools/= python/python3_3.13.2.bb | 32 +++++++++++++++----
1 file changed, 25 = insertions(+), 7 deletions(-)

diff --git a/meta/recipes-devtool= s/python/python3_3.13.2.bb b/meta/recipes-devtools/python/python3_3.13.2.bb=
index 7c36fd92ed..ac74432a2a 100644
--- a/meta/recipes-devtool= s/python/python3_3.13.2.bb
+++ b/meta/recipes-devtools/python/python3= _3.13.2.bb
@@ -217,6 +217,12 @@ do_install:append:class-native() {         mv ${D}/${bindir}/${PN}= /python*config ${D}/${bindir}/
}
 
+# We want bytecod= e precompiled .py files (.pyc's) by default
+# but the user may set i= t on their own conf
+INCLUDE_PYCS ?=3D "1"
+
+PYCS_OPT_LE= VEL ?=3D "0"
+
do_install:append() {
   &n= bsp;    for c in ${D}/${libdir}/python${PYTHON_MAJMIN}/= _sysconfigdata*.py; do
       &nbs= p;    python3 ${UNPACKDIR}/reformat_sysconfig.py $c @@ -249,10 +255,25 @@ do_install:append() {
   &nbs= p;    # so remove it too
    =     rm -f ${D}${libdir}/python${PYTHON_MAJMIN}/__pycach= e__/traceback.cpython*
 
-      &n= bsp; # Remove the opt-1.pyc and opt-2.pyc files. They effectively wast= e space on embedded
-        # sty= le targets as they're only used when python is called with the -O or -OO op= tions
-        # which is rare. -        find ${D} -name *opt-*.pyc = -delete
+        if [ "${INCLUDE_P= YCS}" -eq "1" ]; then
+        &nb= sp;    # Remove only the .pyc files of unused optimizat= ion level.
+          &n= bsp; if [ "${PYCS_OPT_LEVEL}" -eq "0" ]; then
+   &nbs= p;            #= keep only unoptimized .pyc files
+      &nb= sp;         find ${D} -name *.= opt-1.pyc -exec rm -f {} \;
+       &nb= sp;        find ${D} -name *.opt-2.= pyc -exec rm -f {} \;
+        &nb= sp;   elif [ "${PYCS_OPT_LEVEL}" -eq "1" ]; then
+ &nb= sp;            =   # keep only .pyc files with optimization level 1
+  =             &nb= sp; find ${D} -name *.pyc -and -not -name *.opt-1.pyc -exec rm -f {} \= ;
+            = ;elif [ "${PYCS_OPT_LEVEL}" -eq "2" ]; then
+     = ;           # keep o= nly .pyc files with optimization level 2
+     &n= bsp;          find ${D} -= name *.pyc -and -not -name *.opt-2.pyc -exec rm -f {} \;
+  &nbs= p;         else
+  =             &nb= sp; bberror "Python optimization level ${PYCS_OPT_LEVEL} is not suppor= ted"
+           &n= bsp;fi
+        else
+  = ;          # remove all .= pyc files
+          &nb= sp; find ${D} -name *.pyc -delete
+     &nbs= p;  fi
}
 
do_install:append:class-nativesd= k () {
@@ -326,9 +347,6 @@ py_package_preprocess () {
 &nb= sp;      rm -rf ${PKGD}/${libdir}/python-sysc= onfigdata
}
 
-# We want bytecode precompiled .py fil= es (.pyc's) by default
-# but the user may set it on their own conf <= br />-INCLUDE_PYCS ?=3D "1"
 
python(){
  &= nbsp; import collections, json
--  
2.43.0

=
--5qjtHgQtMZcQEAG7tt5l--