[Buildroot] [PATCH 0/2] support/size-stats: actually handle .pyc files

[Buildroot] [PATCH 0/2] support/size-stats: actually handle .pyc files

[yann.morin]

From: "Yann E. MORIN" <yann.morin@orange.com>

Hello All!

This small two-patch series properly associates .pyc files to the
corresponding package.


Regards,
Yann E. MORIN.


The following changes since commit 4cc29326c91877a385eeb288378c1099ea240ee9

  Merge branch 'nyma/misc-fixes' into orange-master (2026-07-01 08:13:38 +0200)


are available as patches in this mail series,

for you to apply patches up to c45315df9f1084fe0f6f932e1523cfa9c4e6d992

  support/scripts/size-stats: properly assign pre-compiled .pyc files (2026-08-10 15:24:11 +0200)


----------------------------------------------------------------
Yann E. MORIN (2):
      support/scripts/size-stats: prepare for more .py associated files
      support/scripts/size-stats: properly assign pre-compiled .pyc files

 support/scripts/size-stats | 37 ++++++++++++++++++++++++++++++++++---
 1 file changed, 34 insertions(+), 3 deletions(-)

-- 
                                        ____________
.-----------------.--------------------:       _    :------------------.
|  Yann E. MORIN  | Real-Time Embedded |    __/ )   | /"\ ASCII RIBBON |
|                 | Software  Designer |  _/ - /'   | \ / CAMPAIGN     |
| +33 638.411.245 '--------------------: (_    `--, |  X  AGAINST      |
| yann.morin (at) orange.com           |_="    ,--' | / \ HTML MAIL    |
'--------------------------------------:______/_____:------------------'

____________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 1/2] support/scripts/size-stats: prepare for more .py associated files

[yann.morin]

From: "Yann E. MORIN" <yann.morin@orange.com>

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Cc: Michael Klein <m.klein@mvz-labor-lb.de>
Cc: Vincent Fazio <vfazio@xes-inc.com>
---
 support/scripts/size-stats | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/support/scripts/size-stats b/support/scripts/size-stats
index 79c0dc571e..fdcaeaa546 100755
--- a/support/scripts/size-stats
+++ b/support/scripts/size-stats
@@ -54,9 +54,8 @@ class Config:
 # pkg: package to which the file belongs
 #
 def add_file(filesdict, relpath, abspath, pkg):
-    if relpath.endswith(".py"):
-        # also check for compiled .pyc file
-        add_file(filesdict, relpath + "c", abspath + "c", pkg)
+    for relpath2, abspath2 in get_associated_files(relpath, abspath):
+        add_file(filesdict, relpath2, abspath2, pkg)
     if not os.path.exists(abspath):
         return
     if os.path.islink(abspath):
@@ -65,6 +64,22 @@ def add_file(filesdict, relpath, abspath, pkg):
     filesdict[relpath] = (pkg, sz)
 
 
+#
+# Returns an iterator over the files associated with the given one
+# as a 2-tuple of (relpath, abspath).
+#
+# Supported files:
+#  *.py => yields the list of corresponding pre-compiled .pyc files
+#
+# relpath: relative path of the file
+# fullpath: absolute path to the file
+#
+def get_associated_files(relpath, abspath):
+    # also check for compiled .pyc file
+    if relpath.endswith(".py"):
+        yield (relpath + "c", abspath + "c")
+
+
 #
 # This function returns a dict where each key is the path of a file in
 # the root filesystem, and the value is a tuple containing two
-- 
2.43.0

____________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/2] support/scripts/size-stats: properly assign pre-compiled .pyc files

[yann.morin]

From: "Yann E. MORIN" <yann.morin@orange.com>

Since commit 3fed42456693 (package/python3: use the provided pyc
compiler), we abide by the PEP 3147 guidelines on where the pyc files
are located. In practice, this means that, for a .py file /foo/bar.py,
the corresponding .pyc file will be /foo/__pycache__/bar.pythonX.Y.pyc
(where X.Y is the current major.minor python version, e.g. 3.14).

Add support for this layout in the size-stats scrip, which so far only
supported the legacy, pre-PEP 3147 layout of having the .pyc next to the
.py.

When we associate files to a package, we only have the original name,
and we must construct the associated filenames. For the .pyc, this would
require that the version of python be passed to the size-stats script,
but this is not so nice. Instead, just scan the pycache directory to
find the corresponding files. This is not much nicer either, but at
least the wart, if bigger, is in a single place that would be easy to
chop out or rework if need be (e.g. if we need to add support for more
similarly target-finalize-generated files in the future)...

Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
Cc: Michael Klein <m.klein@mvz-labor-lb.de>
Cc: Vincent Fazio <vfazio@xes-inc.com>
---
 support/scripts/size-stats | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/support/scripts/size-stats b/support/scripts/size-stats
index fdcaeaa546..9a0aafbf2e 100755
--- a/support/scripts/size-stats
+++ b/support/scripts/size-stats
@@ -77,7 +77,23 @@ def add_file(filesdict, relpath, abspath, pkg):
 def get_associated_files(relpath, abspath):
     # also check for compiled .pyc file
     if relpath.endswith(".py"):
+        # Legacy .pyc next to .py
         yield (relpath + "c", abspath + "c")
+        # PEP 3147 layout
+        basename = os.path.basename(abspath).rsplit(".", 1)[0]
+        reldir = os.path.join(os.path.dirname(relpath), "__pycache__")
+        absdir = os.path.join(os.path.dirname(abspath), "__pycache__")
+        try:
+            for fname in (
+                fn
+                for fn in os.listdir(absdir)
+                if fn.endswith(".pyc")
+                and fn.rsplit(".", 2)[0] == basename
+            ):
+                yield (os.path.join(reldir, fname), os.path.join(absdir, fname))
+        except FileNotFoundError:
+            # No precompiled .pyc files
+            pass
 
 
 #
-- 
2.43.0

____________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 0/2] support/size-stats: actually handle .pyc files

[yann.morin]

Hello All,

On 2026-08-10 15:27 +0200, yann.morin@orange.com spake thusly:
> This small two-patch series properly associates .pyc files to the
> corresponding package.

Even though rc1 has been tagged, I believe this is still material for
master, as this is a fix.

Any feedback on the patches, in case I need to rework something before
the release? ;-)

Regards,
Yann E. MORIN.

-- 
                                        ____________
.-----------------.--------------------:       _    :------------------.
|  Yann E. MORIN  | Real-Time Embedded |    __/ )   | /"\ ASCII RIBBON |
|                 | Software  Designer |  _/ - /'   | \ / CAMPAIGN     |
| +33 638.411.245 '--------------------: (_    `--, |  X  AGAINST      |
| yann.morin (at) orange.com           |_="    ,--' | / \ HTML MAIL    |
'--------------------------------------:______/_____:------------------'

____________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2] support/scripts/size-stats: properly assign pre-compiled .pyc files

[Vincent Fazio]

Yann,


> -----Original Message-----
> From: yann.morin@orange.com <yann.morin@orange.com>
> Sent: Monday, August 10, 2026 8:28 AM
> To: buildroot@buildroot.org
> Cc: yann.morin@orange.com; Michael Klein <m.klein@mvz-labor-lb.de>;
> Vincent Fazio <vfazio@xes-inc.com>
> Subject: [PATCH 2/2] support/scripts/size-stats: properly assign
> pre-compiled .pyc files
> 
> From: "Yann E. MORIN" <yann.morin@orange.com>
> 
> Since commit 3fed42456693 (package/python3: use the provided pyc
> compiler), we abide by the PEP 3147 guidelines on where the pyc files
> are located. In practice, this means that, for a .py file /foo/bar.py,
> the corresponding .pyc file will be /foo/__pycache__/bar.pythonX.Y.pyc
> (where X.Y is the current major.minor python version, e.g. 3.14).
> 
> Add support for this layout in the size-stats scrip, which so far only
> supported the legacy, pre-PEP 3147 layout of having the .pyc next to the
> .py.
> 
> When we associate files to a package, we only have the original name,
> and we must construct the associated filenames. For the .pyc, this would
> require that the version of python be passed to the size-stats script,
> but this is not so nice. Instead, just scan the pycache directory to
> find the corresponding files. This is not much nicer either, but at
> least the wart, if bigger, is in a single place that would be easy to
> chop out or rework if need be (e.g. if we need to add support for more
> similarly target-finalize-generated files in the future)...
> 
> Signed-off-by: Yann E. MORIN <yann.morin@orange.com>
> Cc: Michael Klein <m.klein@mvz-labor-lb.de>
> Cc: Vincent Fazio <vfazio@xes-inc.com>
> ---
>  support/scripts/size-stats | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/support/scripts/size-stats b/support/scripts/size-stats
> index fdcaeaa546..9a0aafbf2e 100755
> --- a/support/scripts/size-stats
> +++ b/support/scripts/size-stats
> @@ -77,7 +77,23 @@ def add_file(filesdict, relpath, abspath, pkg):
>  def get_associated_files(relpath, abspath):
>      # also check for compiled .pyc file
>      if relpath.endswith(".py"):
> +        # Legacy .pyc next to .py
>          yield (relpath + "c", abspath + "c")
> +        # PEP 3147 layout
> +        basename = os.path.basename(abspath).rsplit(".", 1)[0]
> +        reldir = os.path.join(os.path.dirname(relpath), "__pycache__")
> +        absdir = os.path.join(os.path.dirname(abspath), "__pycache__")
> +        try:
> +            for fname in (
> +                fn
> +                for fn in os.listdir(absdir)
> +                if fn.endswith(".pyc")
> +                and fn.rsplit(".", 2)[0] == basename

It's not an immediate concern because we don't have support for it (yet), but there are multiple optimization levels which pyc can be compiled for (1-3) [0][1].

I think (untested) the current rsplit will fail on an optimized file pattern and we can be a bit more future-proof so we don't accidentally run into this same problem again?

Maybe we can leverage `source_from_cache` [2] and compare the file names?

It's probably overkill, but I think this is one of those easily overlooked things where if we did add support for optimization levels that we'd miss fixing the code here to account for it.

Anyway, something to consider.

-Vincent

[0]: https://peps.python.org/pep-0488/
[1]: https://docs.python.org/3.15/using/cmdline.html#cmdoption-O
[2]: https://github.com/python/cpython/blob/20e6c2fc7c174342214d561845419c4030f8638f/Lib/importlib/_bootstrap_external.py#L299
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot