* [Buildroot] [PATCH] package/python3: ensure pyc embed rooted paths
@ 2022-08-25 20:05 Yann E. MORIN
2022-08-25 20:09 ` Yann E. MORIN
2022-08-29 19:16 ` Thomas Petazzoni via buildroot
0 siblings, 2 replies; 3+ messages in thread
From: Yann E. MORIN @ 2022-08-25 20:05 UTC (permalink / raw)
To: buildroot; +Cc: Yann E. MORIN, Vincent Fazio
To report usable tracebacks, pyc files embed the path of the original py
files, so that users can more easily try and debug the reported issue.
We genrate the pyc files by calling the python3-supplied compileall
script, to scan the directry where python modules are installed. Since
this is done on the build machine, we tell compileall.py to strip away
the TARGET_DIR prefix, as that has no meaning at runtime.
However, compileall.py forgets [0] to keep a leading / in the front of
the paths, thus generating non-rooted paths., e.g.:
/path/buildroot.ouput/targt/usr/lib/python3.10/argparse.py
gets embedded as:
usr/lib/python3.10/argparse.py
This is a bit confusing but, as far as we could see, should be mostly be
used for display purposes in tracebacks, and does not seem to impact
actual functkionality.
We fix that by instructing compileall.py that the embeeded paths should
be rooted to / which generates proper paths in tracebacks.
And alternate solution would be to swith gears, and tell compileall.py
exactly the resulting runtime "base" directory, wihch replaces the
stripping and prefixing; i.e. it's either:
-s $(TARGET_DIR) -p /
or
-d /usr/lib/python$(PYTHON3_MAJOR_VERSION)
We choose to keep the first solution, because that is semantically what
we really want to do: to strip the leading build-time path, rather than
to force anything.
Note: the python test-suite was executed with both solutions (in a
pyc-only setup), and the results were exactly the same; so in practice,
-d or -s+-p yield the same results.
Many thanks go to Vincent for reporting the issue and suggesting the
solutions.
Reported-by: Vincent Fazio <vfazio@xes-inc.com>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
package/python3/python3.mk | 1 +
1 file changed, 1 insertion(+)
diff --git a/package/python3/python3.mk b/package/python3/python3.mk
index 9af384712a..a9821d9a33 100644
--- a/package/python3/python3.mk
+++ b/package/python3/python3.mk
@@ -255,6 +255,7 @@ define PYTHON3_CREATE_PYC_FILES
$(if $(VERBOSE),,-q) \
$(if $(BR2_PACKAGE_PYTHON3_PYC_ONLY),-b) \
-s $(TARGET_DIR) \
+ -p / \
$(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
endef
--
2.25.1
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply related [flat|nested] 3+ messages in thread* Re: [Buildroot] [PATCH] package/python3: ensure pyc embed rooted paths
2022-08-25 20:05 [Buildroot] [PATCH] package/python3: ensure pyc embed rooted paths Yann E. MORIN
@ 2022-08-25 20:09 ` Yann E. MORIN
2022-08-29 19:16 ` Thomas Petazzoni via buildroot
1 sibling, 0 replies; 3+ messages in thread
From: Yann E. MORIN @ 2022-08-25 20:09 UTC (permalink / raw)
To: buildroot; +Cc: Vincent Fazio
On 2022-08-25 22:05 +0200, Yann E. MORIN spake thusly:
> To report usable tracebacks, pyc files embed the path of the original py
> files, so that users can more easily try and debug the reported issue.
>
> We genrate the pyc files by calling the python3-supplied compileall
> script, to scan the directry where python modules are installed. Since
> this is done on the build machine, we tell compileall.py to strip away
> the TARGET_DIR prefix, as that has no meaning at runtime.
>
> However, compileall.py forgets [0] to keep a leading / in the front of
This [0] note should have been:
[0] not sure whether this is a bug or a feature...
Regards,
Yann E. MORIN.
> the paths, thus generating non-rooted paths., e.g.:
> /path/buildroot.ouput/targt/usr/lib/python3.10/argparse.py
> gets embedded as:
> usr/lib/python3.10/argparse.py
>
> This is a bit confusing but, as far as we could see, should be mostly be
> used for display purposes in tracebacks, and does not seem to impact
> actual functkionality.
>
> We fix that by instructing compileall.py that the embeeded paths should
> be rooted to / which generates proper paths in tracebacks.
>
> And alternate solution would be to swith gears, and tell compileall.py
> exactly the resulting runtime "base" directory, wihch replaces the
> stripping and prefixing; i.e. it's either:
> -s $(TARGET_DIR) -p /
> or
> -d /usr/lib/python$(PYTHON3_MAJOR_VERSION)
>
> We choose to keep the first solution, because that is semantically what
> we really want to do: to strip the leading build-time path, rather than
> to force anything.
>
> Note: the python test-suite was executed with both solutions (in a
> pyc-only setup), and the results were exactly the same; so in practice,
> -d or -s+-p yield the same results.
>
> Many thanks go to Vincent for reporting the issue and suggesting the
> solutions.
>
> Reported-by: Vincent Fazio <vfazio@xes-inc.com>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> ---
> package/python3/python3.mk | 1 +
> 1 file changed, 1 insertion(+)
>
> diff --git a/package/python3/python3.mk b/package/python3/python3.mk
> index 9af384712a..a9821d9a33 100644
> --- a/package/python3/python3.mk
> +++ b/package/python3/python3.mk
> @@ -255,6 +255,7 @@ define PYTHON3_CREATE_PYC_FILES
> $(if $(VERBOSE),,-q) \
> $(if $(BR2_PACKAGE_PYTHON3_PYC_ONLY),-b) \
> -s $(TARGET_DIR) \
> + -p / \
> $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
> endef
>
> --
> 2.25.1
>
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Buildroot] [PATCH] package/python3: ensure pyc embed rooted paths
2022-08-25 20:05 [Buildroot] [PATCH] package/python3: ensure pyc embed rooted paths Yann E. MORIN
2022-08-25 20:09 ` Yann E. MORIN
@ 2022-08-29 19:16 ` Thomas Petazzoni via buildroot
1 sibling, 0 replies; 3+ messages in thread
From: Thomas Petazzoni via buildroot @ 2022-08-29 19:16 UTC (permalink / raw)
To: Yann E. MORIN; +Cc: Vincent Fazio, buildroot
On Thu, 25 Aug 2022 22:05:10 +0200
"Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
> To report usable tracebacks, pyc files embed the path of the original py
> files, so that users can more easily try and debug the reported issue.
>
> We genrate the pyc files by calling the python3-supplied compileall
> script, to scan the directry where python modules are installed. Since
> this is done on the build machine, we tell compileall.py to strip away
> the TARGET_DIR prefix, as that has no meaning at runtime.
>
> However, compileall.py forgets [0] to keep a leading / in the front of
> the paths, thus generating non-rooted paths., e.g.:
> /path/buildroot.ouput/targt/usr/lib/python3.10/argparse.py
> gets embedded as:
> usr/lib/python3.10/argparse.py
>
> This is a bit confusing but, as far as we could see, should be mostly be
> used for display purposes in tracebacks, and does not seem to impact
> actual functkionality.
>
> We fix that by instructing compileall.py that the embeeded paths should
> be rooted to / which generates proper paths in tracebacks.
>
> And alternate solution would be to swith gears, and tell compileall.py
> exactly the resulting runtime "base" directory, wihch replaces the
> stripping and prefixing; i.e. it's either:
> -s $(TARGET_DIR) -p /
> or
> -d /usr/lib/python$(PYTHON3_MAJOR_VERSION)
>
> We choose to keep the first solution, because that is semantically what
> we really want to do: to strip the leading build-time path, rather than
> to force anything.
>
> Note: the python test-suite was executed with both solutions (in a
> pyc-only setup), and the results were exactly the same; so in practice,
> -d or -s+-p yield the same results.
>
> Many thanks go to Vincent for reporting the issue and suggesting the
> solutions.
>
> Reported-by: Vincent Fazio <vfazio@xes-inc.com>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> ---
> package/python3/python3.mk | 1 +
> 1 file changed, 1 insertion(+)
Applied to master, thanks.
Thomas
--
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2022-08-29 19:16 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2022-08-25 20:05 [Buildroot] [PATCH] package/python3: ensure pyc embed rooted paths Yann E. MORIN
2022-08-25 20:09 ` Yann E. MORIN
2022-08-29 19:16 ` Thomas Petazzoni via buildroot
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox