[Buildroot] [git commit] package/pkg-python: invalidate precompiled _sysconfigdata*.pyc

[Buildroot] [git commit] package/pkg-python: invalidate precompiled _sysconfigdata*.pyc

[Yann E. MORIN]

commit: https://git.buildroot.net/buildroot/commit/?id=836528f03eb89fdc64432f7a7470145ddf553b8c
branch: https://git.buildroot.net/buildroot/commit/?id=refs/heads/master

For per-package directories, we fixup the _sysconfigdata*.py files, so
that they get proper path pointing to the current package's direcotry
structure.

However, the corresponding, pre-compiled blobs _sysconfigdata*.pyc were
left around, and thus are inconsistent with their source. They might
also be regenerated when a package would install a python module; this
regeneration would trigger the soon-to-be-introduced overwrite
detection.

This commit simply removes _sysconfigdata*.pyc files; they will anyway
be regenerated by the PYTHON{,3}_CREATE_PYC_FILES target finalize hooks.
This is an efficient way to guarantee the consistency between the source
and precompiled versions, and to not trigger the overwrite detection.

Signed-off-by: Herve Codina <herve.codina@bootlin.com>
[yann.morin.1998 at free.frs: reword the commit log]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 package/pkg-python.mk | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/package/pkg-python.mk b/package/pkg-python.mk
index 59a48e5a87..1e4fd5ba33 100644
--- a/package/pkg-python.mk
+++ b/package/pkg-python.mk
@@ -97,6 +97,8 @@ define PKG_PYTHON_FIXUP_SYSCONFIGDATA
 	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
 		-name "_sysconfigdata*.py" | xargs --no-run-if-empty \
 		$(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g"
+	find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
+		-name "_sysconfigdata*.pyc" -delete
 endef
 endif
 

[Buildroot] [git commit] package/pkg-python: invalidate precompiled _sysconfigdata*.pyc

[James Hilliard]

This seems to break host-python3-setuptools, see:
http://autobuild.buildroot.net/results/860a188bd270c59b1fee2f56f31e73689f0e4979/build-end.log

Reverting this change fixes that error.

On Tue, Jul 6, 2021 at 3:13 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
>
> commit: https://git.buildroot.net/buildroot/commit/?id=836528f03eb89fdc64432f7a7470145ddf553b8c
> branch: https://git.buildroot.net/buildroot/commit/?id=refs/heads/master
>
> For per-package directories, we fixup the _sysconfigdata*.py files, so
> that they get proper path pointing to the current package's direcotry
> structure.
>
> However, the corresponding, pre-compiled blobs _sysconfigdata*.pyc were
> left around, and thus are inconsistent with their source. They might
> also be regenerated when a package would install a python module; this
> regeneration would trigger the soon-to-be-introduced overwrite
> detection.
>
> This commit simply removes _sysconfigdata*.pyc files; they will anyway
> be regenerated by the PYTHON{,3}_CREATE_PYC_FILES target finalize hooks.
> This is an efficient way to guarantee the consistency between the source
> and precompiled versions, and to not trigger the overwrite detection.
>
> Signed-off-by: Herve Codina <herve.codina@bootlin.com>
> [yann.morin.1998 at free.frs: reword the commit log]
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> ---
>  package/pkg-python.mk | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/package/pkg-python.mk b/package/pkg-python.mk
> index 59a48e5a87..1e4fd5ba33 100644
> --- a/package/pkg-python.mk
> +++ b/package/pkg-python.mk
> @@ -97,6 +97,8 @@ define PKG_PYTHON_FIXUP_SYSCONFIGDATA
>         find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
>                 -name "_sysconfigdata*.py" | xargs --no-run-if-empty \
>                 $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g"
> +       find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
> +               -name "_sysconfigdata*.pyc" -delete
>  endef
>  endif
>
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

[Buildroot] [git commit] package/pkg-python: invalidate precompiled _sysconfigdata*.pyc

[Yann E. MORIN]

Herv?, James, All,

On 2021-07-08 04:16 -0600, James Hilliard spake thusly:
> This seems to break host-python3-setuptools, see:
> http://autobuild.buildroot.net/results/860a188bd270c59b1fee2f56f31e73689f0e4979/build-end.log
> 
> Reverting this change fixes that error.
> 
> On Tue, Jul 6, 2021 at 3:13 PM Yann E. MORIN <yann.morin.1998@free.fr> wrote:
[--SNIP--]
> > Signed-off-by: Herve Codina <herve.codina@bootlin.com>
> > [yann.morin.1998 at free.frs: reword the commit log]
> > Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
[--SNIP--]
> > diff --git a/package/pkg-python.mk b/package/pkg-python.mk
> > index 59a48e5a87..1e4fd5ba33 100644
> > --- a/package/pkg-python.mk
> > +++ b/package/pkg-python.mk
> > @@ -97,6 +97,8 @@ define PKG_PYTHON_FIXUP_SYSCONFIGDATA
> >         find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
> >                 -name "_sysconfigdata*.py" | xargs --no-run-if-empty \
> >                 $(SED) "s:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g"
> > +       find $(HOST_DIR)/lib/python* $(STAGING_DIR)/usr/lib/python* \
> > +               -name "_sysconfigdata*.pyc" -delete

So at first, that this commit introduces a regression was a bit
puzzling. Indeed, the same find command exists just three lines above,
so certainly they should have failed too.

And indeed, they do fail: the find error is visible twice in the build
error reported above.

However, the error code is silently ignored, because find is on the LHS
of a pipe.

So, this had been somewhat borked for a long time, but we jsut did not
notice, up until now that we do the exact same find, but not in a pipe.

And I see that in the previous iteration, that new find was a LHS of a
pipe too, piped into xargs, so that you did not have the issue in the
first iteration...

So, I think the simplest solution is that I revert this patch for now,
and that you take it back in your series and fix it with the proposal
laid out in the following patch (8/18), probably by reorganising the
series slightly...

Regards,
Yann E. MORIN.

> >  endef
> >  endif
> >
> > _______________________________________________
> > buildroot mailing list
> > buildroot at busybox.net
> > http://lists.busybox.net/mailman/listinfo/buildroot
> _______________________________________________
> buildroot mailing list
> buildroot at busybox.net
> http://lists.busybox.net/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [git commit] package/pkg-python: invalidate precompiled _sysconfigdata*.pyc

[Yann E. MORIN]

commit: https://git.buildroot.net/buildroot/commit/?id=afde8fb8f36267e7b025c26e700f76e9bb75706f
branch: https://git.buildroot.net/buildroot/commit/?id=refs/heads/master

For per-package directories, we fixup the _sysconfigdata*.py files, so
that they get proper path pointing to the current package's direcotry
structure.

However, the corresponding, pre-compiled blobs _sysconfigdata*.pyc were
left around, and thus are inconsistent with their source. They might
also be regenerated when a package would install a python module; this
regeneration would trigger the soon-to-be-introduced overwrite
detection.

This commit simply removes _sysconfigdata*.pyc files; they will anyway
be regenerated by the PYTHON{,3}_CREATE_PYC_FILES target finalize hooks.
This is an efficient way to guarantee the consistency between the source
and precompiled versions, and to not trigger the overwrite detection.

Signed-off-by: Herve Codina <herve.codina@bootlin.com>
[yann.morin.1998@free.frs: reword the commit log]
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
---
 package/pkg-python.mk | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/package/pkg-python.mk b/package/pkg-python.mk
index 8b24aeb3b1..ddf9e733fd 100644
--- a/package/pkg-python.mk
+++ b/package/pkg-python.mk
@@ -108,7 +108,9 @@ define PKG_PYTHON_FIXUP_SYSCONFIGDATA
 		\(    -path '$(HOST_DIR)/lib/python*' \
 		   -o -path '$(STAGING_DIR)/usr/lib/python*' \
 		\) \
-		-name "_sysconfigdata*.py" -print0 \
+		\(    \( -name "_sysconfigdata*.pyc" -delete \) \
+		   -o \( -name "_sysconfigdata*.py" -print0 \) \
+		\) \
 	| xargs -0 --no-run-if-empty \
 		$(SED) 's:$(PER_PACKAGE_DIR)/[^/]\+/:$(PER_PACKAGE_DIR)/$($(PKG)_NAME)/:g'
 endef
_______________________________________________
buildroot mailing list
buildroot@lists.buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot