[Buildroot] [PATCH v3 1/2] package/pkg-python: use pyinstaller.py for host python packages

[Buildroot] [PATCH v3 1/2] package/pkg-python: use pyinstaller.py for host python packages

[James Hilliard]

The python installer cli isn't able to overwrite files of packages
that already exist, this causes problems when doing a rebuild or
update without a full clean.

Since we need to add functionality to our pyinstaller.py script to fix
this issue we must also use pyinstaller.py for host python packages.

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Tested-by: Marcus Hoffmann <marcus.hoffmann@othermo.de>
---
Changes v2 -> v3:
  - set interpreter to $(HOST_DIR)/bin/python
---
 package/pkg-python.mk          | 12 ++++++++++--
 support/scripts/pyinstaller.py |  0
 2 files changed, 10 insertions(+), 2 deletions(-)
 mode change 100755 => 100644 support/scripts/pyinstaller.py

diff --git a/package/pkg-python.mk b/package/pkg-python.mk
index 5794e3a195..c15a168f60 100644
--- a/package/pkg-python.mk
+++ b/package/pkg-python.mk
@@ -139,6 +139,14 @@ PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \
 HOST_PKG_PYTHON_PEP517_ENV = \
 	$(HOST_PKG_PYTHON_ENV)
 
+HOST_PKG_PYTHON_PEP517_INSTALL_OPTS = \
+	--interpreter=$(HOST_DIR)/bin/python \
+	--script-kind=posix \
+	--purelib=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages \
+	--headers=$(HOST_DIR)/include/python$(PYTHON3_VERSION_MAJOR) \
+	--scripts=$(HOST_DIR)/bin \
+	--data=$(HOST_DIR)
+
 HOST_PKG_PYTHON_PEP517_BOOTSTRAP_INSTALL_OPTS = \
 	--installdir=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages
 
@@ -200,7 +208,7 @@ $(2)_BASE_INSTALL_STAGING_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/*
 else
 $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV)
 $(2)_BASE_BUILD_CMD = -m build -n -w
-$(2)_BASE_INSTALL_CMD = -m installer dist/*
+$(2)_BASE_INSTALL_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS)
 endif
 else ifeq ($$($(2)_SETUP_TYPE),flit-bootstrap)
 ifeq ($(4),target)
@@ -208,7 +216,7 @@ $$(error flit-bootstrap setup type only supported for host packages)
 else
 $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV)
 $(2)_BASE_BUILD_CMD = -m flit_core.wheel
-$(2)_BASE_INSTALL_CMD ?= -m installer dist/*
+$(2)_BASE_INSTALL_CMD ?= $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS)
 endif
 else
 $$(error "Invalid $(2)_SETUP_TYPE. Valid options are 'distutils', 'setuptools', 'pep517' or 'flit'.")
diff --git a/support/scripts/pyinstaller.py b/support/scripts/pyinstaller.py
old mode 100755
new mode 100644
-- 
2.34.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH v3 2/2] package/pkg-python: clean conflicting pep517 packages before install

[James Hilliard]

The python installer package isn't able to overwrite files of packges
that already exist, this causes problems when doing a rebuild or
update without a full clean.

To fix this we can use functionality from importlib to identify and
remove any conflicting python package files before installation.

Fixes:
Traceback (most recent call last):
  File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 69, in <module>
    main()
  File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 61, in main
    install(
  File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/_core.py", line 109, in install
    record = destination.write_file(
  File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 207, in write_file
    return self.write_to_fs(scheme, path_, stream, is_executable)
  File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 167, in write_to_fs
    raise FileExistsError(message)
FileExistsError: File already exists: /home/buildroot/buildroot/output/target/usr/lib/python3.10/site-packages/tinycss2/__init__.py

Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
Tested-by: Marcus Hoffmann <marcus.hoffmann@othermo.de>
---
Changes v2 -> v3:
  - split out host pyinstaller.py change
Changes v1 -> v2:
  - remove unused os import
---
 support/scripts/pyinstaller.py | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/support/scripts/pyinstaller.py b/support/scripts/pyinstaller.py
index 6dd9242327..d9b77cac19 100644
--- a/support/scripts/pyinstaller.py
+++ b/support/scripts/pyinstaller.py
@@ -2,12 +2,33 @@
 
 import argparse
 import glob
+import pathlib
+
+from importlib.machinery import PathFinder
+from importlib.metadata import DistributionFinder
 
 from installer import install
+from installer._core import _process_WHEEL_file
 from installer.destinations import SchemeDictionaryDestination
 from installer.sources import WheelFile
 
 
+def clean(source, destination):
+    scheme = _process_WHEEL_file(source)
+    scheme_path = destination.scheme_dict[scheme]
+    context = DistributionFinder.Context(
+        name=source.distribution,
+        path=[scheme_path],
+    )
+    for path in PathFinder.find_distributions(context=context):
+        if not path.files:
+            continue
+        for file in path.files:
+            file_path = pathlib.Path(file.locate())
+            if file_path.exists():
+                file_path.unlink()
+
+
 def main():
     """Entry point for CLI."""
     ap = argparse.ArgumentParser("python pyinstaller.py")
@@ -58,6 +79,7 @@ def main():
     )
 
     with WheelFile.open(glob.glob(args.wheel_file)[0]) as source:
+        clean(source, destination)
         install(
             source=source,
             destination=destination,
-- 
2.34.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 1/2] package/pkg-python: use pyinstaller.py for host python packages

[Yann E. MORIN]

James, All,

On 2022-08-22 01:54 -0600, James Hilliard spake thusly:
> The python installer cli isn't able to overwrite files of packages
> that already exist, this causes problems when doing a rebuild or
> update without a full clean.
> 
> Since we need to add functionality to our pyinstaller.py script to fix
> this issue we must also use pyinstaller.py for host python packages.
> 
> Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
> Tested-by: Marcus Hoffmann <marcus.hoffmann@othermo.de>

Applied to master, thanks.

Regards,
Yann E. MORIN.

> ---
> Changes v2 -> v3:
>   - set interpreter to $(HOST_DIR)/bin/python
> ---
>  package/pkg-python.mk          | 12 ++++++++++--
>  support/scripts/pyinstaller.py |  0
>  2 files changed, 10 insertions(+), 2 deletions(-)
>  mode change 100755 => 100644 support/scripts/pyinstaller.py
> 
> diff --git a/package/pkg-python.mk b/package/pkg-python.mk
> index 5794e3a195..c15a168f60 100644
> --- a/package/pkg-python.mk
> +++ b/package/pkg-python.mk
> @@ -139,6 +139,14 @@ PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \
>  HOST_PKG_PYTHON_PEP517_ENV = \
>  	$(HOST_PKG_PYTHON_ENV)
>  
> +HOST_PKG_PYTHON_PEP517_INSTALL_OPTS = \
> +	--interpreter=$(HOST_DIR)/bin/python \
> +	--script-kind=posix \
> +	--purelib=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages \
> +	--headers=$(HOST_DIR)/include/python$(PYTHON3_VERSION_MAJOR) \
> +	--scripts=$(HOST_DIR)/bin \
> +	--data=$(HOST_DIR)
> +
>  HOST_PKG_PYTHON_PEP517_BOOTSTRAP_INSTALL_OPTS = \
>  	--installdir=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages
>  
> @@ -200,7 +208,7 @@ $(2)_BASE_INSTALL_STAGING_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/*
>  else
>  $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV)
>  $(2)_BASE_BUILD_CMD = -m build -n -w
> -$(2)_BASE_INSTALL_CMD = -m installer dist/*
> +$(2)_BASE_INSTALL_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS)
>  endif
>  else ifeq ($$($(2)_SETUP_TYPE),flit-bootstrap)
>  ifeq ($(4),target)
> @@ -208,7 +216,7 @@ $$(error flit-bootstrap setup type only supported for host packages)
>  else
>  $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV)
>  $(2)_BASE_BUILD_CMD = -m flit_core.wheel
> -$(2)_BASE_INSTALL_CMD ?= -m installer dist/*
> +$(2)_BASE_INSTALL_CMD ?= $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS)
>  endif
>  else
>  $$(error "Invalid $(2)_SETUP_TYPE. Valid options are 'distutils', 'setuptools', 'pep517' or 'flit'.")
> diff --git a/support/scripts/pyinstaller.py b/support/scripts/pyinstaller.py
> old mode 100755
> new mode 100644
> -- 
> 2.34.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH v3 2/2] package/pkg-python: clean conflicting pep517 packages before install

[Yann E. MORIN]

James, All,

On 2022-08-22 01:54 -0600, James Hilliard spake thusly:
> The python installer package isn't able to overwrite files of packges
> that already exist, this causes problems when doing a rebuild or
> update without a full clean.
> 
> To fix this we can use functionality from importlib to identify and
> remove any conflicting python package files before installation.
> 
> Fixes:
> Traceback (most recent call last):
>   File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 69, in <module>
>     main()
>   File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 61, in main
>     install(
>   File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/_core.py", line 109, in install
>     record = destination.write_file(
>   File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 207, in write_file
>     return self.write_to_fs(scheme, path_, stream, is_executable)
>   File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 167, in write_to_fs
>     raise FileExistsError(message)
> FileExistsError: File already exists: /home/buildroot/buildroot/output/target/usr/lib/python3.10/site-packages/tinycss2/__init__.py
> 
> Signed-off-by: James Hilliard <james.hilliard1@gmail.com>
> Tested-by: Marcus Hoffmann <marcus.hoffmann@othermo.de>
> ---
> Changes v2 -> v3:
>   - split out host pyinstaller.py change
> Changes v1 -> v2:
>   - remove unused os import
> ---
>  support/scripts/pyinstaller.py | 22 ++++++++++++++++++++++
>  1 file changed, 22 insertions(+)
> 
> diff --git a/support/scripts/pyinstaller.py b/support/scripts/pyinstaller.py
> index 6dd9242327..d9b77cac19 100644
> --- a/support/scripts/pyinstaller.py
> +++ b/support/scripts/pyinstaller.py
> @@ -2,12 +2,33 @@
>  
>  import argparse
>  import glob
> +import pathlib
> +
> +from importlib.machinery import PathFinder
> +from importlib.metadata import DistributionFinder
>  
>  from installer import install
> +from installer._core import _process_WHEEL_file

I've extended the commit log to explain why we need to use private
symbols.

>  from installer.destinations import SchemeDictionaryDestination
>  from installer.sources import WheelFile
>  
>  
> +def clean(source, destination):
> +    scheme = _process_WHEEL_file(source)
> +    scheme_path = destination.scheme_dict[scheme]
> +    context = DistributionFinder.Context(
> +        name=source.distribution,
> +        path=[scheme_path],
> +    )
> +    for path in PathFinder.find_distributions(context=context):
> +        if not path.files:

I've changed this for an explicit test against None.

Applied to master, thanks.

Regards,
Yann E. MORIN.

> +            continue
> +        for file in path.files:
> +            file_path = pathlib.Path(file.locate())
> +            if file_path.exists():
> +                file_path.unlink()
> +
> +
>  def main():
>      """Entry point for CLI."""
>      ap = argparse.ArgumentParser("python pyinstaller.py")
> @@ -58,6 +79,7 @@ def main():
>      )
>  
>      with WheelFile.open(glob.glob(args.wheel_file)[0]) as source:
> +        clean(source, destination)
>          install(
>              source=source,
>              destination=destination,
> -- 
> 2.34.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot