* [Buildroot] [PATCH v3 1/2] package/pkg-python: use pyinstaller.py for host python packages @ 2022-08-22 7:54 James Hilliard 2022-08-22 7:54 ` [Buildroot] [PATCH v3 2/2] package/pkg-python: clean conflicting pep517 packages before install James Hilliard 2022-08-22 9:55 ` [Buildroot] [PATCH v3 1/2] package/pkg-python: use pyinstaller.py for host python packages Yann E. MORIN 0 siblings, 2 replies; 4+ messages in thread From: James Hilliard @ 2022-08-22 7:54 UTC (permalink / raw) To: buildroot; +Cc: James Hilliard, Marcus Hoffmann, Thomas Petazzoni The python installer cli isn't able to overwrite files of packages that already exist, this causes problems when doing a rebuild or update without a full clean. Since we need to add functionality to our pyinstaller.py script to fix this issue we must also use pyinstaller.py for host python packages. Signed-off-by: James Hilliard <james.hilliard1@gmail.com> Tested-by: Marcus Hoffmann <marcus.hoffmann@othermo.de> --- Changes v2 -> v3: - set interpreter to $(HOST_DIR)/bin/python --- package/pkg-python.mk | 12 ++++++++++-- support/scripts/pyinstaller.py | 0 2 files changed, 10 insertions(+), 2 deletions(-) mode change 100755 => 100644 support/scripts/pyinstaller.py diff --git a/package/pkg-python.mk b/package/pkg-python.mk index 5794e3a195..c15a168f60 100644 --- a/package/pkg-python.mk +++ b/package/pkg-python.mk @@ -139,6 +139,14 @@ PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \ HOST_PKG_PYTHON_PEP517_ENV = \ $(HOST_PKG_PYTHON_ENV) +HOST_PKG_PYTHON_PEP517_INSTALL_OPTS = \ + --interpreter=$(HOST_DIR)/bin/python \ + --script-kind=posix \ + --purelib=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages \ + --headers=$(HOST_DIR)/include/python$(PYTHON3_VERSION_MAJOR) \ + --scripts=$(HOST_DIR)/bin \ + --data=$(HOST_DIR) + HOST_PKG_PYTHON_PEP517_BOOTSTRAP_INSTALL_OPTS = \ --installdir=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages @@ -200,7 +208,7 @@ $(2)_BASE_INSTALL_STAGING_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* else $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV) $(2)_BASE_BUILD_CMD = -m build -n -w -$(2)_BASE_INSTALL_CMD = -m installer dist/* +$(2)_BASE_INSTALL_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS) endif else ifeq ($$($(2)_SETUP_TYPE),flit-bootstrap) ifeq ($(4),target) @@ -208,7 +216,7 @@ $$(error flit-bootstrap setup type only supported for host packages) else $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV) $(2)_BASE_BUILD_CMD = -m flit_core.wheel -$(2)_BASE_INSTALL_CMD ?= -m installer dist/* +$(2)_BASE_INSTALL_CMD ?= $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS) endif else $$(error "Invalid $(2)_SETUP_TYPE. Valid options are 'distutils', 'setuptools', 'pep517' or 'flit'.") diff --git a/support/scripts/pyinstaller.py b/support/scripts/pyinstaller.py old mode 100755 new mode 100644 -- 2.34.1 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 4+ messages in thread
* [Buildroot] [PATCH v3 2/2] package/pkg-python: clean conflicting pep517 packages before install 2022-08-22 7:54 [Buildroot] [PATCH v3 1/2] package/pkg-python: use pyinstaller.py for host python packages James Hilliard @ 2022-08-22 7:54 ` James Hilliard 2022-08-22 9:57 ` Yann E. MORIN 2022-08-22 9:55 ` [Buildroot] [PATCH v3 1/2] package/pkg-python: use pyinstaller.py for host python packages Yann E. MORIN 1 sibling, 1 reply; 4+ messages in thread From: James Hilliard @ 2022-08-22 7:54 UTC (permalink / raw) To: buildroot; +Cc: James Hilliard, Marcus Hoffmann, Thomas Petazzoni The python installer package isn't able to overwrite files of packges that already exist, this causes problems when doing a rebuild or update without a full clean. To fix this we can use functionality from importlib to identify and remove any conflicting python package files before installation. Fixes: Traceback (most recent call last): File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 69, in <module> main() File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 61, in main install( File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/_core.py", line 109, in install record = destination.write_file( File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 207, in write_file return self.write_to_fs(scheme, path_, stream, is_executable) File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 167, in write_to_fs raise FileExistsError(message) FileExistsError: File already exists: /home/buildroot/buildroot/output/target/usr/lib/python3.10/site-packages/tinycss2/__init__.py Signed-off-by: James Hilliard <james.hilliard1@gmail.com> Tested-by: Marcus Hoffmann <marcus.hoffmann@othermo.de> --- Changes v2 -> v3: - split out host pyinstaller.py change Changes v1 -> v2: - remove unused os import --- support/scripts/pyinstaller.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/support/scripts/pyinstaller.py b/support/scripts/pyinstaller.py index 6dd9242327..d9b77cac19 100644 --- a/support/scripts/pyinstaller.py +++ b/support/scripts/pyinstaller.py @@ -2,12 +2,33 @@ import argparse import glob +import pathlib + +from importlib.machinery import PathFinder +from importlib.metadata import DistributionFinder from installer import install +from installer._core import _process_WHEEL_file from installer.destinations import SchemeDictionaryDestination from installer.sources import WheelFile +def clean(source, destination): + scheme = _process_WHEEL_file(source) + scheme_path = destination.scheme_dict[scheme] + context = DistributionFinder.Context( + name=source.distribution, + path=[scheme_path], + ) + for path in PathFinder.find_distributions(context=context): + if not path.files: + continue + for file in path.files: + file_path = pathlib.Path(file.locate()) + if file_path.exists(): + file_path.unlink() + + def main(): """Entry point for CLI.""" ap = argparse.ArgumentParser("python pyinstaller.py") @@ -58,6 +79,7 @@ def main(): ) with WheelFile.open(glob.glob(args.wheel_file)[0]) as source: + clean(source, destination) install( source=source, destination=destination, -- 2.34.1 _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [Buildroot] [PATCH v3 2/2] package/pkg-python: clean conflicting pep517 packages before install 2022-08-22 7:54 ` [Buildroot] [PATCH v3 2/2] package/pkg-python: clean conflicting pep517 packages before install James Hilliard @ 2022-08-22 9:57 ` Yann E. MORIN 0 siblings, 0 replies; 4+ messages in thread From: Yann E. MORIN @ 2022-08-22 9:57 UTC (permalink / raw) To: James Hilliard; +Cc: Marcus Hoffmann, Thomas Petazzoni, buildroot James, All, On 2022-08-22 01:54 -0600, James Hilliard spake thusly: > The python installer package isn't able to overwrite files of packges > that already exist, this causes problems when doing a rebuild or > update without a full clean. > > To fix this we can use functionality from importlib to identify and > remove any conflicting python package files before installation. > > Fixes: > Traceback (most recent call last): > File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 69, in <module> > main() > File "/home/buildroot/buildroot/support/scripts/pyinstaller.py", line 61, in main > install( > File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/_core.py", line 109, in install > record = destination.write_file( > File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 207, in write_file > return self.write_to_fs(scheme, path_, stream, is_executable) > File "/home/buildroot/buildroot/output/host/lib/python3.10/site-packages/installer/destinations.py", line 167, in write_to_fs > raise FileExistsError(message) > FileExistsError: File already exists: /home/buildroot/buildroot/output/target/usr/lib/python3.10/site-packages/tinycss2/__init__.py > > Signed-off-by: James Hilliard <james.hilliard1@gmail.com> > Tested-by: Marcus Hoffmann <marcus.hoffmann@othermo.de> > --- > Changes v2 -> v3: > - split out host pyinstaller.py change > Changes v1 -> v2: > - remove unused os import > --- > support/scripts/pyinstaller.py | 22 ++++++++++++++++++++++ > 1 file changed, 22 insertions(+) > > diff --git a/support/scripts/pyinstaller.py b/support/scripts/pyinstaller.py > index 6dd9242327..d9b77cac19 100644 > --- a/support/scripts/pyinstaller.py > +++ b/support/scripts/pyinstaller.py > @@ -2,12 +2,33 @@ > > import argparse > import glob > +import pathlib > + > +from importlib.machinery import PathFinder > +from importlib.metadata import DistributionFinder > > from installer import install > +from installer._core import _process_WHEEL_file I've extended the commit log to explain why we need to use private symbols. > from installer.destinations import SchemeDictionaryDestination > from installer.sources import WheelFile > > > +def clean(source, destination): > + scheme = _process_WHEEL_file(source) > + scheme_path = destination.scheme_dict[scheme] > + context = DistributionFinder.Context( > + name=source.distribution, > + path=[scheme_path], > + ) > + for path in PathFinder.find_distributions(context=context): > + if not path.files: I've changed this for an explicit test against None. Applied to master, thanks. Regards, Yann E. MORIN. > + continue > + for file in path.files: > + file_path = pathlib.Path(file.locate()) > + if file_path.exists(): > + file_path.unlink() > + > + > def main(): > """Entry point for CLI.""" > ap = argparse.ArgumentParser("python pyinstaller.py") > @@ -58,6 +79,7 @@ def main(): > ) > > with WheelFile.open(glob.glob(args.wheel_file)[0]) as source: > + clean(source, destination) > install( > source=source, > destination=destination, > -- > 2.34.1 > > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Buildroot] [PATCH v3 1/2] package/pkg-python: use pyinstaller.py for host python packages 2022-08-22 7:54 [Buildroot] [PATCH v3 1/2] package/pkg-python: use pyinstaller.py for host python packages James Hilliard 2022-08-22 7:54 ` [Buildroot] [PATCH v3 2/2] package/pkg-python: clean conflicting pep517 packages before install James Hilliard @ 2022-08-22 9:55 ` Yann E. MORIN 1 sibling, 0 replies; 4+ messages in thread From: Yann E. MORIN @ 2022-08-22 9:55 UTC (permalink / raw) To: James Hilliard; +Cc: Marcus Hoffmann, Thomas Petazzoni, buildroot James, All, On 2022-08-22 01:54 -0600, James Hilliard spake thusly: > The python installer cli isn't able to overwrite files of packages > that already exist, this causes problems when doing a rebuild or > update without a full clean. > > Since we need to add functionality to our pyinstaller.py script to fix > this issue we must also use pyinstaller.py for host python packages. > > Signed-off-by: James Hilliard <james.hilliard1@gmail.com> > Tested-by: Marcus Hoffmann <marcus.hoffmann@othermo.de> Applied to master, thanks. Regards, Yann E. MORIN. > --- > Changes v2 -> v3: > - set interpreter to $(HOST_DIR)/bin/python > --- > package/pkg-python.mk | 12 ++++++++++-- > support/scripts/pyinstaller.py | 0 > 2 files changed, 10 insertions(+), 2 deletions(-) > mode change 100755 => 100644 support/scripts/pyinstaller.py > > diff --git a/package/pkg-python.mk b/package/pkg-python.mk > index 5794e3a195..c15a168f60 100644 > --- a/package/pkg-python.mk > +++ b/package/pkg-python.mk > @@ -139,6 +139,14 @@ PKG_PYTHON_PEP517_INSTALL_STAGING_OPTS = \ > HOST_PKG_PYTHON_PEP517_ENV = \ > $(HOST_PKG_PYTHON_ENV) > > +HOST_PKG_PYTHON_PEP517_INSTALL_OPTS = \ > + --interpreter=$(HOST_DIR)/bin/python \ > + --script-kind=posix \ > + --purelib=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages \ > + --headers=$(HOST_DIR)/include/python$(PYTHON3_VERSION_MAJOR) \ > + --scripts=$(HOST_DIR)/bin \ > + --data=$(HOST_DIR) > + > HOST_PKG_PYTHON_PEP517_BOOTSTRAP_INSTALL_OPTS = \ > --installdir=$(HOST_DIR)/lib/python$(PYTHON3_VERSION_MAJOR)/site-packages > > @@ -200,7 +208,7 @@ $(2)_BASE_INSTALL_STAGING_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* > else > $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV) > $(2)_BASE_BUILD_CMD = -m build -n -w > -$(2)_BASE_INSTALL_CMD = -m installer dist/* > +$(2)_BASE_INSTALL_CMD = $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS) > endif > else ifeq ($$($(2)_SETUP_TYPE),flit-bootstrap) > ifeq ($(4),target) > @@ -208,7 +216,7 @@ $$(error flit-bootstrap setup type only supported for host packages) > else > $(2)_BASE_ENV = $$(HOST_PKG_PYTHON_PEP517_ENV) > $(2)_BASE_BUILD_CMD = -m flit_core.wheel > -$(2)_BASE_INSTALL_CMD ?= -m installer dist/* > +$(2)_BASE_INSTALL_CMD ?= $(TOPDIR)/support/scripts/pyinstaller.py dist/* $$(HOST_PKG_PYTHON_PEP517_INSTALL_OPTS) > endif > else > $$(error "Invalid $(2)_SETUP_TYPE. Valid options are 'distutils', 'setuptools', 'pep517' or 'flit'.") > diff --git a/support/scripts/pyinstaller.py b/support/scripts/pyinstaller.py > old mode 100755 > new mode 100644 > -- > 2.34.1 > > _______________________________________________ > buildroot mailing list > buildroot@buildroot.org > https://lists.buildroot.org/mailman/listinfo/buildroot -- .-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list buildroot@buildroot.org https://lists.buildroot.org/mailman/listinfo/buildroot ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2022-08-22 9:57 UTC | newest] Thread overview: 4+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2022-08-22 7:54 [Buildroot] [PATCH v3 1/2] package/pkg-python: use pyinstaller.py for host python packages James Hilliard 2022-08-22 7:54 ` [Buildroot] [PATCH v3 2/2] package/pkg-python: clean conflicting pep517 packages before install James Hilliard 2022-08-22 9:57 ` Yann E. MORIN 2022-08-22 9:55 ` [Buildroot] [PATCH v3 1/2] package/pkg-python: use pyinstaller.py for host python packages Yann E. MORIN
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.