* Re: [Buildroot] [PATCH] Makefile: run PPD and RPATH fixup in host-fialize
2024-06-09 7:54 [Buildroot] [PATCH] Makefile: run PPD and RPATH fixup in host-fialize Yann E. MORIN
@ 2024-06-12 17:01 ` Brandon Maier via buildroot
2024-09-06 2:55 ` TIAN Yuanhao
2024-09-07 19:35 ` Yann E. MORIN
2 siblings, 0 replies; 5+ messages in thread
From: Brandon Maier via buildroot @ 2024-06-12 17:01 UTC (permalink / raw)
To: Yann E. MORIN; +Cc: Thomas Petazzoni, buildroot
On 2024.06.09 09:54, Yann E. MORIN wrote:
> Currently, the gdbinit is generated and installed during post install
> hooks, either from the gdb package, or from the external-toolchain
> package. When using per-package directories (PPD), the staging directory
> of the either package is stored in the generated gdbinit, which is not
> going to be valid when all the PPD staging dirs get merged into the
> final staging: it would lack any library installed afterwards, i.e.
> mostly everything would be missing (but the libraries from the C
> toolchain in the case of an external toolchain).
>
> Similarly, all the RPATH will point to various PPD drectories. This
> does not cause any issue when the final host is aggregated, because the
> PPD directories still exist when we call programs from there (e.g. from
> the fs infra, or from post-image scripts).
>
> However, we knew that would not always be possible to keep the PPD
> directories: we have the prepare-sdk rule that runs a cleanup pass on
> the RPATH, and also applies the generic PPD fixups.
>
> When we introduced prepare-sdk in c32ad51cbf2a (core/sdk: generate the
> SDK tarball ourselves), we did not yet have support for PPD for the host
> directory, and especially, we did not have the host-finalize rule, which
> was only introduced in d0f4f95e390b (Makefile: rework main directory
> creation logic) which kick-started the introduction of PPD.
>
> At that point, we did not realise that the rpath fixups from
> prepare-sdk, would be better moved to the new host-finalize rule,
> because that had no impact unless one would need an SDK.
>
> Later, in 25e60fbe1cab (Makefile: fix SDK relocation for
> per-package-dirs), we eventually introduced the PPD generic fixups in
> the prepare-sdk rule. Again, we did not realise that those fixups would
> be better placed in the host-finalize rule rather than the prepare-sdk.
>
> While fixing the RPATH in host-finalize is not critical, fixing up the
> PPD paths actually is, as the gdbinit case demonstrate.
>
> As such, move the PPD fixups to the host-finalize step, and while at it,
> also move the RPATH fixups.
>
> This now does not leave much to do in the prepare-sdk step, and that
> could very well be moved to the host-finalize rule as well. However,
> some people may have started to rely on prepare-sdk in its 6 years of
> existence, and the little script it installs is not needed unless one
> really needs an SDK. So leave it as it is for now.
>
> Reported-by: Casey Reeves <casey@xogium.me>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Tested-by: Casey Reeves <casey@xogium.me>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Brandon Maier <Brandon.Maier@collins.com>
> ---
> Makefile | 16 ++++++++--------
> 1 file changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index 3bef434466..6c2a2ccddb 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -594,14 +594,7 @@ world: target-post-image
>
> .PHONY: prepare-sdk
> prepare-sdk: world
> - @$(call MESSAGE,"Rendering the SDK relocatable")
> - PARALLEL_JOBS=$(PARALLEL_JOBS) \
> - PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> - $(TOPDIR)/support/scripts/fix-rpath host
> - PARALLEL_JOBS=$(PARALLEL_JOBS) \
> - PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> - $(TOPDIR)/support/scripts/fix-rpath staging
> - $(call ppd-fixup-paths,$(BASE_DIR))
> + @$(call MESSAGE,"Preparing the SDK")
> $(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
> mkdir -p $(HOST_DIR)/share/buildroot
> echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
> @@ -719,6 +712,13 @@ STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.t
> host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK)
> @$(call MESSAGE,"Finalizing host directory")
> $(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR),copy)
> + $(Q)PARALLEL_JOBS=$(PARALLEL_JOBS) \
> + PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> + $(TOPDIR)/support/scripts/fix-rpath host
> + $(Q)PARALLEL_JOBS=$(PARALLEL_JOBS) \
> + PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> + $(TOPDIR)/support/scripts/fix-rpath staging
> + $(call ppd-fixup-paths,$(BASE_DIR))
>
> .PHONY: staging-finalize
> staging-finalize: $(STAGING_DIR_SYMLINK)
> --
> 2.45.1
>
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot
This change makes sense. I tested by building some internal boards and
the host directory appears correct.
My only concern is about the final make steps taking too long, as with
per-package-directories doing a simple `make <pkg>-rebuild all` can take
a couple extra minutes. In my test build, the added time for a large
build with 200+ packages is about 15 seconds, which isn't too bad. It
might be possible to optimize this later.
Tested-by: Brandon Maier <brandon.maier@collins.com>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Buildroot] [PATCH] Makefile: run PPD and RPATH fixup in host-fialize
2024-06-09 7:54 [Buildroot] [PATCH] Makefile: run PPD and RPATH fixup in host-fialize Yann E. MORIN
2024-06-12 17:01 ` Brandon Maier via buildroot
@ 2024-09-06 2:55 ` TIAN Yuanhao
2024-09-07 19:35 ` Yann E. MORIN
2 siblings, 0 replies; 5+ messages in thread
From: TIAN Yuanhao @ 2024-09-06 2:55 UTC (permalink / raw)
To: Yann E. MORIN; +Cc: James Hilliard, Thomas Petazzoni, Brandon Maier, buildroot
At 2024-06-09 15:54:44, "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
>Currently, the gdbinit is generated and installed during post install
>hooks, either from the gdb package, or from the external-toolchain
>package. When using per-package directories (PPD), the staging directory
>of the either package is stored in the generated gdbinit, which is not
>going to be valid when all the PPD staging dirs get merged into the
>final staging: it would lack any library installed afterwards, i.e.
>mostly everything would be missing (but the libraries from the C
>toolchain in the case of an external toolchain).
>
>Similarly, all the RPATH will point to various PPD drectories. This
>does not cause any issue when the final host is aggregated, because the
>PPD directories still exist when we call programs from there (e.g. from
>the fs infra, or from post-image scripts).
>
>However, we knew that would not always be possible to keep the PPD
>directories: we have the prepare-sdk rule that runs a cleanup pass on
>the RPATH, and also applies the generic PPD fixups.
>
>When we introduced prepare-sdk in c32ad51cbf2a (core/sdk: generate the
>SDK tarball ourselves), we did not yet have support for PPD for the host
>directory, and especially, we did not have the host-finalize rule, which
>was only introduced in d0f4f95e390b (Makefile: rework main directory
>creation logic) which kick-started the introduction of PPD.
>
>At that point, we did not realise that the rpath fixups from
>prepare-sdk, would be better moved to the new host-finalize rule,
>because that had no impact unless one would need an SDK.
>
>Later, in 25e60fbe1cab (Makefile: fix SDK relocation for
>per-package-dirs), we eventually introduced the PPD generic fixups in
>the prepare-sdk rule. Again, we did not realise that those fixups would
>be better placed in the host-finalize rule rather than the prepare-sdk.
>
>While fixing the RPATH in host-finalize is not critical, fixing up the
>PPD paths actually is, as the gdbinit case demonstrate.
>
>As such, move the PPD fixups to the host-finalize step, and while at it,
>also move the RPATH fixups.
>
>This now does not leave much to do in the prepare-sdk step, and that
>could very well be moved to the host-finalize rule as well. However,
>some people may have started to rely on prepare-sdk in its 6 years of
>existence, and the little script it installs is not needed unless one
>really needs an SDK. So leave it as it is for now.
>
>Reported-by: Casey Reeves <casey@xogium.me>
>Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
>Tested-by: Casey Reeves <casey@xogium.me>
>Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
>Cc: Brandon Maier <Brandon.Maier@collins.com>
>---
> Makefile | 16 ++++++++--------
> 1 file changed, 8 insertions(+), 8 deletions(-)
>
>diff --git a/Makefile b/Makefile
>index 3bef434466..6c2a2ccddb 100644
>--- a/Makefile
>+++ b/Makefile
>@@ -594,14 +594,7 @@ world: target-post-image
>
> .PHONY: prepare-sdk
> prepare-sdk: world
>- @$(call MESSAGE,"Rendering the SDK relocatable")
>- PARALLEL_JOBS=$(PARALLEL_JOBS) \
>- PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
>- $(TOPDIR)/support/scripts/fix-rpath host
>- PARALLEL_JOBS=$(PARALLEL_JOBS) \
>- PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
>- $(TOPDIR)/support/scripts/fix-rpath staging
>- $(call ppd-fixup-paths,$(BASE_DIR))
>+ @$(call MESSAGE,"Preparing the SDK")
> $(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
> mkdir -p $(HOST_DIR)/share/buildroot
> echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
>@@ -719,6 +712,13 @@ STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.t
> host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK)
> @$(call MESSAGE,"Finalizing host directory")
> $(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR),copy)
>+ $(Q)PARALLEL_JOBS=$(PARALLEL_JOBS) \
>+ PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
>+ $(TOPDIR)/support/scripts/fix-rpath host
>+ $(Q)PARALLEL_JOBS=$(PARALLEL_JOBS) \
>+ PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
>+ $(TOPDIR)/support/scripts/fix-rpath staging
>+ $(call ppd-fixup-paths,$(BASE_DIR))
>
> .PHONY: staging-finalize
> staging-finalize: $(STAGING_DIR_SYMLINK)
>--
>2.45.1
>
I found that my patch [1] was duplicated to this patch, and our situation is similar.
[1]: http://patchwork.ozlabs.org/project/buildroot/patch/20240828124440.3882-1-tianyuanhao3@163.com/
Acked-by: TIAN Yuanhao <tianyuanhao3@163.com>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Buildroot] [PATCH] Makefile: run PPD and RPATH fixup in host-fialize
2024-06-09 7:54 [Buildroot] [PATCH] Makefile: run PPD and RPATH fixup in host-fialize Yann E. MORIN
2024-06-12 17:01 ` Brandon Maier via buildroot
2024-09-06 2:55 ` TIAN Yuanhao
@ 2024-09-07 19:35 ` Yann E. MORIN
2024-09-19 7:14 ` Peter Korsgaard
2 siblings, 1 reply; 5+ messages in thread
From: Yann E. MORIN @ 2024-09-07 19:35 UTC (permalink / raw)
To: buildroot; +Cc: Brandon Maier, Thomas Petazzoni
All,
On 2024-06-09 09:54 +0200, Yann E. MORIN spake thusly:
[--SNIP--]
> As such, move the PPD fixups to the host-finalize step, and while at it,
> also move the RPATH fixups.
[--SNIP--]
> Reported-by: Casey Reeves <casey@xogium.me>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Tested-by: Casey Reeves <casey@xogium.me>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Brandon Maier <Brandon.Maier@collins.com>
Applied to master, thanks.
Regards,
Yann E. MORIN.
> ---
> Makefile | 16 ++++++++--------
> 1 file changed, 8 insertions(+), 8 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index 3bef434466..6c2a2ccddb 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -594,14 +594,7 @@ world: target-post-image
>
> .PHONY: prepare-sdk
> prepare-sdk: world
> - @$(call MESSAGE,"Rendering the SDK relocatable")
> - PARALLEL_JOBS=$(PARALLEL_JOBS) \
> - PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> - $(TOPDIR)/support/scripts/fix-rpath host
> - PARALLEL_JOBS=$(PARALLEL_JOBS) \
> - PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> - $(TOPDIR)/support/scripts/fix-rpath staging
> - $(call ppd-fixup-paths,$(BASE_DIR))
> + @$(call MESSAGE,"Preparing the SDK")
> $(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
> mkdir -p $(HOST_DIR)/share/buildroot
> echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
> @@ -719,6 +712,13 @@ STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.t
> host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK)
> @$(call MESSAGE,"Finalizing host directory")
> $(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR),copy)
> + $(Q)PARALLEL_JOBS=$(PARALLEL_JOBS) \
> + PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> + $(TOPDIR)/support/scripts/fix-rpath host
> + $(Q)PARALLEL_JOBS=$(PARALLEL_JOBS) \
> + PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> + $(TOPDIR)/support/scripts/fix-rpath staging
> + $(call ppd-fixup-paths,$(BASE_DIR))
>
> .PHONY: staging-finalize
> staging-finalize: $(STAGING_DIR_SYMLINK)
> --
> 2.45.1
>
--
.-----------------.--------------------.------------------.--------------------.
| Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ |
| +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Buildroot] [PATCH] Makefile: run PPD and RPATH fixup in host-fialize
2024-09-07 19:35 ` Yann E. MORIN
@ 2024-09-19 7:14 ` Peter Korsgaard
0 siblings, 0 replies; 5+ messages in thread
From: Peter Korsgaard @ 2024-09-19 7:14 UTC (permalink / raw)
To: Yann E. MORIN; +Cc: Thomas Petazzoni, Brandon Maier, buildroot
>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:
> All,
> On 2024-06-09 09:54 +0200, Yann E. MORIN spake thusly:
> [--SNIP--]
>> As such, move the PPD fixups to the host-finalize step, and while at it,
>> also move the RPATH fixups.
> [--SNIP--]
>> Reported-by: Casey Reeves <casey@xogium.me>
>> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
>> Tested-by: Casey Reeves <casey@xogium.me>
>> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
>> Cc: Brandon Maier <Brandon.Maier@collins.com>
> Applied to master, thanks.
Committed to 2024.02.x and 2024.08.x, thanks.
--
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot
^ permalink raw reply [flat|nested] 5+ messages in thread