[Buildroot] [PATCH] Makefile: run PPD and RPATH fixup in host-fialize

[Buildroot] [PATCH] Makefile: run PPD and RPATH fixup in host-fialize

[Yann E. MORIN]

Currently, the gdbinit is generated and installed during post install
hooks, either from the gdb package, or from the external-toolchain
package. When using per-package directories (PPD), the staging directory
of the either package is stored in the generated gdbinit, which is not
going to be valid when all the PPD staging dirs get merged into the
final staging: it would lack any library installed afterwards, i.e.
mostly everything would be missing (but the libraries from the C
toolchain in the case of an external toolchain).

Similarly, all the RPATH will point to various PPD drectories. This
does not cause any issue when the final host is aggregated, because the
PPD directories still exist when we call programs from there (e.g. from
the fs infra, or from post-image scripts).

However, we knew that would not always be possible to keep the PPD
directories: we have the prepare-sdk rule that runs a cleanup pass on
the RPATH, and also applies the generic PPD fixups.

When we introduced prepare-sdk in c32ad51cbf2a (core/sdk: generate the
SDK tarball ourselves), we did not yet have support for PPD for the host
directory, and especially, we did not have the host-finalize rule, which
was only introduced in d0f4f95e390b (Makefile: rework main directory
creation logic) which kick-started the introduction of PPD.

At that point, we did not realise that the rpath fixups from
prepare-sdk, would be better moved to the new host-finalize rule,
because that had no impact unless one would need an SDK.

Later, in 25e60fbe1cab (Makefile: fix SDK relocation for
per-package-dirs), we eventually introduced the PPD generic fixups in
the prepare-sdk rule. Again, we did not realise that those fixups would
be better placed in the host-finalize rule rather than the prepare-sdk.

While fixing the RPATH in host-finalize is not critical, fixing up the
PPD paths actually is, as the gdbinit case demonstrate.

As such, move the PPD fixups to the host-finalize step, and while at it,
also move the RPATH fixups.

This now does not leave much to do in the prepare-sdk step, and that
could very well be moved to the host-finalize rule as well. However,
some people may have started to rely on prepare-sdk in its 6 years of
existence, and the little script it installs is not needed unless one
really needs an SDK. So leave it as it is for now.

Reported-by: Casey Reeves <casey@xogium.me>
Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
Tested-by: Casey Reeves <casey@xogium.me>
Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
Cc: Brandon Maier <Brandon.Maier@collins.com>
---
 Makefile | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/Makefile b/Makefile
index 3bef434466..6c2a2ccddb 100644
--- a/Makefile
+++ b/Makefile
@@ -594,14 +594,7 @@ world: target-post-image
 
 .PHONY: prepare-sdk
 prepare-sdk: world
-	@$(call MESSAGE,"Rendering the SDK relocatable")
-	PARALLEL_JOBS=$(PARALLEL_JOBS) \
-		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
-		$(TOPDIR)/support/scripts/fix-rpath host
-	PARALLEL_JOBS=$(PARALLEL_JOBS) \
-		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
-		$(TOPDIR)/support/scripts/fix-rpath staging
-	$(call ppd-fixup-paths,$(BASE_DIR))
+	@$(call MESSAGE,"Preparing the SDK")
 	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
 	mkdir -p $(HOST_DIR)/share/buildroot
 	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
@@ -719,6 +712,13 @@ STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.t
 host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK)
 	@$(call MESSAGE,"Finalizing host directory")
 	$(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR),copy)
+	$(Q)PARALLEL_JOBS=$(PARALLEL_JOBS) \
+		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
+		$(TOPDIR)/support/scripts/fix-rpath host
+	$(Q)PARALLEL_JOBS=$(PARALLEL_JOBS) \
+		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
+		$(TOPDIR)/support/scripts/fix-rpath staging
+	$(call ppd-fixup-paths,$(BASE_DIR))
 
 .PHONY: staging-finalize
 staging-finalize: $(STAGING_DIR_SYMLINK)
-- 
2.45.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] Makefile: run PPD and RPATH fixup in host-fialize

[Brandon Maier via buildroot]

On 2024.06.09 09:54, Yann E. MORIN wrote:
> Currently, the gdbinit is generated and installed during post install
> hooks, either from the gdb package, or from the external-toolchain
> package. When using per-package directories (PPD), the staging directory
> of the either package is stored in the generated gdbinit, which is not
> going to be valid when all the PPD staging dirs get merged into the
> final staging: it would lack any library installed afterwards, i.e.
> mostly everything would be missing (but the libraries from the C
> toolchain in the case of an external toolchain).
> 
> Similarly, all the RPATH will point to various PPD drectories. This
> does not cause any issue when the final host is aggregated, because the
> PPD directories still exist when we call programs from there (e.g. from
> the fs infra, or from post-image scripts).
> 
> However, we knew that would not always be possible to keep the PPD
> directories: we have the prepare-sdk rule that runs a cleanup pass on
> the RPATH, and also applies the generic PPD fixups.
> 
> When we introduced prepare-sdk in c32ad51cbf2a (core/sdk: generate the
> SDK tarball ourselves), we did not yet have support for PPD for the host
> directory, and especially, we did not have the host-finalize rule, which
> was only introduced in d0f4f95e390b (Makefile: rework main directory
> creation logic) which kick-started the introduction of PPD.
> 
> At that point, we did not realise that the rpath fixups from
> prepare-sdk, would be better moved to the new host-finalize rule,
> because that had no impact unless one would need an SDK.
> 
> Later, in 25e60fbe1cab (Makefile: fix SDK relocation for
> per-package-dirs), we eventually introduced the PPD generic fixups in
> the prepare-sdk rule. Again, we did not realise that those fixups would
> be better placed in the host-finalize rule rather than the prepare-sdk.
> 
> While fixing the RPATH in host-finalize is not critical, fixing up the
> PPD paths actually is, as the gdbinit case demonstrate.
> 
> As such, move the PPD fixups to the host-finalize step, and while at it,
> also move the RPATH fixups.
> 
> This now does not leave much to do in the prepare-sdk step, and that
> could very well be moved to the host-finalize rule as well. However,
> some people may have started to rely on prepare-sdk in its 6 years of
> existence, and the little script it installs is not needed unless one
> really needs an SDK. So leave it as it is for now.
> 
> Reported-by: Casey Reeves <casey@xogium.me>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Tested-by: Casey Reeves <casey@xogium.me>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Brandon Maier <Brandon.Maier@collins.com>
> ---
>  Makefile | 16 ++++++++--------
>  1 file changed, 8 insertions(+), 8 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 3bef434466..6c2a2ccddb 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -594,14 +594,7 @@ world: target-post-image
>  
>  .PHONY: prepare-sdk
>  prepare-sdk: world
> -	@$(call MESSAGE,"Rendering the SDK relocatable")
> -	PARALLEL_JOBS=$(PARALLEL_JOBS) \
> -		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> -		$(TOPDIR)/support/scripts/fix-rpath host
> -	PARALLEL_JOBS=$(PARALLEL_JOBS) \
> -		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> -		$(TOPDIR)/support/scripts/fix-rpath staging
> -	$(call ppd-fixup-paths,$(BASE_DIR))
> +	@$(call MESSAGE,"Preparing the SDK")
>  	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
>  	mkdir -p $(HOST_DIR)/share/buildroot
>  	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
> @@ -719,6 +712,13 @@ STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.t
>  host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK)
>  	@$(call MESSAGE,"Finalizing host directory")
>  	$(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR),copy)
> +	$(Q)PARALLEL_JOBS=$(PARALLEL_JOBS) \
> +		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> +		$(TOPDIR)/support/scripts/fix-rpath host
> +	$(Q)PARALLEL_JOBS=$(PARALLEL_JOBS) \
> +		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> +		$(TOPDIR)/support/scripts/fix-rpath staging
> +	$(call ppd-fixup-paths,$(BASE_DIR))
>  
>  .PHONY: staging-finalize
>  staging-finalize: $(STAGING_DIR_SYMLINK)
> -- 
> 2.45.1
> 
> _______________________________________________
> buildroot mailing list
> buildroot@buildroot.org
> https://lists.buildroot.org/mailman/listinfo/buildroot

This change makes sense. I tested by building some internal boards and
the host directory appears correct.

My only concern is about the final make steps taking too long, as with
per-package-directories doing a simple `make <pkg>-rebuild all` can take
a couple extra minutes. In my test build, the added time for a large
build with 200+ packages is about 15 seconds, which isn't too bad. It
might be possible to optimize this later.

Tested-by: Brandon Maier <brandon.maier@collins.com>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] Makefile: run PPD and RPATH fixup in host-fialize

[TIAN Yuanhao]

At 2024-06-09 15:54:44, "Yann E. MORIN" <yann.morin.1998@free.fr> wrote:
>Currently, the gdbinit is generated and installed during post install
>hooks, either from the gdb package, or from the external-toolchain
>package. When using per-package directories (PPD), the staging directory
>of the either package is stored in the generated gdbinit, which is not
>going to be valid when all the PPD staging dirs get merged into the
>final staging: it would lack any library installed afterwards, i.e.
>mostly everything would be missing (but the libraries from the C
>toolchain in the case of an external toolchain).
>
>Similarly, all the RPATH will point to various PPD drectories. This
>does not cause any issue when the final host is aggregated, because the
>PPD directories still exist when we call programs from there (e.g. from
>the fs infra, or from post-image scripts).
>
>However, we knew that would not always be possible to keep the PPD
>directories: we have the prepare-sdk rule that runs a cleanup pass on
>the RPATH, and also applies the generic PPD fixups.
>
>When we introduced prepare-sdk in c32ad51cbf2a (core/sdk: generate the
>SDK tarball ourselves), we did not yet have support for PPD for the host
>directory, and especially, we did not have the host-finalize rule, which
>was only introduced in d0f4f95e390b (Makefile: rework main directory
>creation logic) which kick-started the introduction of PPD.
>
>At that point, we did not realise that the rpath fixups from
>prepare-sdk, would be better moved to the new host-finalize rule,
>because that had no impact unless one would need an SDK.
>
>Later, in 25e60fbe1cab (Makefile: fix SDK relocation for
>per-package-dirs), we eventually introduced the PPD generic fixups in
>the prepare-sdk rule. Again, we did not realise that those fixups would
>be better placed in the host-finalize rule rather than the prepare-sdk.
>
>While fixing the RPATH in host-finalize is not critical, fixing up the
>PPD paths actually is, as the gdbinit case demonstrate.
>
>As such, move the PPD fixups to the host-finalize step, and while at it,
>also move the RPATH fixups.
>
>This now does not leave much to do in the prepare-sdk step, and that
>could very well be moved to the host-finalize rule as well. However,
>some people may have started to rely on prepare-sdk in its 6 years of
>existence, and the little script it installs is not needed unless one
>really needs an SDK. So leave it as it is for now.
>
>Reported-by: Casey Reeves <casey@xogium.me>
>Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
>Tested-by: Casey Reeves <casey@xogium.me>
>Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
>Cc: Brandon Maier <Brandon.Maier@collins.com>
>---
> Makefile | 16 ++++++++--------
> 1 file changed, 8 insertions(+), 8 deletions(-)
>
>diff --git a/Makefile b/Makefile
>index 3bef434466..6c2a2ccddb 100644
>--- a/Makefile
>+++ b/Makefile
>@@ -594,14 +594,7 @@ world: target-post-image
> 
> .PHONY: prepare-sdk
> prepare-sdk: world
>-	@$(call MESSAGE,"Rendering the SDK relocatable")
>-	PARALLEL_JOBS=$(PARALLEL_JOBS) \
>-		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
>-		$(TOPDIR)/support/scripts/fix-rpath host
>-	PARALLEL_JOBS=$(PARALLEL_JOBS) \
>-		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
>-		$(TOPDIR)/support/scripts/fix-rpath staging
>-	$(call ppd-fixup-paths,$(BASE_DIR))
>+	@$(call MESSAGE,"Preparing the SDK")
> 	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
> 	mkdir -p $(HOST_DIR)/share/buildroot
> 	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
>@@ -719,6 +712,13 @@ STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.t
> host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK)
> 	@$(call MESSAGE,"Finalizing host directory")
> 	$(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR),copy)
>+	$(Q)PARALLEL_JOBS=$(PARALLEL_JOBS) \
>+		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
>+		$(TOPDIR)/support/scripts/fix-rpath host
>+	$(Q)PARALLEL_JOBS=$(PARALLEL_JOBS) \
>+		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
>+		$(TOPDIR)/support/scripts/fix-rpath staging
>+	$(call ppd-fixup-paths,$(BASE_DIR))
> 
> .PHONY: staging-finalize
> staging-finalize: $(STAGING_DIR_SYMLINK)
>-- 
>2.45.1
>

I found that my patch [1] was duplicated to this patch, and our situation is similar.

[1]: http://patchwork.ozlabs.org/project/buildroot/patch/20240828124440.3882-1-tianyuanhao3@163.com/

Acked-by: TIAN Yuanhao <tianyuanhao3@163.com>
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] Makefile: run PPD and RPATH fixup in host-fialize

[Yann E. MORIN]

All,

On 2024-06-09 09:54 +0200, Yann E. MORIN spake thusly:
[--SNIP--]
> As such, move the PPD fixups to the host-finalize step, and while at it,
> also move the RPATH fixups.
[--SNIP--]
> Reported-by: Casey Reeves <casey@xogium.me>
> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
> Tested-by: Casey Reeves <casey@xogium.me>
> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
> Cc: Brandon Maier <Brandon.Maier@collins.com>

Applied to master, thanks.

Regards,
Yann E. MORIN.

> ---
>  Makefile | 16 ++++++++--------
>  1 file changed, 8 insertions(+), 8 deletions(-)
> 
> diff --git a/Makefile b/Makefile
> index 3bef434466..6c2a2ccddb 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -594,14 +594,7 @@ world: target-post-image
>  
>  .PHONY: prepare-sdk
>  prepare-sdk: world
> -	@$(call MESSAGE,"Rendering the SDK relocatable")
> -	PARALLEL_JOBS=$(PARALLEL_JOBS) \
> -		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> -		$(TOPDIR)/support/scripts/fix-rpath host
> -	PARALLEL_JOBS=$(PARALLEL_JOBS) \
> -		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> -		$(TOPDIR)/support/scripts/fix-rpath staging
> -	$(call ppd-fixup-paths,$(BASE_DIR))
> +	@$(call MESSAGE,"Preparing the SDK")
>  	$(INSTALL) -m 755 $(TOPDIR)/support/misc/relocate-sdk.sh $(HOST_DIR)/relocate-sdk.sh
>  	mkdir -p $(HOST_DIR)/share/buildroot
>  	echo $(HOST_DIR) > $(HOST_DIR)/share/buildroot/sdk-location
> @@ -719,6 +712,13 @@ STAGING_DIR_FILES_LISTS = $(sort $(wildcard $(BUILD_DIR)/*/.files-list-staging.t
>  host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK)
>  	@$(call MESSAGE,"Finalizing host directory")
>  	$(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR),copy)
> +	$(Q)PARALLEL_JOBS=$(PARALLEL_JOBS) \
> +		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> +		$(TOPDIR)/support/scripts/fix-rpath host
> +	$(Q)PARALLEL_JOBS=$(PARALLEL_JOBS) \
> +		PER_PACKAGE_DIR=$(PER_PACKAGE_DIR) \
> +		$(TOPDIR)/support/scripts/fix-rpath staging
> +	$(call ppd-fixup-paths,$(BASE_DIR))
>  
>  .PHONY: staging-finalize
>  staging-finalize: $(STAGING_DIR_SYMLINK)
> -- 
> 2.45.1
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH] Makefile: run PPD and RPATH fixup in host-fialize

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > All,
 > On 2024-06-09 09:54 +0200, Yann E. MORIN spake thusly:
 > [--SNIP--]
 >> As such, move the PPD fixups to the host-finalize step, and while at it,
 >> also move the RPATH fixups.
 > [--SNIP--]
 >> Reported-by: Casey Reeves <casey@xogium.me>
 >> Signed-off-by: Yann E. MORIN <yann.morin.1998@free.fr>
 >> Tested-by: Casey Reeves <casey@xogium.me>
 >> Cc: Thomas Petazzoni <thomas.petazzoni@bootlin.com>
 >> Cc: Brandon Maier <Brandon.Maier@collins.com>

 > Applied to master, thanks.

Committed to 2024.02.x and 2024.08.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot