* [Buildroot] [PATCH 1/2] Makefile: use HOST_DIR_SYMLINK instead of hardcoding @ 2020-02-18 15:00 Thomas De Schampheleire 2020-02-18 15:01 ` [Buildroot] [PATCH 2/2] Makefile: don't recreate staging symlink if it exists Thomas De Schampheleire 2020-02-19 19:15 ` [Buildroot] [PATCH 1/2] Makefile: use HOST_DIR_SYMLINK instead of hardcoding Thomas Petazzoni 0 siblings, 2 replies; 6+ messages in thread From: Thomas De Schampheleire @ 2020-02-18 15:00 UTC (permalink / raw) To: buildroot From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> --- Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Makefile b/Makefile index a52f1c75fd..facccd32e7 100644 --- a/Makefile +++ b/Makefile @@ -455,7 +455,7 @@ endif ifneq ($(HOST_DIR),$(BASE_DIR)/host) HOST_DIR_SYMLINK = $(BASE_DIR)/host $(HOST_DIR_SYMLINK): $(BASE_DIR) - ln -snf $(HOST_DIR) $(BASE_DIR)/host + ln -snf $(HOST_DIR) $(HOST_DIR_SYMLINK) endif # Quotes are needed for spaces and all in the original PATH content. -- 2.24.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH 2/2] Makefile: don't recreate staging symlink if it exists 2020-02-18 15:00 [Buildroot] [PATCH 1/2] Makefile: use HOST_DIR_SYMLINK instead of hardcoding Thomas De Schampheleire @ 2020-02-18 15:01 ` Thomas De Schampheleire 2020-02-18 15:40 ` Thomas Petazzoni 2020-02-19 19:15 ` [Buildroot] [PATCH 1/2] Makefile: use HOST_DIR_SYMLINK instead of hardcoding Thomas Petazzoni 1 sibling, 1 reply; 6+ messages in thread From: Thomas De Schampheleire @ 2020-02-18 15:01 UTC (permalink / raw) To: buildroot From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> Create the staging symlink the same way as the host symlink. This means using a make dependency rather than recreating it every time. In coreutils versions below 8.27, re-creation of symbolic links was not atomic. This means that there is a period in time where the existing link is removed, before the new one is created. In coreutils 8.27 this was fixed, see [1]. Note that CentOS 7 ships with coreutils 8.22. In the following scenario, this is a problem: - an application is compiled using the sysroot prepared by Buildroot and links against Xenomai userspace libraries, but its build process is steered from outside of Buildroot - to know the correct flags, the application makefile uses the 'xeno-config' file to request them, and passes DESTDIR=/buildroot/output/staging - the xeno-config responds with flags based on the path '/buildroot/output/staging/...' - while the application build is ongoing, a 'make' happens in Buildroot, causing the 'staging' symlink to be recreated (even though it already existed) - when exactly at this time, the application calls the compiler with -I flags pointing to output/staging, the build fails with: -I/buildroot/output/staging/usr/include/xenomai/mercury: Error: ^ is not a directory -I/buildroot/output/staging/usr/include/xenomai: Error: ^ is not a directory -I/buildroot/output/staging/usr/include/xenomai/xenomai: Error: ^ is not a directory -I/buildroot/output/staging/usr/include/xenomai/psos: Error: ^ is not a directory Failed: ** ^ * Work around this problem by only creating the staging symlink once, similar to how the host symlink (if any) is created. See also commit d0f4f95e390bcb1c953efa125f5277a8a235396e which changed the way these symlinks are made. The reasoning in this commit is to move away from the 'dirs' target. [1] https://github.com/coreutils/coreutils/commit/376967889ed7ed561e46ff6d88a66779db62737a Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> --- Makefile | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index facccd32e7..3f301716f1 100644 --- a/Makefile +++ b/Makefile @@ -458,6 +458,10 @@ $(HOST_DIR_SYMLINK): $(BASE_DIR) ln -snf $(HOST_DIR) $(HOST_DIR_SYMLINK) endif +STAGING_DIR_SYMLINK = $(BASE_DIR)/staging +$(STAGING_DIR_SYMLINK): $(BASE_DIR) + ln -snf $(STAGING_DIR) $(STAGING_DIR_SYMLINK) + # Quotes are needed for spaces and all in the original PATH content. BR_PATH = "$(HOST_DIR)/bin:$(HOST_DIR)/sbin:$(PATH)" @@ -726,8 +730,7 @@ host-finalize: $(PACKAGES) $(HOST_DIR) $(HOST_DIR_SYMLINK) $(call per-package-rsync,$(sort $(PACKAGES)),host,$(HOST_DIR)) .PHONY: staging-finalize -staging-finalize: - @ln -snf $(STAGING_DIR) $(BASE_DIR)/staging +staging-finalize: $(STAGING_DIR_SYMLINK) .PHONY: target-finalize target-finalize: $(PACKAGES) $(TARGET_DIR) host-finalize -- 2.24.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH 2/2] Makefile: don't recreate staging symlink if it exists 2020-02-18 15:01 ` [Buildroot] [PATCH 2/2] Makefile: don't recreate staging symlink if it exists Thomas De Schampheleire @ 2020-02-18 15:40 ` Thomas Petazzoni 2020-02-18 17:59 ` Thomas De Schampheleire 0 siblings, 1 reply; 6+ messages in thread From: Thomas Petazzoni @ 2020-02-18 15:40 UTC (permalink / raw) To: buildroot Hello, On Tue, 18 Feb 2020 16:01:00 +0100 Thomas De Schampheleire <patrickdepinguin@gmail.com> wrote: > From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> > > Create the staging symlink the same way as the host symlink. This means > using a make dependency rather than recreating it every time. > > In coreutils versions below 8.27, re-creation of symbolic links was not > atomic. This means that there is a period in time where the existing link is > removed, before the new one is created. In coreutils 8.27 this was fixed, > see [1]. Note that CentOS 7 ships with coreutils 8.22. > > In the following scenario, this is a problem: > > - an application is compiled using the sysroot prepared by Buildroot and > links against Xenomai userspace libraries, but its build process is steered > from outside of Buildroot > - to know the correct flags, the application makefile uses the 'xeno-config' > file to request them, and passes DESTDIR=/buildroot/output/staging > - the xeno-config responds with flags based on the path > '/buildroot/output/staging/...' > - while the application build is ongoing, a 'make' happens in Buildroot, > causing the 'staging' symlink to be recreated (even though it already > existed) > - when exactly at this time, the application calls the compiler with -I > flags pointing to output/staging, the build fails with: > > -I/buildroot/output/staging/usr/include/xenomai/mercury: Error: ^ is not a directory > -I/buildroot/output/staging/usr/include/xenomai: Error: ^ is not a directory > -I/buildroot/output/staging/usr/include/xenomai/xenomai: Error: ^ is not a directory > -I/buildroot/output/staging/usr/include/xenomai/psos: Error: ^ is not a directory > Failed: ** ^ * > > Work around this problem by only creating the staging symlink once, similar > to how the host symlink (if any) is created. > > See also commit d0f4f95e390bcb1c953efa125f5277a8a235396e which changed the > way these symlinks are made. The reasoning in this commit is to move away > from the 'dirs' target. > > [1] https://github.com/coreutils/coreutils/commit/376967889ed7ed561e46ff6d88a66779db62737a Wow, thanks for the investigation and detailed commit log. Quite crazy stuff. The question that comes to mind is: why are you building something against the Buildroot toolchain/sysroot, before the Buildroot build has completed ? Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH 2/2] Makefile: don't recreate staging symlink if it exists 2020-02-18 15:40 ` Thomas Petazzoni @ 2020-02-18 17:59 ` Thomas De Schampheleire 2020-02-18 18:44 ` Thomas Petazzoni 0 siblings, 1 reply; 6+ messages in thread From: Thomas De Schampheleire @ 2020-02-18 17:59 UTC (permalink / raw) To: buildroot El mar., 18 feb. 2020 a las 16:40, Thomas Petazzoni (<thomas.petazzoni@bootlin.com>) escribi?: > > Hello, > > On Tue, 18 Feb 2020 16:01:00 +0100 > Thomas De Schampheleire <patrickdepinguin@gmail.com> wrote: > > > From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> > > > > Create the staging symlink the same way as the host symlink. This means > > using a make dependency rather than recreating it every time. > > > > In coreutils versions below 8.27, re-creation of symbolic links was not > > atomic. This means that there is a period in time where the existing link is > > removed, before the new one is created. In coreutils 8.27 this was fixed, > > see [1]. Note that CentOS 7 ships with coreutils 8.22. > > > > In the following scenario, this is a problem: > > > > - an application is compiled using the sysroot prepared by Buildroot and > > links against Xenomai userspace libraries, but its build process is steered > > from outside of Buildroot > > - to know the correct flags, the application makefile uses the 'xeno-config' > > file to request them, and passes DESTDIR=/buildroot/output/staging > > - the xeno-config responds with flags based on the path > > '/buildroot/output/staging/...' > > - while the application build is ongoing, a 'make' happens in Buildroot, > > causing the 'staging' symlink to be recreated (even though it already > > existed) > > - when exactly at this time, the application calls the compiler with -I > > flags pointing to output/staging, the build fails with: > > > > -I/buildroot/output/staging/usr/include/xenomai/mercury: Error: ^ is not a directory > > -I/buildroot/output/staging/usr/include/xenomai: Error: ^ is not a directory > > -I/buildroot/output/staging/usr/include/xenomai/xenomai: Error: ^ is not a directory > > -I/buildroot/output/staging/usr/include/xenomai/psos: Error: ^ is not a directory > > Failed: ** ^ * > > > > Work around this problem by only creating the staging symlink once, similar > > to how the host symlink (if any) is created. > > > > See also commit d0f4f95e390bcb1c953efa125f5277a8a235396e which changed the > > way these symlinks are made. The reasoning in this commit is to move away > > from the 'dirs' target. > > > > [1] https://github.com/coreutils/coreutils/commit/376967889ed7ed561e46ff6d88a66779db62737a > > Wow, thanks for the investigation and detailed commit log. Quite crazy > stuff. > > The question that comes to mind is: why are you building something > against the Buildroot toolchain/sysroot, before the Buildroot build has > completed ? No, the initial Buildroot build _is_ completed first. Then the application build starts according to the scenario described. In our case, the extra Buildroot 'make' is done to let Buildroot create the images, passing a custom TARGET_DIR which is an extended copy of the original output/target. This step happens multiple times for different boards, out of the same Buildroot (the boards differ a bit in our own software plus perhaps some other customization done in the rootfs). From the perspective of the application build, the fact that Buildroot 'make' is called shouldn't matter because all the files it needs will not be touched. Buildroot itself is also retained as-is, except it is asked to produce images. Best regards, Thomas ^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH 2/2] Makefile: don't recreate staging symlink if it exists 2020-02-18 17:59 ` Thomas De Schampheleire @ 2020-02-18 18:44 ` Thomas Petazzoni 0 siblings, 0 replies; 6+ messages in thread From: Thomas Petazzoni @ 2020-02-18 18:44 UTC (permalink / raw) To: buildroot On Tue, 18 Feb 2020 18:59:49 +0100 Thomas De Schampheleire <patrickdepinguin@gmail.com> wrote: > > The question that comes to mind is: why are you building something > > against the Buildroot toolchain/sysroot, before the Buildroot build has > > completed ? > > No, the initial Buildroot build _is_ completed first. Then the > application build starts according to the scenario described. > In our case, the extra Buildroot 'make' is done to let Buildroot > create the images, passing a custom TARGET_DIR which is an extended > copy of the original output/target. This step happens multiple times > for different boards, out of the same Buildroot (the boards differ a > bit in our own software plus perhaps some other customization done in > the rootfs). From the perspective of the application build, the fact > that Buildroot 'make' is called shouldn't matter because all the files > it needs will not be touched. Buildroot itself is also retained as-is, > except it is asked to produce images. Ah, ok. Convoluted use case, but OK. Anyway, the change makes sense, so we'll apply. Thanks for the additional explanation! Best regards, Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 6+ messages in thread
* [Buildroot] [PATCH 1/2] Makefile: use HOST_DIR_SYMLINK instead of hardcoding 2020-02-18 15:00 [Buildroot] [PATCH 1/2] Makefile: use HOST_DIR_SYMLINK instead of hardcoding Thomas De Schampheleire 2020-02-18 15:01 ` [Buildroot] [PATCH 2/2] Makefile: don't recreate staging symlink if it exists Thomas De Schampheleire @ 2020-02-19 19:15 ` Thomas Petazzoni 1 sibling, 0 replies; 6+ messages in thread From: Thomas Petazzoni @ 2020-02-19 19:15 UTC (permalink / raw) To: buildroot On Tue, 18 Feb 2020 16:00:59 +0100 Thomas De Schampheleire <patrickdepinguin@gmail.com> wrote: > From: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> > > Signed-off-by: Thomas De Schampheleire <thomas.de_schampheleire@nokia.com> > --- > Makefile | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) Thanks. Both applied to master. Thomas -- Thomas Petazzoni, CTO, Bootlin Embedded Linux and Kernel engineering https://bootlin.com ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-02-19 19:15 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2020-02-18 15:00 [Buildroot] [PATCH 1/2] Makefile: use HOST_DIR_SYMLINK instead of hardcoding Thomas De Schampheleire 2020-02-18 15:01 ` [Buildroot] [PATCH 2/2] Makefile: don't recreate staging symlink if it exists Thomas De Schampheleire 2020-02-18 15:40 ` Thomas Petazzoni 2020-02-18 17:59 ` Thomas De Schampheleire 2020-02-18 18:44 ` Thomas Petazzoni 2020-02-19 19:15 ` [Buildroot] [PATCH 1/2] Makefile: use HOST_DIR_SYMLINK instead of hardcoding Thomas Petazzoni
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox