[Buildroot] [PATCH 1/2] package/libopenssl: add (proposed) upstream fix for parallel installation

[Buildroot] [PATCH 1/2] package/libopenssl: add (proposed) upstream fix for parallel installation

[Peter Korsgaard]

The openssl developers have proposed a fix for the parallel installation
issue worked around by commit 27ab880ebb9a (package/libopenssl do not build
in parallel).

Add the fix here so the workaround can dropped again.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
 ...l-process-to-avoid-multiple-make-dep.patch | 44 +++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 package/libopenssl/0004-Serialize-install-process-to-avoid-multiple-make-dep.patch

diff --git a/package/libopenssl/0004-Serialize-install-process-to-avoid-multiple-make-dep.patch b/package/libopenssl/0004-Serialize-install-process-to-avoid-multiple-make-dep.patch
new file mode 100644
index 0000000000..feb0e21ef7
--- /dev/null
+++ b/package/libopenssl/0004-Serialize-install-process-to-avoid-multiple-make-dep.patch
@@ -0,0 +1,44 @@
+From 4562b002366535b3d387fae7fa1dbd8e315ae142 Mon Sep 17 00:00:00 2001
+From: Neil Horman <nhorman@openssl.org>
+Date: Mon, 14 Apr 2025 12:23:15 -0400
+Subject: [PATCH] Serialize install process to avoid multiple make depend
+ operations
+
+If make install is run with a large -j value (make install -j N , where
+N > 1)
+
+We can run into a situation in which the install fails because multiple
+make depend operations are running in parallel, which will fail due to
+makefile rewriting.
+
+Serialize the install process to guarantee that those operations don't
+step on one another
+
+Fixes # 27074
+
+Upstream: https://github.com/openssl/openssl/pull/27388
+Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
+---
+ Configurations/unix-Makefile.tmpl | 6 +++++-
+ 1 file changed, 5 insertions(+), 1 deletion(-)
+
+diff --git a/Configurations/unix-Makefile.tmpl b/Configurations/unix-Makefile.tmpl
+index e85763ccf8..b671723813 100644
+--- a/Configurations/unix-Makefile.tmpl
++++ b/Configurations/unix-Makefile.tmpl
+@@ -650,7 +650,11 @@ depend: Makefile
+ # Install helper targets #############################################
+ ##@ Installation
+ 
+-install: install_sw install_ssldirs {- "install_docs" if !$disabled{docs}; -} {- $disabled{fips} ? "" : "install_fips" -} ## Install software and documentation, create OpenSSL directories
++install: Makefile ## Install software and documentation, create OpenSSL directories
++	$(MAKE) install_sw
++	$(MAKE) install_ssldirs
++	{- "\$(MAKE) install_docs" if !$disabled{docs} -}
++	{- "\$(MAKE) install_fips" if !$disabled{fips} -}
+ 
+ uninstall: {- "uninstall_docs" if !$disabled{docs}; -} uninstall_sw {- $disabled{fips} ? "" : "uninstall_fips" -} ## Uninstall software and documentation
+ 
+-- 
+2.39.5
+
-- 
2.39.5

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

[Buildroot] [PATCH 2/2] Revert "package/libopenssl do not build in parallel"

[Peter Korsgaard]

This reverts commit 27ab880ebb9a3efdec15733841f7b88063604247.

With the (proposed) fix from the openssl developers added as
0004-Serialize-install-process-to-avoid-multiple-make-dep.patch, the
workaround can now be dropped so openssl can again be built and installed in
parallel, significantly speeding up builds.

Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
 package/libopenssl/libopenssl.mk | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/package/libopenssl/libopenssl.mk b/package/libopenssl/libopenssl.mk
index 663e8d9b9f..c1c5b818d7 100644
--- a/package/libopenssl/libopenssl.mk
+++ b/package/libopenssl/libopenssl.mk
@@ -132,26 +132,24 @@ endef
 LIBOPENSSL_POST_CONFIGURE_HOOKS += LIBOPENSSL_FIXUP_STATIC_MAKEFILE
 endif
 
-# Parallel build broken: https://github.com/openssl/openssl/issues/27074
 define HOST_LIBOPENSSL_BUILD_CMDS
-	$(HOST_MAKE_ENV) $(MAKE1) -C $(@D)
+	$(HOST_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
-# Parallel build broken: https://github.com/openssl/openssl/issues/27074
 define LIBOPENSSL_BUILD_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D)
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
 endef
 
 define LIBOPENSSL_INSTALL_STAGING_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) DESTDIR=$(STAGING_DIR) install
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR=$(STAGING_DIR) install
 endef
 
 define HOST_LIBOPENSSL_INSTALL_CMDS
-	$(HOST_MAKE_ENV) $(MAKE1) -C $(@D) install
+	$(HOST_MAKE_ENV) $(MAKE) -C $(@D) install
 endef
 
 define LIBOPENSSL_INSTALL_TARGET_CMDS
-	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) DESTDIR=$(TARGET_DIR) install
+	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR=$(TARGET_DIR) install
 	$(RM) -rf $(TARGET_DIR)/usr/lib/ssl
 	$(RM) -f $(TARGET_DIR)/usr/bin/c_rehash
 endef
-- 
2.39.5

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] package/libopenssl: add (proposed) upstream fix for parallel installation

[Yann E. MORIN]

Peter, All,

On 2025-04-15 22:00 +0200, Peter Korsgaard spake thusly:
> The openssl developers have proposed a fix for the parallel installation
> issue worked around by commit 27ab880ebb9a (package/libopenssl do not build
> in parallel).
> 
> Add the fix here so the workaround can dropped again.
> 
> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

Reviewed-by: Yann E. MORIN <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

> ---
>  ...l-process-to-avoid-multiple-make-dep.patch | 44 +++++++++++++++++++
>  1 file changed, 44 insertions(+)
>  create mode 100644 package/libopenssl/0004-Serialize-install-process-to-avoid-multiple-make-dep.patch
> 
> diff --git a/package/libopenssl/0004-Serialize-install-process-to-avoid-multiple-make-dep.patch b/package/libopenssl/0004-Serialize-install-process-to-avoid-multiple-make-dep.patch
> new file mode 100644
> index 0000000000..feb0e21ef7
> --- /dev/null
> +++ b/package/libopenssl/0004-Serialize-install-process-to-avoid-multiple-make-dep.patch
> @@ -0,0 +1,44 @@
> +From 4562b002366535b3d387fae7fa1dbd8e315ae142 Mon Sep 17 00:00:00 2001
> +From: Neil Horman <nhorman@openssl.org>
> +Date: Mon, 14 Apr 2025 12:23:15 -0400
> +Subject: [PATCH] Serialize install process to avoid multiple make depend
> + operations
> +
> +If make install is run with a large -j value (make install -j N , where
> +N > 1)
> +
> +We can run into a situation in which the install fails because multiple
> +make depend operations are running in parallel, which will fail due to
> +makefile rewriting.
> +
> +Serialize the install process to guarantee that those operations don't
> +step on one another
> +
> +Fixes # 27074
> +
> +Upstream: https://github.com/openssl/openssl/pull/27388
> +Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
> +---
> + Configurations/unix-Makefile.tmpl | 6 +++++-
> + 1 file changed, 5 insertions(+), 1 deletion(-)
> +
> +diff --git a/Configurations/unix-Makefile.tmpl b/Configurations/unix-Makefile.tmpl
> +index e85763ccf8..b671723813 100644
> +--- a/Configurations/unix-Makefile.tmpl
> ++++ b/Configurations/unix-Makefile.tmpl
> +@@ -650,7 +650,11 @@ depend: Makefile
> + # Install helper targets #############################################
> + ##@ Installation
> + 
> +-install: install_sw install_ssldirs {- "install_docs" if !$disabled{docs}; -} {- $disabled{fips} ? "" : "install_fips" -} ## Install software and documentation, create OpenSSL directories
> ++install: Makefile ## Install software and documentation, create OpenSSL directories
> ++	$(MAKE) install_sw
> ++	$(MAKE) install_ssldirs
> ++	{- "\$(MAKE) install_docs" if !$disabled{docs} -}
> ++	{- "\$(MAKE) install_fips" if !$disabled{fips} -}
> + 
> + uninstall: {- "uninstall_docs" if !$disabled{docs}; -} uninstall_sw {- $disabled{fips} ? "" : "uninstall_fips" -} ## Uninstall software and documentation
> + 
> +-- 
> +2.39.5
> +
> -- 
> 2.39.5
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2] Revert "package/libopenssl do not build in parallel"

[Yann E. MORIN]

Pter, All,

On 2025-04-15 22:00 +0200, Peter Korsgaard spake thusly:
> This reverts commit 27ab880ebb9a3efdec15733841f7b88063604247.

I don't think this really is a "revert". Reverting a commit is for when
that commit was wrong when it was applied. But here, the commit was not
wrong, so it should not be "reverted". Even if the change is technically
undone, that's semantically a different thing. But oh well... ;-)

> With the (proposed) fix from the openssl developers added as
> 0004-Serialize-install-process-to-avoid-multiple-make-dep.patch, the
> workaround can now be dropped so openssl can again be built and installed in
> parallel, significantly speeding up builds.

Yes, that was a shame that we had to lose parallelism...

Note that I also hit a hihly-parallel highly-loaded build (not install)
issue that is still not fixed. Alas, that one is also very, very
difficult to hit. We can probably ignore it until I get bitten again and
get more traces to inspect...

> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

Reviewed-by: Yann E. MORIN <yann.morin.1998@free.fr>

Regards,
Yann E. MORIN.

> ---
>  package/libopenssl/libopenssl.mk | 12 +++++-------
>  1 file changed, 5 insertions(+), 7 deletions(-)
> 
> diff --git a/package/libopenssl/libopenssl.mk b/package/libopenssl/libopenssl.mk
> index 663e8d9b9f..c1c5b818d7 100644
> --- a/package/libopenssl/libopenssl.mk
> +++ b/package/libopenssl/libopenssl.mk
> @@ -132,26 +132,24 @@ endef
>  LIBOPENSSL_POST_CONFIGURE_HOOKS += LIBOPENSSL_FIXUP_STATIC_MAKEFILE
>  endif
>  
> -# Parallel build broken: https://github.com/openssl/openssl/issues/27074
>  define HOST_LIBOPENSSL_BUILD_CMDS
> -	$(HOST_MAKE_ENV) $(MAKE1) -C $(@D)
> +	$(HOST_MAKE_ENV) $(MAKE) -C $(@D)
>  endef
>  
> -# Parallel build broken: https://github.com/openssl/openssl/issues/27074
>  define LIBOPENSSL_BUILD_CMDS
> -	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D)
> +	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D)
>  endef
>  
>  define LIBOPENSSL_INSTALL_STAGING_CMDS
> -	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) DESTDIR=$(STAGING_DIR) install
> +	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR=$(STAGING_DIR) install
>  endef
>  
>  define HOST_LIBOPENSSL_INSTALL_CMDS
> -	$(HOST_MAKE_ENV) $(MAKE1) -C $(@D) install
> +	$(HOST_MAKE_ENV) $(MAKE) -C $(@D) install
>  endef
>  
>  define LIBOPENSSL_INSTALL_TARGET_CMDS
> -	$(TARGET_MAKE_ENV) $(MAKE1) -C $(@D) DESTDIR=$(TARGET_DIR) install
> +	$(TARGET_MAKE_ENV) $(MAKE) -C $(@D) DESTDIR=$(TARGET_DIR) install
>  	$(RM) -rf $(TARGET_DIR)/usr/lib/ssl
>  	$(RM) -f $(TARGET_DIR)/usr/bin/c_rehash
>  endef
> -- 
> 2.39.5
> 

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 2/2] Revert "package/libopenssl do not build in parallel"

[Peter Korsgaard]

>>>>> "Yann" == Yann E MORIN <yann.morin.1998@free.fr> writes:

 > Pter, All,
 > On 2025-04-15 22:00 +0200, Peter Korsgaard spake thusly:
 >> This reverts commit 27ab880ebb9a3efdec15733841f7b88063604247.

 > I don't think this really is a "revert". Reverting a commit is for when
 > that commit was wrong when it was applied. But here, the commit was not
 > wrong, so it should not be "reverted". Even if the change is technically
 > undone, that's semantically a different thing. But oh well... ;-)

Well, it reverted the workaround?


 >> With the (proposed) fix from the openssl developers added as
 >> 0004-Serialize-install-process-to-avoid-multiple-make-dep.patch, the
 >> workaround can now be dropped so openssl can again be built and installed in
 >> parallel, significantly speeding up builds.

 > Yes, that was a shame that we had to lose parallelism...

I agree, especially since openssl takes so long to build. I have a
defconfig at $WORK where openssl is the package taking longest to
build. This defconfig builds both libopenssl and host-libopenssl, and
together they take over 25% of the build time.


 > Note that I also hit a hihly-parallel highly-loaded build (not install)
 > issue that is still not fixed. Alas, that one is also very, very
 > difficult to hit. We can probably ignore it until I get bitten again and
 > get more traces to inspect...

OK. I have so far not been able to trigger that here.


 >> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

 > Reviewed-by: Yann E. MORIN <yann.morin.1998@free.fr>

Thanks!

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/2] package/libopenssl: add (proposed) upstream fix for parallel installation

[Julien Olivain]

On 15/04/2025 22:00, Peter Korsgaard wrote:
> The openssl developers have proposed a fix for the parallel 
> installation
> issue worked around by commit 27ab880ebb9a (package/libopenssl do not 
> build
> in parallel).
> 
> Add the fix here so the workaround can dropped again.
> 
> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

Series applied to master, thanks.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot