[Buildroot] [PATCH 1/1] package/fwts: disable parallel build

[Buildroot] [PATCH 1/1] package/fwts: disable parallel build

[Julien Olivain]

fwts randomly fail to build in parallel with error:

    mv: cannot stat 'dtcompilerparser.tab.c': No such file or directory
    mv: cannot stat 'prparser.tab.c': No such file or directory

This is a known upstream issue. See:
https://github.com/fwts/fwts/issues/7

This commit workaround this issue by disabling parallel builds.

Signed-off-by: Julien Olivain <ju.o@free.fr>
---
 package/fwts/fwts.mk | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/package/fwts/fwts.mk b/package/fwts/fwts.mk
index f06609777db..cf3243c437a 100644
--- a/package/fwts/fwts.mk
+++ b/package/fwts/fwts.mk
@@ -14,6 +14,9 @@ FWTS_AUTORECONF = YES
 FWTS_DEPENDENCIES = host-bison host-flex host-pkgconf libglib2 libbsd \
 	$(if $(BR2_PACKAGE_BASH_COMPLETION),bash-completion) \
 	$(if $(BR2_PACKAGE_DTC),dtc)
+# fwts randomly fail to build in parallel
+# see: https://github.com/fwts/fwts/issues/7
+FWTS_MAKE = $(MAKE1)
 
 ifeq ($(BR2_OPTIMIZE_0),y)
 FWTS_CONF_ENV = CFLAGS="$(TARGET_CFLAGS) -O1"
-- 
2.44.0

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/fwts: disable parallel build

[Thomas Petazzoni via buildroot]

On Sat, 20 Apr 2024 15:22:51 +0200
Julien Olivain <ju.o@free.fr> wrote:

> fwts randomly fail to build in parallel with error:
> 
>     mv: cannot stat 'dtcompilerparser.tab.c': No such file or directory
>     mv: cannot stat 'prparser.tab.c': No such file or directory
> 
> This is a known upstream issue. See:
> https://github.com/fwts/fwts/issues/7
> 
> This commit workaround this issue by disabling parallel builds.
> 
> Signed-off-by: Julien Olivain <ju.o@free.fr>
> ---
>  package/fwts/fwts.mk | 3 +++
>  1 file changed, 3 insertions(+)

Hm, I don't see any failures in the autobuilders. Did you encounter
this issue personally? Why don't we see it in the autobuilders?

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/fwts: disable parallel build

[Julien Olivain]

Hi Thomas,

On 01/05/2024 21:55, Thomas Petazzoni wrote:
> On Sat, 20 Apr 2024 15:22:51 +0200
> Julien Olivain <ju.o@free.fr> wrote:
> 
>> fwts randomly fail to build in parallel with error:
>> 
>>     mv: cannot stat 'dtcompilerparser.tab.c': No such file or 
>> directory
>>     mv: cannot stat 'prparser.tab.c': No such file or directory
>> 
>> This is a known upstream issue. See:
>> https://github.com/fwts/fwts/issues/7
>> 
>> This commit workaround this issue by disabling parallel builds.
>> 
>> Signed-off-by: Julien Olivain <ju.o@free.fr>
>> ---
>>  package/fwts/fwts.mk | 3 +++
>>  1 file changed, 3 insertions(+)
> 
> Hm, I don't see any failures in the autobuilders. Did you encounter
> this issue personally? Why don't we see it in the autobuilders?

On my side, I see those failures on systems with with 20 cores and more 
on an
idle system doing only the Buildroot build. Maybe the autobuilders have 
a
different load, or less cores?

> 
> Thomas
> --
> Thomas Petazzoni, CTO, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com

Best regards,

Julien.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] package/fwts: disable parallel build

[Julien Olivain]

Hi again Thomas,

On 01/05/2024 22:27, Julien Olivain wrote:
> Hi Thomas,
> 
> On 01/05/2024 21:55, Thomas Petazzoni wrote:
>> On Sat, 20 Apr 2024 15:22:51 +0200
>> Julien Olivain <ju.o@free.fr> wrote:
>> 
>>> fwts randomly fail to build in parallel with error:
>>> 
>>>     mv: cannot stat 'dtcompilerparser.tab.c': No such file or 
>>> directory
>>>     mv: cannot stat 'prparser.tab.c': No such file or directory
>>> 
>>> This is a known upstream issue. See:
>>> https://github.com/fwts/fwts/issues/7
>>> 
>>> This commit workaround this issue by disabling parallel builds.
>>> 
>>> Signed-off-by: Julien Olivain <ju.o@free.fr>
>>> ---
>>>  package/fwts/fwts.mk | 3 +++
>>>  1 file changed, 3 insertions(+)
>> 
>> Hm, I don't see any failures in the autobuilders. Did you encounter
>> this issue personally? Why don't we see it in the autobuilders?
> 
> On my side, I see those failures on systems with with 20 cores and more 
> on an
> idle system doing only the Buildroot build. Maybe the autobuilders have 
> a
> different load, or less cores?

After testing a bit more, it appears the build always fail with
GNU make 4.4.1. Testing in the Buildroot docker reference image with
the same conditions (20+ cores, idle system), and which includes GNU
make 4.3 does not fail.

Looking at the host load, it seems GNU make 4.3 serializes some part of
the build, whereas GNU make 4.4.1 parallelize most of the build (and 
fail).

FWTS uses the .NOTPARALLEL special target with prerequisites:
https://github.com/fwts/fwts/blob/V24.03.00/src/libfwtsiasl/Makefile.am#L67

It seems the two versions of GNU make are not honoring this the same 
way.
It also seems the .NOTPARALLEL _with_ prerequisites is a feature added
in GNU make 4.4. See:
https://git.savannah.gnu.org/cgit/make.git/commit/?id=f6ea899d83bf00fe9201fde0ca9cf7af8e443677
https://lists.gnu.org/archive/html/help-make/2022-10/msg00020.html

Finally, the FWTS Makefile rules might also be incorrect, at:
https://github.com/fwts/fwts/blob/V24.03.00/src/libfwtsiasl/Makefile.am#L67

Quoting the Make manual:
https://www.gnu.org/software/make/manual/html_node/Parallel-Disable.html
"""
If the .NOTPARALLEL special target has prerequisites, then each of those
prerequisites will be considered a target and all prerequisites of these
targets will be run serially.
"""

Note the serialization will happen on the prerequisites of the target
set as prerequisites of .NOTPARALLEL.

My current understanding is that GNU Make 4.3 just ignores the
prerequisites of the .NOTPARALLEL and interprets it as if there
was none, which makes it serialize a bigger part of the build.

With this analysis, I may have an idea of a patch in the FWTS
Makefile.am. I'll do some more tests and propose a new patch.

Best regards,

Julien.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot