[PATCH] bison: Reset load average settings if specified in environment

[PATCH] bison: Reset load average settings if specified in environment

[Khem Raj]

In some cases, we run into parallel build failures where BUILT_SOURCES
is skipped, as a result required header files are not generated and the
build fails with missing header errors like

../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h: No such file or directory
 #include "uniwidth.h"
          ^~~~~~~~~~~~
compilation terminated.

BUILT_SOURCES should be built automatically with `make all` [1] therefore
ensure that make is invoked with `all` target

bison-native parallel build fails when -l<n> is passed globally from
build environment, errors like below due to race starts to show up

Therefore removes a previous load limit if set

[1] https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example

Signed-off-by: Khem Raj <raj.khem@gmail.com>
---
 meta/recipes-devtools/bison/bison_3.5.3.bb | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/meta/recipes-devtools/bison/bison_3.5.3.bb b/meta/recipes-devtools/bison/bison_3.5.3.bb
index 09e4b18f9e..27e09434f8 100644
--- a/meta/recipes-devtools/bison/bison_3.5.3.bb
+++ b/meta/recipes-devtools/bison/bison_3.5.3.bb
@@ -18,6 +18,12 @@ SRC_URI[sha256sum] = "2bf85b5f88a5f2fa8069aed2a2dfc3a9f8d15a97e59c713e3906e5fdd9
 # No point in hardcoding path to m4, just use PATH
 EXTRA_OECONF += "M4=m4"
 
+# Reset any loadavg set via environment, it breaks parallel build
+# | ../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h: No such file or directory
+# |  #include "uniwidth.h"
+# |           ^~~~~~~~~~~~
+EXTRA_OEMAKE_append = " -l"
+
 inherit autotools gettext texinfo
 
 # The automatic m4 path detection gets confused, so force the right value
-- 
2.25.1

✗ patchtest: failure for bison: Reset load average settings if specified in environment

[Patchwork]

== Series Details ==

Series: bison: Reset load average settings if specified in environment
Revision: 1
URL   : https://patchwork.openembedded.org/series/23244/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at b1ae2178fa)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: [PATCH] bison: Reset load average settings if specified in environment

[Alexander Kanavin]

[-- Attachment #1: Type: text/plain, Size: 2217 bytes --]

Shouldn’t the race be fixed? This might still happen because -j still
parallelises the build,no?

Alex

On Sun 15. Mar 2020 at 5.30, Khem Raj <raj.khem@gmail.com> wrote:

> In some cases, we run into parallel build failures where BUILT_SOURCES
> is skipped, as a result required header files are not generated and the
> build fails with missing header errors like
>
> ../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h: No
> such file or directory
>  #include "uniwidth.h"
>           ^~~~~~~~~~~~
> compilation terminated.
>
> BUILT_SOURCES should be built automatically with `make all` [1] therefore
> ensure that make is invoked with `all` target
>
> bison-native parallel build fails when -l<n> is passed globally from
> build environment, errors like below due to race starts to show up
>
> Therefore removes a previous load limit if set
>
> [1]
> https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example
>
> Signed-off-by: Khem Raj <raj.khem@gmail.com>
> ---
>  meta/recipes-devtools/bison/bison_3.5.3.bb | 6 ++++++
>  1 file changed, 6 insertions(+)
>
> diff --git a/meta/recipes-devtools/bison/bison_3.5.3.bb
> b/meta/recipes-devtools/bison/bison_3.5.3.bb
> index 09e4b18f9e..27e09434f8 100644
> --- a/meta/recipes-devtools/bison/bison_3.5.3.bb
> +++ b/meta/recipes-devtools/bison/bison_3.5.3.bb
> @@ -18,6 +18,12 @@ SRC_URI[sha256sum] =
> "2bf85b5f88a5f2fa8069aed2a2dfc3a9f8d15a97e59c713e3906e5fdd9
>  # No point in hardcoding path to m4, just use PATH
>  EXTRA_OECONF += "M4=m4"
>
> +# Reset any loadavg set via environment, it breaks parallel build
> +# | ../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h:
> No such file or directory
> +# |  #include "uniwidth.h"
> +# |           ^~~~~~~~~~~~
> +EXTRA_OEMAKE_append = " -l"
> +
>  inherit autotools gettext texinfo
>
>  # The automatic m4 path detection gets confused, so force the right value
> --
> 2.25.1
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 3448 bytes --]

Re: [PATCH] bison: Reset load average settings if specified in environment

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 2590 bytes --]

On Sat, Mar 14, 2020 at 10:38 PM Alexander Kanavin <alex.kanavin@gmail.com>
wrote:

> Shouldn’t the race be fixed? This might still happen because -j still
> parallelises the build,no?
>

It perhaps is a make bug which only triggers with -j and -l being both
present  it’s perhaps easily reproducible using these together on bison
builds
As such I don’t see anything wrong with bison makery


> Alex
>
> On Sun 15. Mar 2020 at 5.30, Khem Raj <raj.khem@gmail.com> wrote:
>
>> In some cases, we run into parallel build failures where BUILT_SOURCES
>> is skipped, as a result required header files are not generated and the
>> build fails with missing header errors like
>>
>> ../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h: No
>> such file or directory
>>  #include "uniwidth.h"
>>           ^~~~~~~~~~~~
>> compilation terminated.
>>
>> BUILT_SOURCES should be built automatically with `make all` [1] therefore
>> ensure that make is invoked with `all` target
>>
>> bison-native parallel build fails when -l<n> is passed globally from
>> build environment, errors like below due to race starts to show up
>>
>> Therefore removes a previous load limit if set
>>
>> [1]
>> https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example
>>
>> Signed-off-by: Khem Raj <raj.khem@gmail.com>
>> ---
>>  meta/recipes-devtools/bison/bison_3.5.3.bb | 6 ++++++
>>  1 file changed, 6 insertions(+)
>>
>> diff --git a/meta/recipes-devtools/bison/bison_3.5.3.bb
>> b/meta/recipes-devtools/bison/bison_3.5.3.bb
>> index 09e4b18f9e..27e09434f8 100644
>> --- a/meta/recipes-devtools/bison/bison_3.5.3.bb
>> +++ b/meta/recipes-devtools/bison/bison_3.5.3.bb
>> @@ -18,6 +18,12 @@ SRC_URI[sha256sum] =
>> "2bf85b5f88a5f2fa8069aed2a2dfc3a9f8d15a97e59c713e3906e5fdd9
>>  # No point in hardcoding path to m4, just use PATH
>>  EXTRA_OECONF += "M4=m4"
>>
>> +# Reset any loadavg set via environment, it breaks parallel build
>> +# | ../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h:
>> No such file or directory
>> +# |  #include "uniwidth.h"
>> +# |           ^~~~~~~~~~~~
>> +EXTRA_OEMAKE_append = " -l"
>> +
>>  inherit autotools gettext texinfo
>>
>>  # The automatic m4 path detection gets confused, so force the right value
>> --
>> 2.25.1
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
>

[-- Attachment #2: Type: text/html, Size: 4314 bytes --]

Re: [PATCH] bison: Reset load average settings if specified in environment

[Adrian Bunk]

On Sun, Mar 15, 2020 at 12:52:25AM -0700, Khem Raj wrote:
> On Sat, Mar 14, 2020 at 10:38 PM Alexander Kanavin <alex.kanavin@gmail.com>
> wrote:
> 
> > Shouldn’t the race be fixed? This might still happen because -j still
> > parallelises the build,no?
> 
> It perhaps is a make bug which only triggers with -j and -l being both
> present  it’s perhaps easily reproducible using these together on bison
> builds
> As such I don’t see anything wrong with bison makery

This is the second bison-native problem in addition to
https://bugzilla.yoctoproject.org/show_bug.cgi?id=13825

It is possible that the gnulib issue was a red herring and it is
actually is a parallel build issue, but both being in bison-native
does point more towards a problem specific to bison.

cu
Adrian

Re: [PATCH] bison: Reset load average settings if specified in environment

[Richard Purdie]

On Sat, 2020-03-14 at 21:29 -0700, Khem Raj wrote:
> In some cases, we run into parallel build failures where
> BUILT_SOURCES
> is skipped, as a result required header files are not generated and
> the
> build fails with missing header errors like
> 
> ../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h:
> No such file or directory
>  #include "uniwidth.h"
>           ^~~~~~~~~~~~
> compilation terminated.
> 
> BUILT_SOURCES should be built automatically with `make all` [1]
> therefore
> ensure that make is invoked with `all` target
> 
> bison-native parallel build fails when -l<n> is passed globally from
> build environment, errors like below due to race starts to show up
> 
> Therefore removes a previous load limit if set
> 
> [1] 
> https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example

I'm not sure I understand this. We've seen a failure on our ubuntu1604
performance tester which I think is the same bug as this. The
environment on that machine doesn't change, so how/where is this -l
option coming from?

I suspect there is some issue we're not quite getting to still :(

Cheers,

Richard

Re: [PATCH] bison: Reset load average settings if specified in environment

[Khem Raj]

On Sun, Mar 15, 2020 at 3:14 PM Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
>
> On Sat, 2020-03-14 at 21:29 -0700, Khem Raj wrote:
> > In some cases, we run into parallel build failures where
> > BUILT_SOURCES
> > is skipped, as a result required header files are not generated and
> > the
> > build fails with missing header errors like
> >
> > ../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h:
> > No such file or directory
> >  #include "uniwidth.h"
> >           ^~~~~~~~~~~~
> > compilation terminated.
> >
> > BUILT_SOURCES should be built automatically with `make all` [1]
> > therefore
> > ensure that make is invoked with `all` target
> >
> > bison-native parallel build fails when -l<n> is passed globally from
> > build environment, errors like below due to race starts to show up
> >
> > Therefore removes a previous load limit if set
> >
> > [1]
> > https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example
>
> I'm not sure I understand this. We've seen a failure on our ubuntu1604
> performance tester which I think is the same bug as this. The
> environment on that machine doesn't change, so how/where is this -l
> option coming from?
>

From custom distro global settings for EXTRA_OEMAKE

> I suspect there is some issue we're not quite getting to still :(
>
 its well reproducible now with -j and -l together on our builders, From what
I see in makefiles bison is doing right thing in using BUILT_SOURCES so
I suspect it could be make acting in a certain way when those options
are present.

our custom settings use -l options globally in EXTRA_OEMAKE
this is to let machines with certain configurations build when machines are
shared, we don't want a given build to use all resources on box. Yes one
could say why not do something else to limit the resources build uses but
this is a global setting that works well

This patch just ensures that any -l setting passed from environment is
cleaned up and reset. So for general case it will be a noop.

> Cheers,
>
> Richard
>

Re: [PATCH] bison: Reset load average settings if specified in environment

[Richard Purdie]

On Sun, 2020-03-15 at 15:23 -0700, Khem Raj wrote:
> On Sun, Mar 15, 2020 at 3:14 PM Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
> > On Sat, 2020-03-14 at 21:29 -0700, Khem Raj wrote:
> > > In some cases, we run into parallel build failures where
> > > BUILT_SOURCES
> > > is skipped, as a result required header files are not generated and
> > > the
> > > build fails with missing header errors like
> > > 
> > > ../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h:
> > > No such file or directory
> > >  #include "uniwidth.h"
> > >           ^~~~~~~~~~~~
> > > compilation terminated.
> > > 
> > > BUILT_SOURCES should be built automatically with `make all` [1]
> > > therefore
> > > ensure that make is invoked with `all` target
> > > 
> > > bison-native parallel build fails when -l<n> is passed globally from
> > > build environment, errors like below due to race starts to show up
> > > 
> > > Therefore removes a previous load limit if set
> > > 
> > > [1]
> > > https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example
> > 
> > I'm not sure I understand this. We've seen a failure on our ubuntu1604
> > performance tester which I think is the same bug as this. The
> > environment on that machine doesn't change, so how/where is this -l
> > option coming from?
> > 
> 
> From custom distro global settings for EXTRA_OEMAKE
> 
> > I suspect there is some issue we're not quite getting to still :(
> > 
>  its well reproducible now with -j and -l together on our builders, From what
> I see in makefiles bison is doing right thing in using BUILT_SOURCES so
> I suspect it could be make acting in a certain way when those options
> are present.
> 
> our custom settings use -l options globally in EXTRA_OEMAKE
> this is to let machines with certain configurations build when machines are
> shared, we don't want a given build to use all resources on box. Yes one
> could say why not do something else to limit the resources build uses but
> this is a global setting that works well
> 
> This patch just ensures that any -l setting passed from environment is
> cleaned up and reset. So for general case it will be a noop.

Ok, that explains your failure but not the one on our performance
testing worker :/.

Cheers,

Richard

Re: [PATCH] bison: Reset load average settings if specified in environment

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 2656 bytes --]

On Sun, Mar 15, 2020 at 3:25 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:

> On Sun, 2020-03-15 at 15:23 -0700, Khem Raj wrote:
> > On Sun, Mar 15, 2020 at 3:14 PM Richard Purdie
> > <richard.purdie@linuxfoundation.org> wrote:
> > > On Sat, 2020-03-14 at 21:29 -0700, Khem Raj wrote:
> > > > In some cases, we run into parallel build failures where
> > > > BUILT_SOURCES
> > > > is skipped, as a result required header files are not generated and
> > > > the
> > > > build fails with missing header errors like
> > > >
> > > > ../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h:
> > > > No such file or directory
> > > >  #include "uniwidth.h"
> > > >           ^~~~~~~~~~~~
> > > > compilation terminated.
> > > >
> > > > BUILT_SOURCES should be built automatically with `make all` [1]
> > > > therefore
> > > > ensure that make is invoked with `all` target
> > > >
> > > > bison-native parallel build fails when -l<n> is passed globally from
> > > > build environment, errors like below due to race starts to show up
> > > >
> > > > Therefore removes a previous load limit if set
> > > >
> > > > [1]
> > > >
> https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example
> > >
> > > I'm not sure I understand this. We've seen a failure on our ubuntu1604
> > > performance tester which I think is the same bug as this. The
> > > environment on that machine doesn't change, so how/where is this -l
> > > option coming from?
> > >
> >
> > From custom distro global settings for EXTRA_OEMAKE
> >
> > > I suspect there is some issue we're not quite getting to still :(
> > >
> >  its well reproducible now with -j and -l together on our builders, From
> what
> > I see in makefiles bison is doing right thing in using BUILT_SOURCES so
> > I suspect it could be make acting in a certain way when those options
> > are present.
> >
> > our custom settings use -l options globally in EXTRA_OEMAKE
> > this is to let machines with certain configurations build when machines
> are
> > shared, we don't want a given build to use all resources on box. Yes one
> > could say why not do something else to limit the resources build uses but
> > this is a global setting that works well
> >
> > This patch just ensures that any -l setting passed from environment is
> > cleaned up and reset. So for general case it will be a noop.
>
> Ok, that explains your failure but not the one on our performance
> testing worker :/.


What errors do you see on perf testing worker is it exactly same ?

>
>
> Cheers,
>
> Richard
>
>

[-- Attachment #2: Type: text/html, Size: 3796 bytes --]

Re: [OE-core] [PATCH] bison: Reset load average settings if specified in environment

[Andre McCurdy]

On Sun, Mar 15, 2020 at 3:40 PM Khem Raj <raj.khem@gmail.com> wrote:
> On Sun, Mar 15, 2020 at 3:25 PM Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
>> On Sun, 2020-03-15 at 15:23 -0700, Khem Raj wrote:
>> > On Sun, Mar 15, 2020 at 3:14 PM Richard Purdie
>> > <richard.purdie@linuxfoundation.org> wrote:
>> > > On Sat, 2020-03-14 at 21:29 -0700, Khem Raj wrote:
>> > > > In some cases, we run into parallel build failures where
>> > > > BUILT_SOURCES
>> > > > is skipped, as a result required header files are not generated and
>> > > > the
>> > > > build fails with missing header errors like
>> > > >
>> > > > ../bison-3.5.2/lib/uniwidth/width.c:21:10: fatal error: uniwidth.h:
>> > > > No such file or directory
>> > > >  #include "uniwidth.h"
>> > > >           ^~~~~~~~~~~~
>> > > > compilation terminated.
>> > > >
>> > > > BUILT_SOURCES should be built automatically with `make all` [1]
>> > > > therefore
>> > > > ensure that make is invoked with `all` target
>> > > >
>> > > > bison-native parallel build fails when -l<n> is passed globally from
>> > > > build environment, errors like below due to race starts to show up
>> > > >
>> > > > Therefore removes a previous load limit if set
>> > > >
>> > > > [1]
>> > > > https://www.gnu.org/software/automake/manual/html_node/Built-Sources-Example.html#Built-Sources-Example
>> > >
>> > > I'm not sure I understand this. We've seen a failure on our ubuntu1604
>> > > performance tester which I think is the same bug as this. The
>> > > environment on that machine doesn't change, so how/where is this -l
>> > > option coming from?
>> >
>> > From custom distro global settings for EXTRA_OEMAKE
>> >
>> > > I suspect there is some issue we're not quite getting to still :(
>> > >
>> >  its well reproducible now with -j and -l together on our builders, From what
>> > I see in makefiles bison is doing right thing in using BUILT_SOURCES so
>> > I suspect it could be make acting in a certain way when those options
>> > are present.
>> >
>> > our custom settings use -l options globally in EXTRA_OEMAKE
>> > this is to let machines with certain configurations build when machines are
>> > shared, we don't want a given build to use all resources on box. Yes one
>> > could say why not do something else to limit the resources build uses but
>> > this is a global setting that works well
>> >
>> > This patch just ensures that any -l setting passed from environment is
>> > cleaned up and reset. So for general case it will be a noop.
>>
>> Ok, that explains your failure but not the one on our performance
>> testing worker :/.

I've heard reports of this still being an issue with latest OE 3.1
when -j is set high enough, without any -l options in EXTRA_OEMAKE (so
Khem's workaround to over-ride -l doesn't help).

Disabling PARALLEL_MAKE completely might be the best workaround for
now until we understand the real root cause.