[PATCH] gmp: depends on flex-native to fix parallel building issue

[PATCH] gmp: depends on flex-native to fix parallel building issue

[Haiqing Bai]

fix below parallel building issue:
configure:27365: result: flex
configure:27403: flex conftest.l
.../sysroots/x86_64-linux/usr/bin/flex.real: No such file or directory
configure:27407: $? = 127
configure:27409: checking lex output file root
configure:27420: error: cannot find output from flex; giving up

Signed-off-by: Haiqing Bai <Haiqing.Bai@windriver.com>
---
 meta/recipes-support/gmp/gmp.inc | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/recipes-support/gmp/gmp.inc b/meta/recipes-support/gmp/gmp.inc
index abac8cf..1b35eaa 100644
--- a/meta/recipes-support/gmp/gmp.inc
+++ b/meta/recipes-support/gmp/gmp.inc
@@ -10,3 +10,5 @@ PACKAGECONFIG[readline] = "--with-readline=yes,--with-readline=no,readline"
 
 ARM_INSTRUCTION_SET_armv4 = "arm"
 ARM_INSTRUCTION_SET_armv5 = "arm"
+
+DEPENDS = "flex-native"
-- 
2.0.1

Re: [PATCH] gmp: depends on flex-native to fix parallel building issue

[Richard Purdie]

On Wed, 2017-11-29 at 14:54 +0800, Haiqing Bai wrote:
> fix below parallel building issue:
> configure:27365: result: flex
> configure:27403: flex conftest.l
> .../sysroots/x86_64-linux/usr/bin/flex.real: No such file or
> directory
> configure:27407: $? = 127
> configure:27409: checking lex output file root
> configure:27420: error: cannot find output from flex; giving up
> 
> Signed-off-by: Haiqing Bai <Haiqing.Bai@windriver.com>
> ---
>  meta/recipes-support/gmp/gmp.inc | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/meta/recipes-support/gmp/gmp.inc b/meta/recipes-
> support/gmp/gmp.inc
> index abac8cf..1b35eaa 100644
> --- a/meta/recipes-support/gmp/gmp.inc
> +++ b/meta/recipes-support/gmp/gmp.inc
> @@ -10,3 +10,5 @@ PACKAGECONFIG[readline] = "--with-readline=yes,
> --with-readline=no,readline"
>  
>  ARM_INSTRUCTION_SET_armv4 = "arm"
>  ARM_INSTRUCTION_SET_armv5 = "arm"
> +
> +DEPENDS = "flex-native"

With recipe specific sysroots this should now be deterministic. The log
above suggests you were not using recipe specific sysroots? This would
therefore only be applicable to morty and earlier?

Cheers,

Richard

Re: [PATCH] gmp: depends on flex-native to fix parallel building issue

[Bai, Haiqing]

Comments below. thanks

-----Original Message-----
From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] 
Sent: 2017年12月11日 6:48
To: Bai, Haiqing; openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH] gmp: depends on flex-native to fix parallel building issue

On Wed, 2017-11-29 at 14:54 +0800, Haiqing Bai wrote:
> fix below parallel building issue:
> configure:27365: result: flex
> configure:27403: flex conftest.l
> .../sysroots/x86_64-linux/usr/bin/flex.real: No such file or directory
> configure:27407: $? = 127
> configure:27409: checking lex output file root
> configure:27420: error: cannot find output from flex; giving up
> 
> Signed-off-by: Haiqing Bai <Haiqing.Bai@windriver.com>
> ---
>  meta/recipes-support/gmp/gmp.inc | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/meta/recipes-support/gmp/gmp.inc b/meta/recipes- 
> support/gmp/gmp.inc index abac8cf..1b35eaa 100644
> --- a/meta/recipes-support/gmp/gmp.inc
> +++ b/meta/recipes-support/gmp/gmp.inc
> @@ -10,3 +10,5 @@ PACKAGECONFIG[readline] = "--with-readline=yes, 
> --with-readline=no,readline"
>  
>  ARM_INSTRUCTION_SET_armv4 = "arm"
>  ARM_INSTRUCTION_SET_armv5 = "arm"
> +
> +DEPENDS = "flex-native"

With recipe specific sysroots this should now be deterministic. The log above suggests you were not using recipe specific sysroots? This would therefore only be applicable to morty and earlier?
[This issue is founded on x86-64 building,  but it does not mean it is only related with x86.  Actually this is caused by the defect of the traditional probe mechanism of 'configure',  the package configure
script try to probe whether has package 'flex' , then some optional actions will be done by it.  In this issue,  when this probe happens,   /usr/bin/flex exists but '/usr/bin/flex.real' has not created for the
parallel building,  then configure reports the error and exits from building.   Since there are no atomic guarantee for the package output during parallel building, so here add this depends]

Cheers,

Richard

Re: [PATCH] gmp: depends on flex-native to fix parallel building issue

[Richard Purdie]

Let me make this simpler. Which release of the project did you see this
issue with?

I believe this issue would only occur with morty and early, pyro, rocko
and master are not affected.

Cheers,

Richard

On Tue, 2017-12-12 at 07:13 +0000, Bai, Haiqing wrote:
> Comments below. thanks
> 
> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] 
> Sent: 2017年12月11日 6:48
> To: Bai, Haiqing; openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH] gmp: depends on flex-native to fix
> parallel building issue
> 
> On Wed, 2017-11-29 at 14:54 +0800, Haiqing Bai wrote:
> > 
> > fix below parallel building issue:
> > configure:27365: result: flex
> > configure:27403: flex conftest.l
> > .../sysroots/x86_64-linux/usr/bin/flex.real: No such file or
> > directory
> > configure:27407: $? = 127
> > configure:27409: checking lex output file root
> > configure:27420: error: cannot find output from flex; giving up
> > 
> > Signed-off-by: Haiqing Bai <Haiqing.Bai@windriver.com>
> > ---
> >  meta/recipes-support/gmp/gmp.inc | 2 ++
> >  1 file changed, 2 insertions(+)
> > 
> > diff --git a/meta/recipes-support/gmp/gmp.inc b/meta/recipes- 
> > support/gmp/gmp.inc index abac8cf..1b35eaa 100644
> > --- a/meta/recipes-support/gmp/gmp.inc
> > +++ b/meta/recipes-support/gmp/gmp.inc
> > @@ -10,3 +10,5 @@ PACKAGECONFIG[readline] = "--with-readline=yes, 
> > --with-readline=no,readline"
> >  
> >  ARM_INSTRUCTION_SET_armv4 = "arm"
> >  ARM_INSTRUCTION_SET_armv5 = "arm"
> > +
> > +DEPENDS = "flex-native"
> With recipe specific sysroots this should now be deterministic. The
> log above suggests you were not using recipe specific sysroots? This
> would therefore only be applicable to morty and earlier?
> [This issue is founded on x86-64 building,  but it does not mean it
> is only related with x86.  Actually this is caused by the defect of
> the traditional probe mechanism of 'configure',  the package
> configure
> script try to probe whether has package 'flex' , then some optional
> actions will be done by it.  In this issue,  when this probe
> happens,   /usr/bin/flex exists but '/usr/bin/flex.real' has not
> created for the
> parallel building,  then configure reports the error and exits from
> building.   Since there are no atomic guarantee for the package
> output during parallel building, so here add this depends]
> 
> Cheers,
> 
> Richard
> 

Re: [PATCH] gmp: depends on flex-native to fix parallel building issue

[Bai, Haiqing]

Yes, sir,  I did not reproduce this on master.  Even on morty, I only saw
twice.  Maybe we can hold on this and let me keep monitoring.  Thanks.


-----Original Message-----
From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org] 
Sent: 2017年12月12日 17:23
To: Bai, Haiqing; openembedded-core@lists.openembedded.org
Subject: Re: [OE-core] [PATCH] gmp: depends on flex-native to fix parallel building issue

Let me make this simpler. Which release of the project did you see this issue with?

I believe this issue would only occur with morty and early, pyro, rocko and master are not affected.

Cheers,

Richard

On Tue, 2017-12-12 at 07:13 +0000, Bai, Haiqing wrote:
> Comments below. thanks
> 
> -----Original Message-----
> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
> Sent: 2017年12月11日 6:48
> To: Bai, Haiqing; openembedded-core@lists.openembedded.org
> Subject: Re: [OE-core] [PATCH] gmp: depends on flex-native to fix 
> parallel building issue
> 
> On Wed, 2017-11-29 at 14:54 +0800, Haiqing Bai wrote:
> > 
> > fix below parallel building issue:
> > configure:27365: result: flex
> > configure:27403: flex conftest.l
> > .../sysroots/x86_64-linux/usr/bin/flex.real: No such file or 
> > directory
> > configure:27407: $? = 127
> > configure:27409: checking lex output file root
> > configure:27420: error: cannot find output from flex; giving up
> > 
> > Signed-off-by: Haiqing Bai <Haiqing.Bai@windriver.com>
> > ---
> >  meta/recipes-support/gmp/gmp.inc | 2 ++
> >  1 file changed, 2 insertions(+)
> > 
> > diff --git a/meta/recipes-support/gmp/gmp.inc b/meta/recipes- 
> > support/gmp/gmp.inc index abac8cf..1b35eaa 100644
> > --- a/meta/recipes-support/gmp/gmp.inc
> > +++ b/meta/recipes-support/gmp/gmp.inc
> > @@ -10,3 +10,5 @@ PACKAGECONFIG[readline] = "--with-readline=yes, 
> > --with-readline=no,readline"
> >  
> >  ARM_INSTRUCTION_SET_armv4 = "arm"
> >  ARM_INSTRUCTION_SET_armv5 = "arm"
> > +
> > +DEPENDS = "flex-native"
> With recipe specific sysroots this should now be deterministic. The 
> log above suggests you were not using recipe specific sysroots? This 
> would therefore only be applicable to morty and earlier?
> [This issue is founded on x86-64 building,  but it does not mean it is 
> only related with x86.  Actually this is caused by the defect of the 
> traditional probe mechanism of 'configure',  the package configure 
> script try to probe whether has package 'flex' , then some optional 
> actions will be done by it.  In this issue,  when this probe happens,   
> /usr/bin/flex exists but '/usr/bin/flex.real' has not created for the 
> parallel building,  then configure reports the error and exits from 
> building.   Since there are no atomic guarantee for the package output 
> during parallel building, so here add this depends]
> 
> Cheers,
> 
> Richard
> 

Re: [PATCH] gmp: depends on flex-native to fix parallel building issue

[Randy MacLeod]

On 2017-12-12 04:22 AM, Richard Purdie wrote:
> Let me make this simpler. Which release of the project did you see this
> issue with?
> 
> I believe this issue would only occur with morty and early, pyro, rocko
> and master are not affected.

I expect I'm missing an undocumented recipe rule here but...


Yes, the race seems to only occur on morty and earlier, but
the dependency is real and not yet listed explicitly in the recipe.

  $ grep flex tmp/work/i586-poky-linux/gmp/6.1.2-r0/temp/log.do_configure
    checking for flex... flex

It's resolved transitively through:
    gmp -> something -> binutils-cross-foo -> flex-native

I was on the fence about whether to be explicit or to rely on
most/all recipes pulling in binutils-cross-foo via default
rules. In looking at the guidance in 4.3.9 Dependencies:
 
http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#new-dependencies
and briefly checking out LLVM's lld loader recipe where it seems
that flex-native might not be needed so it's best to be explicit.

Unless there's an objection, Bai will re-submit the simple
fix with an updated commit log. If anyone has time, they
can check if the LLVM lld build really does NOT pull in flex-native
and if not, then add explicit dependencies as required.

../Randy


> 
> Cheers,
> 
> Richard
> 
> On Tue, 2017-12-12 at 07:13 +0000, Bai, Haiqing wrote:
>> Comments below. thanks
>>
>> -----Original Message-----
>> From: Richard Purdie [mailto:richard.purdie@linuxfoundation.org]
>> Sent: 2017年12月11日 6:48
>> To: Bai, Haiqing; openembedded-core@lists.openembedded.org
>> Subject: Re: [OE-core] [PATCH] gmp: depends on flex-native to fix
>> parallel building issue
>>
>> On Wed, 2017-11-29 at 14:54 +0800, Haiqing Bai wrote:
>>>
>>> fix below parallel building issue:
>>> configure:27365: result: flex
>>> configure:27403: flex conftest.l
>>> .../sysroots/x86_64-linux/usr/bin/flex.real: No such file or
>>> directory
>>> configure:27407: $? = 127
>>> configure:27409: checking lex output file root
>>> configure:27420: error: cannot find output from flex; giving up
>>>
>>> Signed-off-by: Haiqing Bai <Haiqing.Bai@windriver.com>
>>> ---
>>>   meta/recipes-support/gmp/gmp.inc | 2 ++
>>>   1 file changed, 2 insertions(+)
>>>
>>> diff --git a/meta/recipes-support/gmp/gmp.inc b/meta/recipes-
>>> support/gmp/gmp.inc index abac8cf..1b35eaa 100644
>>> --- a/meta/recipes-support/gmp/gmp.inc
>>> +++ b/meta/recipes-support/gmp/gmp.inc
>>> @@ -10,3 +10,5 @@ PACKAGECONFIG[readline] = "--with-readline=yes,
>>> --with-readline=no,readline"
>>>   
>>>   ARM_INSTRUCTION_SET_armv4 = "arm"
>>>   ARM_INSTRUCTION_SET_armv5 = "arm"
>>> +
>>> +DEPENDS = "flex-native"
>> With recipe specific sysroots this should now be deterministic. The
>> log above suggests you were not using recipe specific sysroots? This
>> would therefore only be applicable to morty and earlier?
>> [This issue is founded on x86-64 building,  but it does not mean it
>> is only related with x86.  Actually this is caused by the defect of
>> the traditional probe mechanism of 'configure',  the package
>> configure
>> script try to probe whether has package 'flex' , then some optional
>> actions will be done by it.  In this issue,  when this probe
>> happens,   /usr/bin/flex exists but '/usr/bin/flex.real' has not
>> created for the
>> parallel building,  then configure reports the error and exits from
>> building.   Since there are no atomic guarantee for the package
>> output during parallel building, so here add this depends]
>>
>> Cheers,
>>
>> Richard
>>


-- 
# Randy MacLeod.  WR Linux
# Wind River an Intel Company

Re: [PATCH] gmp: depends on flex-native to fix parallel building issue

[Richard Purdie]

On Wed, 2018-01-03 at 16:35 -0500, Randy MacLeod wrote:
> I expect I'm missing an undocumented recipe rule here but...
> 
> 
> Yes, the race seems to only occur on morty and earlier, but
> the dependency is real and not yet listed explicitly in the recipe.
> 
>   $ grep flex tmp/work/i586-poky-linux/gmp/6.1.2-
> r0/temp/log.do_configure
>     checking for flex... flex
> 
> It's resolved transitively through:
>     gmp -> something -> binutils-cross-foo -> flex-native

I think it can happily build without flex by not re-generating certain
files. Whilst on target gmp does seem to have an indirect dependency,
our cross compiler does not:

$ grep flex tmp/work/x86_64-linux/gmp-native/6.1.2-r0/temp/log.do_configure
checking for flex... no

due to recipe specific sysroots and HOSTTOOLS, this is all at least
deterministic and it either builds with or without it deterministically
in all cases.

> I was on the fence about whether to be explicit or to rely on
> most/all recipes pulling in binutils-cross-foo via default
> rules. In looking at the guidance in 4.3.9 Dependencies:
>  
> http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#n
> ew-dependencies
> and briefly checking out LLVM's lld loader recipe where it seems
> that flex-native might not be needed so it's best to be explicit.
> 
> Unless there's an objection, Bai will re-submit the simple
> fix with an updated commit log. If anyone has time, they
> can check if the LLVM lld build really does NOT pull in flex-native
> and if not, then add explicit dependencies as required.

Actually, I do strongly object to adding in a flex-native dependency to
gmp-native and lengthening an already long/cumbersome toolchain
bootstrap process.

Cheers,

Richard