[PATCH] glibc: Fix timestamp of plural.c after unpack

[PATCH] glibc: Fix timestamp of plural.c after unpack

[Yuanjie Huang]

From: Yuanjie Huang <yuanjie.huang@windriver.com>

Source file plural.c might be updated with bison from plural.y, touch it
before applying patches so that this file is consistent across builds.

Signed-off-by: Yuanjie Huang <yuanjie.huang@windriver.com>
---
 meta/recipes-core/glibc/glibc.inc | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/meta/recipes-core/glibc/glibc.inc b/meta/recipes-core/glibc/glibc.inc
index e85c704..5225dfc 100644
--- a/meta/recipes-core/glibc/glibc.inc
+++ b/meta/recipes-core/glibc/glibc.inc
@@ -78,3 +78,10 @@ do_configure_prepend() {
 }
 
 GLIBC_ADDONS ?= "nptl,libidn"
+
+# Ensure that plural.c is newer than its Bison source plural.y,
+# unless the latter is patched.
+do_unpack_workaround() {
+	touch ${S}/intl/plural.c
+}
+addtask unpack_workaround after do_unpack before do_patch
-- 
1.9.1

Re: [PATCH] glibc: Fix timestamp of plural.c after unpack

[Mark Hatle]

FYI this particular issue was observed by one of our customers.  Starting around
the YP 2.0 time period.  (We did not observe the problem prior to 2.0, but it
may be present.)

My understanding of the issue is that the extraction of the plural.c and
plural.y files happens so quickly that it is unclear to make if one is older
then the other -- so it sometimes uses bison to rebuild the plural.c.  This
produces a non-deterministic build.

The patch below addresses this by inserting a task between unpack and patch that
makes sure we use the plural.c from the archive.  If a later patch changes
plural.y -- it will then be reconstructed as expected from source.

--Mark

On 10/11/16 10:02 AM, Yuanjie Huang wrote:
> From: Yuanjie Huang <yuanjie.huang@windriver.com>
> 
> Source file plural.c might be updated with bison from plural.y, touch it
> before applying patches so that this file is consistent across builds.
> 
> Signed-off-by: Yuanjie Huang <yuanjie.huang@windriver.com>
> ---
>  meta/recipes-core/glibc/glibc.inc | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/meta/recipes-core/glibc/glibc.inc b/meta/recipes-core/glibc/glibc.inc
> index e85c704..5225dfc 100644
> --- a/meta/recipes-core/glibc/glibc.inc
> +++ b/meta/recipes-core/glibc/glibc.inc
> @@ -78,3 +78,10 @@ do_configure_prepend() {
>  }
>  
>  GLIBC_ADDONS ?= "nptl,libidn"
> +
> +# Ensure that plural.c is newer than its Bison source plural.y,
> +# unless the latter is patched.
> +do_unpack_workaround() {
> +	touch ${S}/intl/plural.c
> +}
> +addtask unpack_workaround after do_unpack before do_patch
> 

Re: [PATCH] glibc: Fix timestamp of plural.c after unpack

[Burton, Ross]

[-- Attachment #1: Type: text/plain, Size: 469 bytes --]

On 11 October 2016 at 09:37, Mark Hatle <mark.hatle@windriver.com> wrote:

> My understanding of the issue is that the extraction of the plural.c and
> plural.y files happens so quickly that it is unclear to make if one is
> older
> then the other -- so it sometimes uses bison to rebuild the plural.c.  This
> produces a non-deterministic build.
>

Surely the rebuilt plural.c should be identical to the shipped one?
(semantically if not literally)

Ross

[-- Attachment #2: Type: text/html, Size: 883 bytes --]

Re: [PATCH] glibc: Fix timestamp of plural.c after unpack

[Yuanjie Huang]

[-- Attachment #1: Type: text/plain, Size: 756 bytes --]

The generated parser contains huge hunks of code actually shipped with 
bison, so I think we should also expect some semantic difference.

Yuanjie

On 10/11/2016 05:17 PM, Burton, Ross wrote:
>
> On 11 October 2016 at 09:37, Mark Hatle <mark.hatle@windriver.com 
> <mailto:mark.hatle@windriver.com>> wrote:
>
>     My understanding of the issue is that the extraction of the
>     plural.c and
>     plural.y files happens so quickly that it is unclear to make if
>     one is older
>     then the other -- so it sometimes uses bison to rebuild the
>     plural.c.  This
>     produces a non-deterministic build.
>
>
> Surely the rebuilt plural.c should be identical to the shipped one? 
> (semantically if not literally)
>
> Ross


[-- Attachment #2: Type: text/html, Size: 1939 bytes --]

Re: [PATCH] glibc: Fix timestamp of plural.c after unpack

[Mark Hatle]

On 10/11/16 11:50 AM, Yuanjie Huang wrote:
> The generated parser contains huge hunks of code actually shipped with bison, so
> I think we should also expect some semantic difference.

Exactly the problem.  Unless the plural.y has been modified, we should not be
regenerating the .c file as shipped by glibc.  It has the possibility of
introducing problems.... but the immediate concern is the non-determinism where
some builds rebuild, some do not.

--Mark

> Yuanjie
> 
> On 10/11/2016 05:17 PM, Burton, Ross wrote:
>>
>> On 11 October 2016 at 09:37, Mark Hatle <mark.hatle@windriver.com
>> <mailto:mark.hatle@windriver.com>> wrote:
>>
>>     My understanding of the issue is that the extraction of the plural.c and
>>     plural.y files happens so quickly that it is unclear to make if one is older
>>     then the other -- so it sometimes uses bison to rebuild the plural.c.  This
>>     produces a non-deterministic build.
>>
>>
>> Surely the rebuilt plural.c should be identical to the shipped one?
>> (semantically if not literally)
>>
>> Ross
> 

Re: [PATCH] glibc: Fix timestamp of plural.c after unpack

[Enrico Scholz]

Mark Hatle <mark.hatle-CWA4WttNNZF54TAoqtyWWQ@public.gmane.org> writes:

> My understanding of the issue is that the extraction of the plural.c and
> plural.y files happens so quickly that it is unclear to make if one is older
> then the other -- so it sometimes uses bison to rebuild the plural.c.

When this is really the case, this can apply to the workaround too.  So,
there should be added a 'sleep 1' before the 'touch' or (better?) remove
plural.c to enforce a rebuild.

>> +do_unpack_workaround() {
>> +	touch ${S}/intl/plural.c
>> +}


Enrico

Re: [PATCH] glibc: Fix timestamp of plural.c after unpack

[Gary Thomas]

On 2016-10-11 11:22, Enrico Scholz wrote:
> Mark Hatle <mark.hatle-CWA4WttNNZF54TAoqtyWWQ@public.gmane.org> writes:
>
>> My understanding of the issue is that the extraction of the plural.c and
>> plural.y files happens so quickly that it is unclear to make if one is older
>> then the other -- so it sometimes uses bison to rebuild the plural.c.
>
> When this is really the case, this can apply to the workaround too.  So,
> there should be added a 'sleep 1' before the 'touch' or (better?) remove
> plural.c to enforce a rebuild.

I believe that's the opposite of the desired action, i.e. to always use
the packaged plural.c and never rely on the plural.y=>plural.c transformation.

>
>>> +do_unpack_workaround() {
>>> +	touch ${S}/intl/plural.c
>>> +}

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: [PATCH] glibc: Fix timestamp of plural.c after unpack

[Mark Hatle]

On 10/11/16 11:46 AM, Gary Thomas wrote:
> On 2016-10-11 11:22, Enrico Scholz wrote:
>> Mark Hatle <mark.hatle-CWA4WttNNZF54TAoqtyWWQ@public.gmane.org> writes:
>>
>>> My understanding of the issue is that the extraction of the plural.c and
>>> plural.y files happens so quickly that it is unclear to make if one is older
>>> then the other -- so it sometimes uses bison to rebuild the plural.c.
>>
>> When this is really the case, this can apply to the workaround too.  So,
>> there should be added a 'sleep 1' before the 'touch' or (better?) remove
>> plural.c to enforce a rebuild.
> 
> I believe that's the opposite of the desired action, i.e. to always use
> the packaged plural.c and never rely on the plural.y=>plural.c transformation.
> 
>>
>>>> +do_unpack_workaround() {
>>>> +	touch ${S}/intl/plural.c
>>>> +}
> 

Correct.  We want to use the included version from glibc, -unless- there is a
patch that has modified the plural.y.

So doing this as an intermediate operation between unpack and patch should
correct the timestamp issue, while still allowing a patch to modify plural.y to
trigger an automatic regeneration (and any advantages or problems introduced by
the regeneration operation.)

As for the sleep 1, we've generally seen that is a long enough delay in tasks
that that is not necessary.  Most filesystems seem to have sub second accuracy.
However, if it does become an issue, I agree the sleep 1 is likely the only
solution to ensure the stamp has changed sufficiently..

--Mark

Re: [PATCH] glibc: Fix timestamp of plural.c after unpack

[Phil Blundell]

On Tue, 2016-10-11 at 16:02 +0800, Yuanjie Huang wrote:
> From: Yuanjie Huang <yuanjie.huang@windriver.com>
> 
> Source file plural.c might be updated with bison from plural.y, touch
> it
> before applying patches so that this file is consistent across
> builds.

I think it would be better to just configure glibc explicitly without
bison (BISON=no).  It doesn't DEPEND on bison-native anyway so it ought
never to be using it.

p.

Re: [PATCH] glibc: Fix timestamp of plural.c after unpack

[Mark Hatle]

On 10/11/16 1:53 PM, Phil Blundell wrote:
> On Tue, 2016-10-11 at 16:02 +0800, Yuanjie Huang wrote:
>> From: Yuanjie Huang <yuanjie.huang@windriver.com>
>>
>> Source file plural.c might be updated with bison from plural.y, touch
>> it
>> before applying patches so that this file is consistent across
>> builds.
> 
> I think it would be better to just configure glibc explicitly without
> bison (BISON=no).  It doesn't DEPEND on bison-native anyway so it ought
> never to be using it.

This was the first instance of the patch.  However after discussing the issue w/
Khem Raj (in person), we came to the conclusion that we did want a way to permit
people to modify the plural.y via a patch, otherwise someone could make a change
and not realize it was ignored.

--Mark

> p.
> 

Re: [PATCH] glibc: Fix timestamp of plural.c after unpack

[Phil Blundell]

On Tue, 2016-10-11 at 14:56 +0200, Mark Hatle wrote:
> 
> This was the first instance of the patch.  However after discussing
> the issue w/
> Khem Raj (in person), we came to the conclusion that we did want a
> way to permit
> people to modify the plural.y via a patch, otherwise someone could
> make a change
> and not realize it was ignored.

In that case they would need to add a dependency on bison-native in
order to get reliable builds.  Otherwise, if the build host happens to
not have bison installed, their change to plural.y will once again end
up being silently ignored.  I think it would be better to make it
consistent and deterministic.

p.