[PATCH 0/2] Make webkit-gtk depend on make-replacement-native 3.81

[PATCH 0/2] Make webkit-gtk depend on make-replacement-native 3.81

[Ross Burton]

This isn't my finest moment but I welcome suggestions...

webkit-gtk fails with -j when the host has GNU Make 3.82.  This appears to be a
GNU Make regression where it prunes rules it actually needs.

So add make-replacement-native 3.81, and depend on that in webkit-gtk.

I'm now off to scrub myself clean.

The following changes since commit 0c6ac62a9fb81146ace64ae7493dcc56b4bed197:

  documentation: remove references to Pimlico (2012-08-15 15:26:27 +0100)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib ross/make

for you to fetch changes up to 127bffbf550f6357ad14c1b5b03c1c908e4e51e2:

  webkit-gtk: add depends on make-replacement-native (2012-08-15 17:15:55 +0100)

----------------------------------------------------------------
Ross Burton (2):
      make: add make-replacement-native 3.81
      webkit-gtk: add depends on make-replacement-native

 meta/recipes-devtools/make/make-replacement-native_3.81.bb |    6 ++++++
 meta/recipes-sato/webkit/webkit-gtk_1.8.1.bb               |    8 +++++---
 2 files changed, 11 insertions(+), 3 deletions(-)
 create mode 100644 meta/recipes-devtools/make/make-replacement-native_3.81.bb

Ross Burton (2):
  make: add make-replacement-native 3.81
  webkit-gtk: add depends on make-replacement-native

 meta/recipes-devtools/make/make-replacement-native_3.81.bb |    6 ++++++
 meta/recipes-sato/webkit/webkit-gtk_1.8.1.bb               |    8 +++++---
 2 files changed, 11 insertions(+), 3 deletions(-)
 create mode 100644 meta/recipes-devtools/make/make-replacement-native_3.81.bb

-- 
1.7.10

[PATCH 1/2] make: add make-replacement-native 3.81

[Ross Burton]

GNU Make 3.82 has some regressions regarding parallel processing that causes
some packages to fail, specifically WebKitGTK+.  Add make-replacement-native
3.81 so that those packages can depend on a Make which is known to work.

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 meta/recipes-devtools/make/make-replacement-native_3.81.bb |    6 ++++++
 1 file changed, 6 insertions(+)
 create mode 100644 meta/recipes-devtools/make/make-replacement-native_3.81.bb

diff --git a/meta/recipes-devtools/make/make-replacement-native_3.81.bb b/meta/recipes-devtools/make/make-replacement-native_3.81.bb
new file mode 100644
index 0000000..716a8b5
--- /dev/null
+++ b/meta/recipes-devtools/make/make-replacement-native_3.81.bb
@@ -0,0 +1,6 @@
+require make_${PV}.bb
+
+inherit native
+
+BPN = "make"
+EXTRAINSTALL = ""
-- 
1.7.10

Re: [PATCH 1/2] make: add make-replacement-native 3.81

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1224 bytes --]

On Wed, Aug 15, 2012 at 05:18:49PM +0100, Ross Burton wrote:
> GNU Make 3.82 has some regressions regarding parallel processing that causes
> some packages to fail, specifically WebKitGTK+.  Add make-replacement-native
> 3.81 so that those packages can depend on a Make which is known to work.
> 
> Signed-off-by: Ross Burton <ross.burton@intel.com>
> ---
>  meta/recipes-devtools/make/make-replacement-native_3.81.bb |    6 ++++++
>  1 file changed, 6 insertions(+)
>  create mode 100644 meta/recipes-devtools/make/make-replacement-native_3.81.bb
> 
> diff --git a/meta/recipes-devtools/make/make-replacement-native_3.81.bb b/meta/recipes-devtools/make/make-replacement-native_3.81.bb
> new file mode 100644
> index 0000000..716a8b5
> --- /dev/null
> +++ b/meta/recipes-devtools/make/make-replacement-native_3.81.bb
> @@ -0,0 +1,6 @@
> +require make_${PV}.bb
> +
> +inherit native
> +
> +BPN = "make"
> +EXTRAINSTALL = ""

eglibc was failing to build with make-3.81
https://bugzilla.yoctoproject.org/show_bug.cgi?id=2314

so I guess once this is built (as webkit-gtk dependency), eglibc will maybe sometimes fail again..

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH 1/2] make: add make-replacement-native 3.81

[Burton, Ross]

On 15 August 2012 22:12, Martin Jansa <martin.jansa@gmail.com> wrote:
> eglibc was failing to build with make-3.81
> https://bugzilla.yoctoproject.org/show_bug.cgi?id=2314
>
> so I guess once this is built (as webkit-gtk dependency), eglibc will maybe sometimes fail again..

Interesting, this is the first time I've encountered 3.81 having
problems.  So we can't use 3.81 and we can't use 3.82 either. :)

Ross

Re: [PATCH 1/2] make: add make-replacement-native 3.81

[Phil Blundell]

On Wed, 2012-08-15 at 22:17 +0100, Burton, Ross wrote:
> On 15 August 2012 22:12, Martin Jansa <martin.jansa@gmail.com> wrote:
> > eglibc was failing to build with make-3.81
> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=2314
> >
> > so I guess once this is built (as webkit-gtk dependency), eglibc will maybe sometimes fail again..
> 
> Interesting, this is the first time I've encountered 3.81 having
> problems.  So we can't use 3.81 and we can't use 3.82 either. :)

You can use 3.81, you just need to install it either under a different
name (e.g. call it "make-3.81" and set $MAKE) or in a different
directory (e.g. ${bindir}/make-3.81/make and set $PATH) so that other
recipes don't pick it up unexpectedly.

p.

Re: [PATCH 1/2] make: add make-replacement-native 3.81

[Andreas Müller]

On Wed, Aug 15, 2012 at 11:40 PM, Phil Blundell <philb@gnu.org> wrote:
> On Wed, 2012-08-15 at 22:17 +0100, Burton, Ross wrote:
>> On 15 August 2012 22:12, Martin Jansa <martin.jansa@gmail.com> wrote:
>> > eglibc was failing to build with make-3.81
>> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=2314
>> >
>> > so I guess once this is built (as webkit-gtk dependency), eglibc will maybe sometimes fail again..
>>
>> Interesting, this is the first time I've encountered 3.81 having
>> problems.  So we can't use 3.81 and we can't use 3.82 either. :)
>
> You can use 3.81, you just need to install it either under a different
> name (e.g. call it "make-3.81" and set $MAKE) or in a different
> directory (e.g. ${bindir}/make-3.81/make and set $PATH) so that other
> recipes don't pick it up unexpectedly.
>
> p.
>
The following is really dirty but at least to me it succeeds and to me
it seems to have less influences than the 'make 3.81 dance':

# dirty hack for gnu make 3.82 race condition
do_compile() {
    if [ x"$MAKE" = x ]; then MAKE=make; fi
    bbnote make ${PARALLEL_MAKE} "$@"
    for error_count in 1 2 3; do
        bbnote "Number of trials: $error_count"
        exit_code=0
        make ${PARALLEL_MAKE} "$@" || exit_code=1
        if [ $exit_code = 0 ]; then
            break
        fi
    done
    if [ ! $exit_code = 0 ]; then
        die "oe_runmake failed"
    fi
}

I have make 3.82 and before this I had 100% error at first do_compile.
For rerun (without cleanup) I had 100% success. Now I get 100% success
with make called twice. Would something like have a chance to be
accepted?

Andreas

Re: [PATCH 1/2] make: add make-replacement-native 3.81

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2027 bytes --]

On Thu, Aug 16, 2012 at 10:30:37PM +0200, Andreas Müller wrote:
> On Wed, Aug 15, 2012 at 11:40 PM, Phil Blundell <philb@gnu.org> wrote:
> > On Wed, 2012-08-15 at 22:17 +0100, Burton, Ross wrote:
> >> On 15 August 2012 22:12, Martin Jansa <martin.jansa@gmail.com> wrote:
> >> > eglibc was failing to build with make-3.81
> >> > https://bugzilla.yoctoproject.org/show_bug.cgi?id=2314
> >> >
> >> > so I guess once this is built (as webkit-gtk dependency), eglibc will maybe sometimes fail again..
> >>
> >> Interesting, this is the first time I've encountered 3.81 having
> >> problems.  So we can't use 3.81 and we can't use 3.82 either. :)
> >
> > You can use 3.81, you just need to install it either under a different
> > name (e.g. call it "make-3.81" and set $MAKE) or in a different
> > directory (e.g. ${bindir}/make-3.81/make and set $PATH) so that other
> > recipes don't pick it up unexpectedly.
> >
> > p.
> >
> The following is really dirty but at least to me it succeeds and to me
> it seems to have less influences than the 'make 3.81 dance':
> 
> # dirty hack for gnu make 3.82 race condition
> do_compile() {
>     if [ x"$MAKE" = x ]; then MAKE=make; fi
>     bbnote make ${PARALLEL_MAKE} "$@"
>     for error_count in 1 2 3; do
>         bbnote "Number of trials: $error_count"
>         exit_code=0
>         make ${PARALLEL_MAKE} "$@" || exit_code=1
>         if [ $exit_code = 0 ]; then
>             break
>         fi
>     done
>     if [ ! $exit_code = 0 ]; then
>         die "oe_runmake failed"
>     fi
> }
> 
> I have make 3.82 and before this I had 100% error at first do_compile.
> For rerun (without cleanup) I had 100% success. Now I get 100% success
> with make called twice. Would something like have a chance to be
> accepted?

Same for me, except I didn't go that far to reexecute make from recipe,
just executing feed build twice in for loop to be sure it will succeed.

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH 1/2] make: add make-replacement-native 3.81

[Richard Purdie]

On Wed, 2012-08-15 at 17:18 +0100, Ross Burton wrote:
> GNU Make 3.82 has some regressions regarding parallel processing that causes
> some packages to fail, specifically WebKitGTK+.  Add make-replacement-native
> 3.81 so that those packages can depend on a Make which is known to work.
> 
> Signed-off-by: Ross Burton <ross.burton@intel.com>
> ---
>  meta/recipes-devtools/make/make-replacement-native_3.81.bb |    6 ++++++
>  1 file changed, 6 insertions(+)
>  create mode 100644 meta/recipes-devtools/make/make-replacement-native_3.81.bb
> 
> diff --git a/meta/recipes-devtools/make/make-replacement-native_3.81.bb b/meta/recipes-devtools/make/make-replacement-native_3.81.bb
> new file mode 100644
> index 0000000..716a8b5
> --- /dev/null
> +++ b/meta/recipes-devtools/make/make-replacement-native_3.81.bb
> @@ -0,0 +1,6 @@
> +require make_${PV}.bb
> +
> +inherit native
> +
> +BPN = "make"
> +EXTRAINSTALL = ""

Unfortunately we have some experience with these replacement recipes and
they need some "special" handling. You need to install make into a
subdirectory off ${bindir} and then add it to PATH in the webkit recipe.
There is precedent for this:

http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=3f05622bac998f351168eb49a5ca96e7473f51be

So BBCLASSEXTEND the make recipe, add a PROVIDES, then do a
EXTRANATIVEPATH += "make-native" in the webkit recipe.

The reason for this elaborate dance is so we avoid races, nothing should
be executing the make binary when we do anything to it with sstate
(install or remove it). Admittedly this is much more critical when there
is an associated library like bzip but we need to set the right example.

Cheers,

Richard

 

replacement natives - was Re: [PATCH 1/2] make: add make-replacement-native 3.81

[Mark Hatle]

On 8/15/12 4:39 PM, Richard Purdie wrote:
> On Wed, 2012-08-15 at 17:18 +0100, Ross Burton wrote:
>> GNU Make 3.82 has some regressions regarding parallel processing that causes
>> some packages to fail, specifically WebKitGTK+.  Add make-replacement-native
>> 3.81 so that those packages can depend on a Make which is known to work.
>>
>> Signed-off-by: Ross Burton <ross.burton@intel.com>
>> ---
>>   meta/recipes-devtools/make/make-replacement-native_3.81.bb |    6 ++++++
>>   1 file changed, 6 insertions(+)
>>   create mode 100644 meta/recipes-devtools/make/make-replacement-native_3.81.bb
>>
>> diff --git a/meta/recipes-devtools/make/make-replacement-native_3.81.bb b/meta/recipes-devtools/make/make-replacement-native_3.81.bb
>> new file mode 100644
>> index 0000000..716a8b5
>> --- /dev/null
>> +++ b/meta/recipes-devtools/make/make-replacement-native_3.81.bb
>> @@ -0,0 +1,6 @@
>> +require make_${PV}.bb
>> +
>> +inherit native
>> +
>> +BPN = "make"
>> +EXTRAINSTALL = ""
>
> Unfortunately we have some experience with these replacement recipes and
> they need some "special" handling. You need to install make into a
> subdirectory off ${bindir} and then add it to PATH in the webkit recipe.
> There is precedent for this:

I was looking at the staging code the other day.. I suspect we could eliminate 
some of these problems with a few changes.

The order the items are staged is the first.  Right now bin and sbin are staged 
before libs.. so if we change that, it would resolve some of the problems we've 
seen.  The second thing is the way the copy itself occurs.. We should really do 
this in an atomic fashion..  use install, or copy to an alternative name, and 
then 'mv' to the actual name.

Only concern w/ the change to copy could be performance, but I'm really not sure 
how severe that is.... might also be possible to do that only in the 'native' cases.

--Mark

> http://git.yoctoproject.org/cgit.cgi/poky/commit/?id=3f05622bac998f351168eb49a5ca96e7473f51be
>
> So BBCLASSEXTEND the make recipe, add a PROVIDES, then do a
> EXTRANATIVEPATH += "make-native" in the webkit recipe.
>
> The reason for this elaborate dance is so we avoid races, nothing should
> be executing the make binary when we do anything to it with sstate
> (install or remove it). Admittedly this is much more critical when there
> is an associated library like bzip but we need to set the right example.
>
> Cheers,
>
> Richard
>
>
>
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
>

[PATCH 2/2] webkit-gtk: add depends on make-replacement-native

[Ross Burton]

Depend on make-replacement-native 3.81 so that we're certainly not building with
make 3.82, which some distributions have moved to.  3.82 is known to fail to
build webkit-gtk with -j and building without parallelism really isn't an
option.

See https://bugs.webkit.org/show_bug.cgi?id=79498 for the WebKitGTK bug, and
savannah.gnu.org/bugs/?30653 for the GNU Make bug regarding the problems with
3.82.

[ YOCTO #2816 ]

Signed-off-by: Ross Burton <ross.burton@intel.com>
---
 meta/recipes-sato/webkit/webkit-gtk_1.8.1.bb |    8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-sato/webkit/webkit-gtk_1.8.1.bb b/meta/recipes-sato/webkit/webkit-gtk_1.8.1.bb
index dd43811..e6aed7d 100644
--- a/meta/recipes-sato/webkit/webkit-gtk_1.8.1.bb
+++ b/meta/recipes-sato/webkit/webkit-gtk_1.8.1.bb
@@ -7,14 +7,16 @@ LIC_FILES_CHKSUM = "file://Source/WebCore/rendering/RenderApplet.h;endline=22;md
                     file://Source/WebKit/gtk/webkit/webkit.h;endline=21;md5=b4fbe9f4a944f1d071dba1d2c76b3351 \
                     file://Source/JavaScriptCore/parser/Parser.h;endline=23;md5=b57c8a2952a8d0e655988fa0ecb2bf7f"
 
-PR = "r1"
+PR = "r2"
 
 # Choice of language backends - icu has issues on Big Endian machines so use pango
 ICU_LIB = "icu"
 ICU_LIB_powerpc = "pango"
 
-DEPENDS = "zlib enchant gnome-keyring libsoup-2.4 curl libxml2 cairo libxslt libxt libidn gnutls \
-           gtk+ gstreamer gst-plugins-base flex-native gperf-native perl-native-runtime sqlite3 ${ICU_LIB}"
+DEPENDS = "make-replacement-native zlib enchant gnome-keyring libsoup-2.4 curl \
+           libxml2 cairo libxslt libxt libidn gnutls gtk+ gstreamer \
+           gst-plugins-base flex-native gperf-native perl-native-runtime \
+           sqlite3 ${ICU_LIB}"
 DEPENDS += " ${@base_contains('DISTRO_FEATURES', 'opengl', 'virtual/libgl', '', d)}"
 
 PR = "r1"
-- 
1.7.10