[PATCH 1/1] webkit-gtk: limit ld memory requirement

[PATCH 1/1] webkit-gtk: limit ld memory requirement

[Joe Slater]

Add --no-keep-memory to LDFLAGS.

Signed-off-by: Joe Slater <jslater@windriver.com>
---
 meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb |    4 ++++
 1 files changed, 4 insertions(+), 0 deletions(-)

diff --git a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
index 5691d3f..9304243 100644
--- a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
+++ b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
@@ -58,6 +58,10 @@ CPPFLAGS_append_powerpc = " -I${STAGING_INCDIR}/pango-1.0 \
                             -I${STAGING_LIBDIR}/glib-2.0/include \
                             -I${STAGING_INCDIR}/glib-2.0"
 
+# ld can run out of memory linking libwebkitgtk!
+#
+LDFLAGS += "-Wl,--no-keep-memory"
+
 EXTRA_AUTORECONF = " -I Source/autotools "
 
 
-- 
1.7.3.4

Re: [PATCH 1/1] webkit-gtk: limit ld memory requirement

[Khem Raj]

[-- Attachment #1: Type: text/plain, Size: 1369 bytes --]

On Friday, September 13, 2013, Joe Slater wrote:

> Add --no-keep-memory to LDFLAGS.


It does not come with out a price. Have you measured how much performance
degradation it brings in ?
We need that to assess the trade off and also describe the machine
configuration where this option is helping


>
> Signed-off-by: Joe Slater <jslater@windriver.com <javascript:;>>
> ---
>  meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb |    4 ++++
>  1 files changed, 4 insertions(+), 0 deletions(-)
>
> diff --git a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bbb/meta/recipes-sato/webkit/
> webkit-gtk_1.8.3.bb
> index 5691d3f..9304243 100644
> --- a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
> +++ b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
> @@ -58,6 +58,10 @@ CPPFLAGS_append_powerpc = "
> -I${STAGING_INCDIR}/pango-1.0 \
>                              -I${STAGING_LIBDIR}/glib-2.0/include \
>                              -I${STAGING_INCDIR}/glib-2.0"
>
> +# ld can run out of memory linking libwebkitgtk!
> +#
> +LDFLAGS += "-Wl,--no-keep-memory"
> +
>  EXTRA_AUTORECONF = " -I Source/autotools "
>
>
> --
> 1.7.3.4
>
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org <javascript:;>
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

[-- Attachment #2: Type: text/html, Size: 2335 bytes --]

Re: [PATCH 1/1] webkit-gtk: limit ld memory requirement

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1913 bytes --]

On Fri, Sep 13, 2013 at 09:35:36PM -0700, Khem Raj wrote:
> On Friday, September 13, 2013, Joe Slater wrote:
> 
> > Add --no-keep-memory to LDFLAGS.
> 

I think it was discussed on this ML before and conclusion was that it's
distro/builder policy and shouldn't be default in the recipe.
 
> It does not come with out a price. Have you measured how much performance
> degradation it brings in ?
> We need that to assess the trade off and also describe the machine
> configuration where this option is helping
> 
> 
> >
> > Signed-off-by: Joe Slater <jslater@windriver.com <javascript:;>>
> > ---
> >  meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb |    4 ++++
> >  1 files changed, 4 insertions(+), 0 deletions(-)
> >
> > diff --git a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bbb/meta/recipes-sato/webkit/
> > webkit-gtk_1.8.3.bb
> > index 5691d3f..9304243 100644
> > --- a/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
> > +++ b/meta/recipes-sato/webkit/webkit-gtk_1.8.3.bb
> > @@ -58,6 +58,10 @@ CPPFLAGS_append_powerpc = "
> > -I${STAGING_INCDIR}/pango-1.0 \
> >                              -I${STAGING_LIBDIR}/glib-2.0/include \
> >                              -I${STAGING_INCDIR}/glib-2.0"
> >
> > +# ld can run out of memory linking libwebkitgtk!
> > +#
> > +LDFLAGS += "-Wl,--no-keep-memory"
> > +
> >  EXTRA_AUTORECONF = " -I Source/autotools "
> >
> >
> > --
> > 1.7.3.4
> >
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org <javascript:;>
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core
> >

> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core


-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH 1/1] webkit-gtk: limit ld memory requirement

[Burton, Ross]

On 14 September 2013 05:35, Khem Raj <raj.khem@gmail.com> wrote:
> It does not come with out a price. Have you measured how much performance
> degradation it brings in ?
> We need that to assess the trade off and also describe the machine
> configuration where this option is helping

Agreed - there's a bug which is tracking this from Kai.  As far as I'm
aware the failure cases are 32-bit hosts (3GB limit per process) or
64-bit hosts with limited memory.  I suggested that we benchmark with
and without this - if the performance hit is measured in seconds then
we can probably enable it for this recipe out of the box.

Ross

Re: [PATCH 1/1] webkit-gtk: limit ld memory requirement

[Saul Wold]

On 09/14/2013 12:10 PM, Burton, Ross wrote:
> On 14 September 2013 05:35, Khem Raj <raj.khem@gmail.com> wrote:
>> It does not come with out a price. Have you measured how much performance
>> degradation it brings in ?
>> We need that to assess the trade off and also describe the machine
>> configuration where this option is helping
>
> Agreed - there's a bug which is tracking this from Kai.  As far as I'm
> aware the failure cases are 32-bit hosts (3GB limit per process) or
> 64-bit hosts with limited memory.  I suggested that we benchmark with
> and without this - if the performance hit is measured in seconds then
> we can probably enable it for this recipe out of the box.
>
Joe / Kai:

Please test as Khem and Ross suggest and report back with the results, 
also please include the [YOCTO #5160].

Ross, I am going to assign this to Joe.

Sau!

> Ross
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>

Re: [PATCH 1/1] webkit-gtk: limit ld memory requirement

[Kang Kai]

On 2013年09月15日 08:39, Saul Wold wrote:
> On 09/14/2013 12:10 PM, Burton, Ross wrote:
>> On 14 September 2013 05:35, Khem Raj <raj.khem@gmail.com> wrote:
>>> It does not come with out a price. Have you measured how much 
>>> performance
>>> degradation it brings in ?
>>> We need that to assess the trade off and also describe the machine
>>> configuration where this option is helping
>>
>> Agreed - there's a bug which is tracking this from Kai. As far as I'm
>> aware the failure cases are 32-bit hosts (3GB limit per process) or
>> 64-bit hosts with limited memory. I suggested that we benchmark with
>> and without this - if the performance hit is measured in seconds then
>> we can probably enable it for this recipe out of the box.
>>
> Joe / Kai:
>
> Please test as Khem and Ross suggest and report back with the results, 
> also please include the [YOCTO #5160].

run command 'time bitbake webkit-gtk -c compile' on the host which failed:

without patch till compile fails:
real 49m2.496s
user 102m47.789s
sys 31m18.698s

with the patch:
real 44m9.000s
user 106m45.182s
sys 30m53.223s


On the host build doesn't fail:

Without patch:
real 76m1.646s
user 86m36.704s
sys 38m39.103s

With patch:
real 63m36.193s
user 86m9.274s
sys 37m48.431s

It seems the link option doesn't increase the compile time.

Regards,
Kai

>
> Ross, I am going to assign this to Joe.
>
> Sau!
>
>> Ross
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>
>>
>
>


-- 
Regards,
Neil | Kai Kang

Re: [PATCH 1/1] webkit-gtk: limit ld memory requirement

[Khem Raj]

On Mon, Sep 16, 2013 at 10:11 PM, Kang Kai <Kai.Kang@windriver.com> wrote:
> Without patch:
> real 76m1.646s
> user 86m36.704s
> sys 38m39.103s
>
> With patch:
> real 63m36.193s
> user 86m9.274s
> sys 37m48.431s
>
> It seems the link option doesn't increase the compile time.

on the contrary its reducing the build time which is kind of
interesting and seems a win-win. are you able to extract the failing
link command and just run that with/without the linker option to
discard memory ? and measure just the linking time on the host where
it already builds without this option?

Re: [PATCH 1/1] webkit-gtk: limit ld memory requirement

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 967 bytes --]

On Mon, Sep 16, 2013 at 10:31:18PM -0700, Khem Raj wrote:
> On Mon, Sep 16, 2013 at 10:11 PM, Kang Kai <Kai.Kang@windriver.com> wrote:
> > Without patch:
> > real 76m1.646s
> > user 86m36.704s
> > sys 38m39.103s
> >
> > With patch:
> > real 63m36.193s
> > user 86m9.274s
> > sys 37m48.431s
> >
> > It seems the link option doesn't increase the compile time.
> 
> on the contrary its reducing the build time which is kind of
> interesting and seems a win-win. are you able to extract the failing
> link command and just run that with/without the linker option to
> discard memory ? and measure just the linking time on the host where
> it already builds without this option?

I think someone should measure it on machine with a lot of ram. If the
test was executed on machine where it was linking mostly from swap,
then it's not so surprising it was faster with the option enabled.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH 1/1] webkit-gtk: limit ld memory requirement

[Burton, Ross]

On 17 September 2013 11:30, Martin Jansa <martin.jansa@gmail.com> wrote:
> I think someone should measure it on machine with a lot of ram. If the
> test was executed on machine where it was linking mostly from swap,
> then it's not so surprising it was faster with the option enabled.

My very rough test says no significant difference between the option
being enabled or disabled (~23m +- 30s each way).  i7 with 16GB RAM.

Anyone else want to add a data point?

Ross

Re: [PATCH 1/1] webkit-gtk: limit ld memory requirement

[Slater, Joseph]

> -----Original Message-----
> From: Burton, Ross [mailto:ross.burton@intel.com]
> Sent: Tuesday, September 17, 2013 3:53 AM
> To: Martin Jansa
> Cc: Khem Raj; Slater, Joseph; Patches and discussions about the oe-core layer
> Subject: Re: [OE-core] [PATCH 1/1] webkit-gtk: limit ld memory requirement
> 
> On 17 September 2013 11:30, Martin Jansa <martin.jansa@gmail.com> wrote:
> > I think someone should measure it on machine with a lot of ram. If the
> > test was executed on machine where it was linking mostly from swap,
> > then it's not so surprising it was faster with the option enabled.
> 
> My very rough test says no significant difference between the option
> being enabled or disabled (~23m +- 30s each way).  i7 with 16GB RAM.
> 
> Anyone else want to add a data point?

I am the only user on the machine below.  I compiled twice, each way --

model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
model name	: Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz

MemTotal:       12326492 kB


build - present recipe
----------------------
real	48m26.620s
user	96m15.939s
sys	17m43.639s

compile - present recipe
------------------------
real	25m39.845s
user	85m31.102s
sys	10m17.016s

real	25m31.344s
user	85m35.210s
sys	9m46.309s

compile - new recipe
------------------------
real	25m28.338s
user	85m27.088s
sys	10m19.696s

real	25m22.355s
user	85m24.133s
sys	10m13.151s





> 
> Ross

Re: [PATCH 1/1] webkit-gtk: limit ld memory requirement

[Burton, Ross]

On 23 September 2013 22:51, Slater, Joseph <joe.slater@windriver.com> wrote:
>> Anyone else want to add a data point?
>
> I am the only user on the machine below.  I compiled twice, each way --

[snip]

Unless I'm reading that wrong, it looks like again its actually a
small win to use -no-keep-memory.  Sounds like we should just add it.

Ross

Re: [PATCH 1/1] webkit-gtk: limit ld memory requirement

[Richard Purdie]

On Tue, 2013-09-24 at 17:34 +0100, Burton, Ross wrote:
> On 23 September 2013 22:51, Slater, Joseph <joe.slater@windriver.com> wrote:
> >> Anyone else want to add a data point?
> >
> > I am the only user on the machine below.  I compiled twice, each way --
> 
> [snip]
> 
> Unless I'm reading that wrong, it looks like again its actually a
> small win to use -no-keep-memory.  Sounds like we should just add it.

Its in.

Cheers,

Richard

Re: [PATCH 1/1] webkit-gtk: limit ld memory requirement

[Khem Raj]

On Sep 24, 2013, at 9:34 AM, "Burton, Ross" <ross.burton@intel.com> wrote:

> On 23 September 2013 22:51, Slater, Joseph <joe.slater@windriver.com> wrote:
>>> Anyone else want to add a data point?
>> 
>> I am the only user on the machine below.  I compiled twice, each way --
> 
> [snip]
> 
> Unless I'm reading that wrong, it looks like again its actually a
> small win to use -no-keep-memory.  Sounds like we should just add it.
> 

yes indeed. Thanks all for chiming in.

> Ross