[PATCH 0/2] sstate improvements

[PATCH 0/2] sstate improvements

[Martin Jansa]

Include native/cross/nativesdk deps in target signatures to make sstate
reuse a bit safer.

Make allarch a bit more independent on TARGET_* variables.

The following changes since commit 8163854adf87ac42a8f08ee25685d0ce1efb4724:

  oe-selftest: separated the SStateBase and SStateTests in different modules (2014-01-16 12:18:44 +0000)

are available in the git repository at:

  git://git.openembedded.org/openembedded-core-contrib jansa/sstate
  http://cgit.openembedded.org/cgit.cgi/openembedded-core-contrib/log/?h=jansa/sstate

Martin Jansa (2):
  sstatesig: include native/cross/nativesdk deps in target signatures
  allarch: Set empty TARGET_PREFIX and TARGET_FPU

 meta/classes/allarch.bbclass | 2 ++
 meta/lib/oe/sstatesig.py     | 4 ----
 2 files changed, 2 insertions(+), 4 deletions(-)

-- 
1.8.5.2

[PATCH 1/2] sstatesig: include native/cross/nativesdk deps in target signatures

[Martin Jansa]

* I don't have any real evidence or good statistics for this, but when
  comparing signature dumps from my big bitbake world builds I usually
  see a lot of rebuilds caused by changes in .bbclasses and only very
  rare would be the case where oe-core upgrade brings changes in -native
  recipes and no change in .bbclasses used from target recipes
* changing the default to include them shouldn't cause significant
  increase in rebuilds and sstate reuse a bit safer
* people working on toolchain (e.g. using gcc from AUTOREV) can easily
  extend sstate_rundepfilter to ignore them again (it's easier than
  removing existing filter), example how add own signature handler in
  your layer is here:
  https://github.com/openwebos/meta-webos/commit/9ac3a7c803e7793b3274e4998f167b6278db8042

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/lib/oe/sstatesig.py | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index 3011f16..8b0e344 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -36,10 +36,6 @@ def sstate_rundepfilter(siggen, fn, recipename, task, dep, depname, dataCache):
     if isPackageGroup(fn):
         return False  
 
-    # Drop native/cross/nativesdk dependencies from target recipes
-    if isNative(depname) or isCross(depname) or isNativeSDK(depname):
-        return False
-
     # Exclude well defined machine specific configurations which don't change ABI
     if depname in siggen.abisaferecipes and not isImage(fn):
         return False
-- 
1.8.5.2

Re: [PATCH 1/2] sstatesig: include native/cross/nativesdk deps in target signatures

[Richard Purdie]

On Sat, 2014-01-18 at 15:02 +0100, Martin Jansa wrote:
> * I don't have any real evidence or good statistics for this, but when
>   comparing signature dumps from my big bitbake world builds I usually
>   see a lot of rebuilds caused by changes in .bbclasses and only very
>   rare would be the case where oe-core upgrade brings changes in -native
>   recipes and no change in .bbclasses used from target recipes
> * changing the default to include them shouldn't cause significant
>   increase in rebuilds and sstate reuse a bit safer
> * people working on toolchain (e.g. using gcc from AUTOREV) can easily
>   extend sstate_rundepfilter to ignore them again (it's easier than
>   removing existing filter), example how add own signature handler in
>   your layer is here:
>   https://github.com/openwebos/meta-webos/commit/9ac3a7c803e7793b3274e4998f167b6278db8042
> 
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  meta/lib/oe/sstatesig.py | 4 ----
>  1 file changed, 4 deletions(-)

Whilst I merged this, we do have a problem here. This change means that
the sstate is locked to the build system architecture.

For example, if you have a 32 bit machine, it can't use any of the
sstate built on a 64 bit machine and vice versa.

I'm not sure what we do about this but its a problem...

Cheers,

Richard

> diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
> index 3011f16..8b0e344 100644
> --- a/meta/lib/oe/sstatesig.py
> +++ b/meta/lib/oe/sstatesig.py
> @@ -36,10 +36,6 @@ def sstate_rundepfilter(siggen, fn, recipename, task, dep, depname, dataCache):
>      if isPackageGroup(fn):
>          return False  
>  
> -    # Drop native/cross/nativesdk dependencies from target recipes
> -    if isNative(depname) or isCross(depname) or isNativeSDK(depname):
> -        return False
> -
>      # Exclude well defined machine specific configurations which don't change ABI
>      if depname in siggen.abisaferecipes and not isImage(fn):
>          return False

Re: [PATCH 1/2] sstatesig: include native/cross/nativesdk deps in target signatures

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2535 bytes --]

On Mon, Jan 27, 2014 at 02:51:15PM +0000, Richard Purdie wrote:
> On Sat, 2014-01-18 at 15:02 +0100, Martin Jansa wrote:
> > * I don't have any real evidence or good statistics for this, but when
> >   comparing signature dumps from my big bitbake world builds I usually
> >   see a lot of rebuilds caused by changes in .bbclasses and only very
> >   rare would be the case where oe-core upgrade brings changes in -native
> >   recipes and no change in .bbclasses used from target recipes
> > * changing the default to include them shouldn't cause significant
> >   increase in rebuilds and sstate reuse a bit safer
> > * people working on toolchain (e.g. using gcc from AUTOREV) can easily
> >   extend sstate_rundepfilter to ignore them again (it's easier than
> >   removing existing filter), example how add own signature handler in
> >   your layer is here:
> >   https://github.com/openwebos/meta-webos/commit/9ac3a7c803e7793b3274e4998f167b6278db8042
> > 
> > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> > ---
> >  meta/lib/oe/sstatesig.py | 4 ----
> >  1 file changed, 4 deletions(-)
> 
> Whilst I merged this, we do have a problem here. This change means that
> the sstate is locked to the build system architecture.
> 
> For example, if you have a 32 bit machine, it can't use any of the
> sstate built on a 64 bit machine and vice versa.
> 
> I'm not sure what we do about this but its a problem...

Agreed, that's bad, I was testing it in setup where we publish the
sstate archive from the builder with the same architecture and have
similar setup building just -native recipes on 32bit for 32bit devs.

So this problem should have hit us as well, I'll recheck.

> Cheers,
> 
> Richard
> 
> > diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
> > index 3011f16..8b0e344 100644
> > --- a/meta/lib/oe/sstatesig.py
> > +++ b/meta/lib/oe/sstatesig.py
> > @@ -36,10 +36,6 @@ def sstate_rundepfilter(siggen, fn, recipename, task, dep, depname, dataCache):
> >      if isPackageGroup(fn):
> >          return False  
> >  
> > -    # Drop native/cross/nativesdk dependencies from target recipes
> > -    if isNative(depname) or isCross(depname) or isNativeSDK(depname):
> > -        return False
> > -
> >      # Exclude well defined machine specific configurations which don't change ABI
> >      if depname in siggen.abisaferecipes and not isImage(fn):
> >          return False
> 
> 

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

[PATCH 2/2] allarch: Set empty TARGET_PREFIX and TARGET_FPU

[Martin Jansa]

* set empty TARGET_PREFIX
  This has a bit weird reason caused by unsupported setup where
  external-toolchain is used in some DISTRO only for some MACHINEs
  and internal is used for other MACHINEs.
  Because external-toolchain usually comes with different TARGET_PREFIX
  it was causing allarch recipes to have different signatures even
  when they don't use toolchain at all.
  Empty TARGET_PREFIX also helps to find allarch recipes which still
  have default dependency on e.g. virtual/${TARGET_PREFIX}gcc.
* add TARGET_FPU just for completeness (it was used in icecc.bbclass
  but now it's vardepexcluded there as well)

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 meta/classes/allarch.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/allarch.bbclass b/meta/classes/allarch.bbclass
index 3ef1e9c..d41dd4b 100644
--- a/meta/classes/allarch.bbclass
+++ b/meta/classes/allarch.bbclass
@@ -22,6 +22,8 @@ python () {
         d.setVar("TARGET_CC_ARCH", "none")
         d.setVar("TARGET_LD_ARCH", "none")
         d.setVar("TARGET_AS_ARCH", "none")
+        d.setVar("TARGET_FPU", "")
+        d.setVar("TARGET_PREFIX", "")
         d.setVar("PACKAGE_EXTRA_ARCHS", "")
         d.setVar("SDK_ARCH", "none")
         d.setVar("SDK_CC_ARCH", "none")
-- 
1.8.5.2