[PATCH] bitbake.conf: define shared PERSISTENT_DIR

[PATCH] bitbake.conf: define shared PERSISTENT_DIR

[Martin Jansa]

* import from poky
  commit 6a11cf7dfe930461a6660e3b783b546fa2634900
  Author: Richard Purdie <rpurdie@linux.intel.com>
  Date:   Sun May 31 22:23:30 2009 +0100
* we had CACHE var pointing to machine specific dir since
  commit caf077679022f37ce55d758101f130e4e93bd7b5
  Author: Richard Purdie <rpurdie@rpsys.net>
  Date:   Mon Nov 20 17:40:20 2006 +0000

* current bitbake is looking for cache dir like this:
  cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or
              bb.data.getVar("CACHE", d, True))
  which without PERSISTENT_DIR defined picks machine-specific CACHE dir
  so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch
  in same builddir :/

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 conf/bitbake.conf |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)

diff --git a/conf/bitbake.conf b/conf/bitbake.conf
index dcc5c70..53bb7b7 100644
--- a/conf/bitbake.conf
+++ b/conf/bitbake.conf
@@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}"
 
 TMPDIR = "${TOPDIR}/tmp"
 CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}"
+# The persistent cache should be shared by all builds
+PERSISTENT_DIR = "${TMPDIR}/cache"
 CO_DIR = "${DL_DIR}"
 CVSDIR = "${CO_DIR}/cvs"
 SVNDIR = "${CO_DIR}/svn"
-- 
1.7.4.1

Re: [PATCH] bitbake.conf: define shared PERSISTENT_DIR

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 3691 bytes --]

On Thu, Feb 24, 2011 at 04:25:56PM +0100, Martin Jansa wrote:
> * import from poky
>   commit 6a11cf7dfe930461a6660e3b783b546fa2634900
>   Author: Richard Purdie <rpurdie@linux.intel.com>
>   Date:   Sun May 31 22:23:30 2009 +0100
> * we had CACHE var pointing to machine specific dir since
>   commit caf077679022f37ce55d758101f130e4e93bd7b5
>   Author: Richard Purdie <rpurdie@rpsys.net>
>   Date:   Mon Nov 20 17:40:20 2006 +0000
> 
> * current bitbake is looking for cache dir like this:
>   cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or
>               bb.data.getVar("CACHE", d, True))
>   which without PERSISTENT_DIR defined picks machine-specific CACHE dir
>   so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch
>   in same builddir :/
> 
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  conf/bitbake.conf |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)
> 
> diff --git a/conf/bitbake.conf b/conf/bitbake.conf
> index dcc5c70..53bb7b7 100644
> --- a/conf/bitbake.conf
> +++ b/conf/bitbake.conf
> @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}"
>  
>  TMPDIR = "${TOPDIR}/tmp"
>  CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}"
> +# The persistent cache should be shared by all builds
> +PERSISTENT_DIR = "${TMPDIR}/cache"

Or even better we can move it outside ${TMPDIR} and add sanity check to
warn user that he still has ${CACHE}/bb_persist_data.sqlite3 and should
move it to ${PERSISTENT_DIR}/bb_persist_data.sqlite3. (I'll send such
patch if requested).

But while thinking about it I found that there are probably other issues
with that too.

Machine-specific persistent cache was rebuilding packages with
FEED_ARCH, which works ok only as long as all machines started with same
cache and all machines are always built (so _count are incremented
consistently in all caches).

Shared persistent cache is much better, but still fails if some machine
is overriding SRCREV_machine ie in u-boot_git it's quite common with
same git SRC_URI and same branch (so key in persistent cache is the
same, but _rev is different so _count is incremented twice every time
this _machine is built. But overriding SRCREV is less common than
multiple machines building _same_ package with FEED_ARCH or even "all".

It could be maybe improved if we add actual PACKAGE_ARCH to key, but I
think that's not available while bitbake is parsing recipe (ie before
calling utils.is_machine_specific and changing PACKAGE_ARCH to
MACHINE_ARCH.

This improved cache would provide consistent _count, but will still fail
on target. ie when SRCREV for FEED_ARCH was changed twice and for
om-gta02 just once:

Package: frameworkd-config
Version: 1:0.9.5.9+gitr2+e31fa77906981a56be2271246693fd929c973abf-r12.6
Architecture: armv4t

will be preferred instead of

Package: frameworkd-config
Version: 1:0.9.5.9+gitr1+aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-r12.6
Architecture: om-gta02

even with:
SHR root@gjama / $ cat /etc/opkg/arch.conf 
..
arch armv4t 26
arch om-gta02 31

because afaik opkg prefers newer version before higher arch priority.

I don't see how to fix this in all corner cases :/.
But shared PERSISTENT_DIR outside TMPDIR seems much better than machine
specific in CACHE.

BTW: I'm quite frustrated lately to see that OE is great only for 99%,
but last 1% of corner cases like this and ie substrings in packages with
debian.bbclass are quite hard to find/debug/explain/fix :(.

Regards,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

[PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR

[Martin Jansa]

* import from poky 6a11cf7dfe930461a6660e3b783b546fa2634900
* we had CACHE var pointing to machine specific dir since 2006
  caf077679022f37ce55d758101f130e4e93bd7b5
* current bitbake is looking for cache dir like this:
  cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or
              bb.data.getVar("CACHE", d, True))
  which without PERSISTENT_DIR defined picks machine-specific CACHE dir
  so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch
  in same builddir :/

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 classes/sanity.bbclass |   12 ++++++++++++
 conf/bitbake.conf      |    2 ++
 conf/sanity.conf       |    1 +
 3 files changed, 15 insertions(+), 0 deletions(-)

diff --git a/classes/sanity.bbclass b/classes/sanity.bbclass
index 1e6b170..ffe3a49 100644
--- a/classes/sanity.bbclass
+++ b/classes/sanity.bbclass
@@ -189,6 +189,18 @@ def check_sanity(e):
 	if messages != "":
 		raise_sanity_error(messages)
 
+	#
+	# Check there isn't old persistent cache
+	#
+	cache = data.getVar('CACHE', e.data, True)
+	persistent_dir = data.getVar('PERSISTENT_DIR', e.data, True)
+	persistent_cache_filename = data.getVar('SANITY_PERSIST_DATA_FILE', e.data, True)
+	if os.path.exists(cache + '/' + persistent_cache_filename):
+		messages = messages + "Error, persistent cache file '%s' exists in old location '%s', please migrate it to new location '%s' and merge them together if you have one for each MACHINE.\n" % (persistent_cache_filename, cache, persistent_dir)
+
+	if messages != "":
+		raise_sanity_error(messages)
+
 python check_sanity_eventhandler() {
     if isinstance(e, bb.event.BuildStarted):
         check_sanity(e)
diff --git a/conf/bitbake.conf b/conf/bitbake.conf
index 88ad2ba..0c11cdb 100644
--- a/conf/bitbake.conf
+++ b/conf/bitbake.conf
@@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}"
 
 TMPDIR = "${TOPDIR}/tmp"
 CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}"
+# The persistent cache should be shared by all builds
+PERSISTENT_DIR = "${TOPDIR}/conf/persistent"
 CO_DIR = "${DL_DIR}"
 CVSDIR = "${CO_DIR}/cvs"
 SVNDIR = "${CO_DIR}/svn"
diff --git a/conf/sanity.conf b/conf/sanity.conf
index e9902f6..c1ac332 100644
--- a/conf/sanity.conf
+++ b/conf/sanity.conf
@@ -7,5 +7,6 @@ BB_MIN_VERSION = "1.10.2"
 
 SANITY_ABIFILE = "${TMPDIR}/abi_version"
 SANITY_PRFILE = "${TMPDIR}/distro_pr"
+SANITY_PERSIST_DATA_FILE = "bb_persist_data.sqlite3"
 
 INHERIT += "sanity"
-- 
1.7.4.1

Re: [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2879 bytes --]

On Fri, Feb 25, 2011 at 04:27:16PM +0100, Martin Jansa wrote:
> * import from poky 6a11cf7dfe930461a6660e3b783b546fa2634900
> * we had CACHE var pointing to machine specific dir since 2006
>   caf077679022f37ce55d758101f130e4e93bd7b5
> * current bitbake is looking for cache dir like this:
>   cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or
>               bb.data.getVar("CACHE", d, True))
>   which without PERSISTENT_DIR defined picks machine-specific CACHE dir
>   so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch
>   in same builddir :/

ping?

> 
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
> ---
>  classes/sanity.bbclass |   12 ++++++++++++
>  conf/bitbake.conf      |    2 ++
>  conf/sanity.conf       |    1 +
>  3 files changed, 15 insertions(+), 0 deletions(-)
> 
> diff --git a/classes/sanity.bbclass b/classes/sanity.bbclass
> index 1e6b170..ffe3a49 100644
> --- a/classes/sanity.bbclass
> +++ b/classes/sanity.bbclass
> @@ -189,6 +189,18 @@ def check_sanity(e):
>  	if messages != "":
>  		raise_sanity_error(messages)
>  
> +	#
> +	# Check there isn't old persistent cache
> +	#
> +	cache = data.getVar('CACHE', e.data, True)
> +	persistent_dir = data.getVar('PERSISTENT_DIR', e.data, True)
> +	persistent_cache_filename = data.getVar('SANITY_PERSIST_DATA_FILE', e.data, True)
> +	if os.path.exists(cache + '/' + persistent_cache_filename):
> +		messages = messages + "Error, persistent cache file '%s' exists in old location '%s', please migrate it to new location '%s' and merge them together if you have one for each MACHINE.\n" % (persistent_cache_filename, cache, persistent_dir)
> +
> +	if messages != "":
> +		raise_sanity_error(messages)
> +
>  python check_sanity_eventhandler() {
>      if isinstance(e, bb.event.BuildStarted):
>          check_sanity(e)
> diff --git a/conf/bitbake.conf b/conf/bitbake.conf
> index 88ad2ba..0c11cdb 100644
> --- a/conf/bitbake.conf
> +++ b/conf/bitbake.conf
> @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}"
>  
>  TMPDIR = "${TOPDIR}/tmp"
>  CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}"
> +# The persistent cache should be shared by all builds
> +PERSISTENT_DIR = "${TOPDIR}/conf/persistent"
>  CO_DIR = "${DL_DIR}"
>  CVSDIR = "${CO_DIR}/cvs"
>  SVNDIR = "${CO_DIR}/svn"
> diff --git a/conf/sanity.conf b/conf/sanity.conf
> index e9902f6..c1ac332 100644
> --- a/conf/sanity.conf
> +++ b/conf/sanity.conf
> @@ -7,5 +7,6 @@ BB_MIN_VERSION = "1.10.2"
>  
>  SANITY_ABIFILE = "${TMPDIR}/abi_version"
>  SANITY_PRFILE = "${TMPDIR}/distro_pr"
> +SANITY_PERSIST_DATA_FILE = "bb_persist_data.sqlite3"
>  
>  INHERIT += "sanity"
> -- 
> 1.7.4.1
> 

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR

[Tom Rini]

On 03/03/2011 12:27 AM, Martin Jansa wrote:
> On Fri, Feb 25, 2011 at 04:27:16PM +0100, Martin Jansa wrote:
>> * import from poky 6a11cf7dfe930461a6660e3b783b546fa2634900
>> * we had CACHE var pointing to machine specific dir since 2006
>>    caf077679022f37ce55d758101f130e4e93bd7b5
>> * current bitbake is looking for cache dir like this:
>>    cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or
>>                bb.data.getVar("CACHE", d, True))
>>    which without PERSISTENT_DIR defined picks machine-specific CACHE dir
>>    so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch
>>    in same builddir :/
>
> ping?
>

I'd like Richards comments here.

>>
>> Signed-off-by: Martin Jansa<Martin.Jansa@gmail.com>
>> ---
>>   classes/sanity.bbclass |   12 ++++++++++++
>>   conf/bitbake.conf      |    2 ++
>>   conf/sanity.conf       |    1 +
>>   3 files changed, 15 insertions(+), 0 deletions(-)
>>
>> diff --git a/classes/sanity.bbclass b/classes/sanity.bbclass
>> index 1e6b170..ffe3a49 100644
>> --- a/classes/sanity.bbclass
>> +++ b/classes/sanity.bbclass
>> @@ -189,6 +189,18 @@ def check_sanity(e):
>>   	if messages != "":
>>   		raise_sanity_error(messages)
>>
>> +	#
>> +	# Check there isn't old persistent cache
>> +	#
>> +	cache = data.getVar('CACHE', e.data, True)
>> +	persistent_dir = data.getVar('PERSISTENT_DIR', e.data, True)
>> +	persistent_cache_filename = data.getVar('SANITY_PERSIST_DATA_FILE', e.data, True)
>> +	if os.path.exists(cache + '/' + persistent_cache_filename):
>> +		messages = messages + "Error, persistent cache file '%s' exists in old location '%s', please migrate it to new location '%s' and merge them together if you have one for each MACHINE.\n" % (persistent_cache_filename, cache, persistent_dir)
>> +
>> +	if messages != "":
>> +		raise_sanity_error(messages)
>> +
>>   python check_sanity_eventhandler() {
>>       if isinstance(e, bb.event.BuildStarted):
>>           check_sanity(e)
>> diff --git a/conf/bitbake.conf b/conf/bitbake.conf
>> index 88ad2ba..0c11cdb 100644
>> --- a/conf/bitbake.conf
>> +++ b/conf/bitbake.conf
>> @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}"
>>
>>   TMPDIR = "${TOPDIR}/tmp"
>>   CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}"
>> +# The persistent cache should be shared by all builds
>> +PERSISTENT_DIR = "${TOPDIR}/conf/persistent"
>>   CO_DIR = "${DL_DIR}"
>>   CVSDIR = "${CO_DIR}/cvs"
>>   SVNDIR = "${CO_DIR}/svn"
>> diff --git a/conf/sanity.conf b/conf/sanity.conf
>> index e9902f6..c1ac332 100644
>> --- a/conf/sanity.conf
>> +++ b/conf/sanity.conf
>> @@ -7,5 +7,6 @@ BB_MIN_VERSION = "1.10.2"
>>
>>   SANITY_ABIFILE = "${TMPDIR}/abi_version"
>>   SANITY_PRFILE = "${TMPDIR}/distro_pr"
>> +SANITY_PERSIST_DATA_FILE = "bb_persist_data.sqlite3"
>>
>>   INHERIT += "sanity"
>> --
>> 1.7.4.1
>>
>
>
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel


-- 
Tom Rini
Mentor Graphics Corporation

Re: [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR

[Richard Purdie]

On Thu, 2011-03-03 at 07:38 -0700, Tom Rini wrote:
> On 03/03/2011 12:27 AM, Martin Jansa wrote:
> > On Fri, Feb 25, 2011 at 04:27:16PM +0100, Martin Jansa wrote:
> >> * import from poky 6a11cf7dfe930461a6660e3b783b546fa2634900
> >> * we had CACHE var pointing to machine specific dir since 2006
> >>    caf077679022f37ce55d758101f130e4e93bd7b5
> >> * current bitbake is looking for cache dir like this:
> >>    cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or
> >>                bb.data.getVar("CACHE", d, True))
> >>    which without PERSISTENT_DIR defined picks machine-specific CACHE dir
> >>    so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch
> >>    in same builddir :/
> >
> > ping?
> >
> 
> I'd like Richards comments here.

The intent was always to have CACHE be machine specific and
PERSISTENT_DIR not be machine specific. This means all builds share the
idea of revision numbers yet the main cache is machine specific so you
don't hit reparses when switching machines unnecessarily.

So yes, I think the change is valid, I hadn't realised OE didn't have
that change and I'm happy to ack it...

Cheers,

Richard


> >>
> >> Signed-off-by: Martin Jansa<Martin.Jansa@gmail.com>
> >> ---
> >>   classes/sanity.bbclass |   12 ++++++++++++
> >>   conf/bitbake.conf      |    2 ++
> >>   conf/sanity.conf       |    1 +
> >>   3 files changed, 15 insertions(+), 0 deletions(-)
> >>
> >> diff --git a/classes/sanity.bbclass b/classes/sanity.bbclass
> >> index 1e6b170..ffe3a49 100644
> >> --- a/classes/sanity.bbclass
> >> +++ b/classes/sanity.bbclass
> >> @@ -189,6 +189,18 @@ def check_sanity(e):
> >>   	if messages != "":
> >>   		raise_sanity_error(messages)
> >>
> >> +	#
> >> +	# Check there isn't old persistent cache
> >> +	#
> >> +	cache = data.getVar('CACHE', e.data, True)
> >> +	persistent_dir = data.getVar('PERSISTENT_DIR', e.data, True)
> >> +	persistent_cache_filename = data.getVar('SANITY_PERSIST_DATA_FILE', e.data, True)
> >> +	if os.path.exists(cache + '/' + persistent_cache_filename):
> >> +		messages = messages + "Error, persistent cache file '%s' exists in old location '%s', please migrate it to new location '%s' and merge them together if you have one for each MACHINE.\n" % (persistent_cache_filename, cache, persistent_dir)
> >> +
> >> +	if messages != "":
> >> +		raise_sanity_error(messages)
> >> +
> >>   python check_sanity_eventhandler() {
> >>       if isinstance(e, bb.event.BuildStarted):
> >>           check_sanity(e)
> >> diff --git a/conf/bitbake.conf b/conf/bitbake.conf
> >> index 88ad2ba..0c11cdb 100644
> >> --- a/conf/bitbake.conf
> >> +++ b/conf/bitbake.conf
> >> @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}"
> >>
> >>   TMPDIR = "${TOPDIR}/tmp"
> >>   CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}"
> >> +# The persistent cache should be shared by all builds
> >> +PERSISTENT_DIR = "${TOPDIR}/conf/persistent"
> >>   CO_DIR = "${DL_DIR}"
> >>   CVSDIR = "${CO_DIR}/cvs"
> >>   SVNDIR = "${CO_DIR}/svn"
> >> diff --git a/conf/sanity.conf b/conf/sanity.conf
> >> index e9902f6..c1ac332 100644
> >> --- a/conf/sanity.conf
> >> +++ b/conf/sanity.conf
> >> @@ -7,5 +7,6 @@ BB_MIN_VERSION = "1.10.2"
> >>
> >>   SANITY_ABIFILE = "${TMPDIR}/abi_version"
> >>   SANITY_PRFILE = "${TMPDIR}/distro_pr"
> >> +SANITY_PERSIST_DATA_FILE = "bb_persist_data.sqlite3"
> >>
> >>   INHERIT += "sanity"
> >> --
> >> 1.7.4.1
> >>
> >
> >
> >
> > _______________________________________________
> > Openembedded-devel mailing list
> > Openembedded-devel@lists.openembedded.org
> > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
> 
> 

Re: [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 4916 bytes --]

On Fri, Mar 04, 2011 at 12:11:46AM +0000, Richard Purdie wrote:
> On Thu, 2011-03-03 at 07:38 -0700, Tom Rini wrote:
> > On 03/03/2011 12:27 AM, Martin Jansa wrote:
> > > On Fri, Feb 25, 2011 at 04:27:16PM +0100, Martin Jansa wrote:
> > >> * import from poky 6a11cf7dfe930461a6660e3b783b546fa2634900
> > >> * we had CACHE var pointing to machine specific dir since 2006
> > >>    caf077679022f37ce55d758101f130e4e93bd7b5
> > >> * current bitbake is looking for cache dir like this:
> > >>    cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or
> > >>                bb.data.getVar("CACHE", d, True))
> > >>    which without PERSISTENT_DIR defined picks machine-specific CACHE dir
> > >>    so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch
> > >>    in same builddir :/
> > >
> > > ping?
> > >
> > 
> > I'd like Richards comments here.
> 
> The intent was always to have CACHE be machine specific and
> PERSISTENT_DIR not be machine specific. This means all builds share the

> idea of revision numbers yet the main cache is machine specific so you
> don't hit reparses when switching machines unnecessarily.

BTW: with bitbake master and shared CACHE on one system I noticed that
if I set MACHINE/DISTRO only in shell environment (with correct
BB_ENV_EXTRAWHITE), then cache is not invalidated and metadata reparsed,
resulting in broken builds where parts were respecting ${MACHINE} from
shell and some parts had different machine from previous build.

IIRC I noticed it on different machine listed in build summary and then
another one in filename of machine-specific ipk. Removing cache after
switching MACHINE in shell env worked fine.

Regards,

> So yes, I think the change is valid, I hadn't realised OE didn't have
> that change and I'm happy to ack it...
> 
> Cheers,
> 
> Richard
> 
> 
> > >>
> > >> Signed-off-by: Martin Jansa<Martin.Jansa@gmail.com>
> > >> ---
> > >>   classes/sanity.bbclass |   12 ++++++++++++
> > >>   conf/bitbake.conf      |    2 ++
> > >>   conf/sanity.conf       |    1 +
> > >>   3 files changed, 15 insertions(+), 0 deletions(-)
> > >>
> > >> diff --git a/classes/sanity.bbclass b/classes/sanity.bbclass
> > >> index 1e6b170..ffe3a49 100644
> > >> --- a/classes/sanity.bbclass
> > >> +++ b/classes/sanity.bbclass
> > >> @@ -189,6 +189,18 @@ def check_sanity(e):
> > >>   	if messages != "":
> > >>   		raise_sanity_error(messages)
> > >>
> > >> +	#
> > >> +	# Check there isn't old persistent cache
> > >> +	#
> > >> +	cache = data.getVar('CACHE', e.data, True)
> > >> +	persistent_dir = data.getVar('PERSISTENT_DIR', e.data, True)
> > >> +	persistent_cache_filename = data.getVar('SANITY_PERSIST_DATA_FILE', e.data, True)
> > >> +	if os.path.exists(cache + '/' + persistent_cache_filename):
> > >> +		messages = messages + "Error, persistent cache file '%s' exists in old location '%s', please migrate it to new location '%s' and merge them together if you have one for each MACHINE.\n" % (persistent_cache_filename, cache, persistent_dir)
> > >> +
> > >> +	if messages != "":
> > >> +		raise_sanity_error(messages)
> > >> +
> > >>   python check_sanity_eventhandler() {
> > >>       if isinstance(e, bb.event.BuildStarted):
> > >>           check_sanity(e)
> > >> diff --git a/conf/bitbake.conf b/conf/bitbake.conf
> > >> index 88ad2ba..0c11cdb 100644
> > >> --- a/conf/bitbake.conf
> > >> +++ b/conf/bitbake.conf
> > >> @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}"
> > >>
> > >>   TMPDIR = "${TOPDIR}/tmp"
> > >>   CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}"
> > >> +# The persistent cache should be shared by all builds
> > >> +PERSISTENT_DIR = "${TOPDIR}/conf/persistent"
> > >>   CO_DIR = "${DL_DIR}"
> > >>   CVSDIR = "${CO_DIR}/cvs"
> > >>   SVNDIR = "${CO_DIR}/svn"
> > >> diff --git a/conf/sanity.conf b/conf/sanity.conf
> > >> index e9902f6..c1ac332 100644
> > >> --- a/conf/sanity.conf
> > >> +++ b/conf/sanity.conf
> > >> @@ -7,5 +7,6 @@ BB_MIN_VERSION = "1.10.2"
> > >>
> > >>   SANITY_ABIFILE = "${TMPDIR}/abi_version"
> > >>   SANITY_PRFILE = "${TMPDIR}/distro_pr"
> > >> +SANITY_PERSIST_DATA_FILE = "bb_persist_data.sqlite3"
> > >>
> > >>   INHERIT += "sanity"
> > >> --
> > >> 1.7.4.1
> > >>
> > >
> > >
> > >
> > > _______________________________________________
> > > Openembedded-devel mailing list
> > > Openembedded-devel@lists.openembedded.org
> > > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
> > 
> > 
> 
> 
> 
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1515 bytes --]

On Fri, Mar 04, 2011 at 12:11:46AM +0000, Richard Purdie wrote:
> On Thu, 2011-03-03 at 07:38 -0700, Tom Rini wrote:
> > On 03/03/2011 12:27 AM, Martin Jansa wrote:
> > > On Fri, Feb 25, 2011 at 04:27:16PM +0100, Martin Jansa wrote:
> > >> * import from poky 6a11cf7dfe930461a6660e3b783b546fa2634900
> > >> * we had CACHE var pointing to machine specific dir since 2006
> > >>    caf077679022f37ce55d758101f130e4e93bd7b5
> > >> * current bitbake is looking for cache dir like this:
> > >>    cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or
> > >>                bb.data.getVar("CACHE", d, True))
> > >>    which without PERSISTENT_DIR defined picks machine-specific CACHE dir
> > >>    so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch
> > >>    in same builddir :/
> > >
> > > ping?
> > >
> > 
> > I'd like Richards comments here.
> 
> The intent was always to have CACHE be machine specific and
> PERSISTENT_DIR not be machine specific. This means all builds share the
> idea of revision numbers yet the main cache is machine specific so you
> don't hit reparses when switching machines unnecessarily.
> 
> So yes, I think the change is valid, I hadn't realised OE didn't have
> that change and I'm happy to ack it...

Can I get at least one more ACK?

And if we all agree on new location
"${TOPDIR}/conf/persistent"
then I would send similar patch to oe-core and move it too

Regards,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH] bitbake.conf: define shared PERSISTENT_DIR

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1959 bytes --]

On Thu, Feb 24, 2011 at 04:25:56PM +0100, Martin Jansa wrote:
> * import from poky
>   commit 6a11cf7dfe930461a6660e3b783b546fa2634900
>   Author: Richard Purdie <rpurdie@linux.intel.com>
>   Date:   Sun May 31 22:23:30 2009 +0100
> * we had CACHE var pointing to machine specific dir since
>   commit caf077679022f37ce55d758101f130e4e93bd7b5
>   Author: Richard Purdie <rpurdie@rpsys.net>
>   Date:   Mon Nov 20 17:40:20 2006 +0000
> 
> * current bitbake is looking for cache dir like this:
>   cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or
>               bb.data.getVar("CACHE", d, True))
>   which without PERSISTENT_DIR defined picks machine-specific CACHE dir
>   so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch
>   in same builddir :/
> 
> Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>

ping?

2nd version which moved it outside TMPDIR
[oe] [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR
was sort of rejected:
http://lists.linuxtogo.org/pipermail/openembedded-core/2011-March/000298.html

but this part without moving it outside TMPDIR is still valid and I'll
push it after month of review if nobody objects :).

> ---
>  conf/bitbake.conf |    2 ++
>  1 files changed, 2 insertions(+), 0 deletions(-)
> 
> diff --git a/conf/bitbake.conf b/conf/bitbake.conf
> index dcc5c70..53bb7b7 100644
> --- a/conf/bitbake.conf
> +++ b/conf/bitbake.conf
> @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}"
>  
>  TMPDIR = "${TOPDIR}/tmp"
>  CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}"
> +# The persistent cache should be shared by all builds
> +PERSISTENT_DIR = "${TMPDIR}/cache"
>  CO_DIR = "${DL_DIR}"
>  CVSDIR = "${CO_DIR}/cvs"
>  SVNDIR = "${CO_DIR}/svn"
> -- 
> 1.7.4.1
> 

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --]