* [PATCH] bitbake.conf: define shared PERSISTENT_DIR
@ 2011-02-24 15:25 Martin Jansa
2011-02-24 16:45 ` Martin Jansa
2011-03-21 9:52 ` [PATCH] bitbake.conf: define shared PERSISTENT_DIR Martin Jansa
0 siblings, 2 replies; 9+ messages in thread
From: Martin Jansa @ 2011-02-24 15:25 UTC (permalink / raw)
To: openembedded-devel
* import from poky
commit 6a11cf7dfe930461a6660e3b783b546fa2634900
Author: Richard Purdie <rpurdie@linux.intel.com>
Date: Sun May 31 22:23:30 2009 +0100
* we had CACHE var pointing to machine specific dir since
commit caf077679022f37ce55d758101f130e4e93bd7b5
Author: Richard Purdie <rpurdie@rpsys.net>
Date: Mon Nov 20 17:40:20 2006 +0000
* current bitbake is looking for cache dir like this:
cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or
bb.data.getVar("CACHE", d, True))
which without PERSISTENT_DIR defined picks machine-specific CACHE dir
so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch
in same builddir :/
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
conf/bitbake.conf | 2 ++
1 files changed, 2 insertions(+), 0 deletions(-)
diff --git a/conf/bitbake.conf b/conf/bitbake.conf
index dcc5c70..53bb7b7 100644
--- a/conf/bitbake.conf
+++ b/conf/bitbake.conf
@@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}"
TMPDIR = "${TOPDIR}/tmp"
CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}"
+# The persistent cache should be shared by all builds
+PERSISTENT_DIR = "${TMPDIR}/cache"
CO_DIR = "${DL_DIR}"
CVSDIR = "${CO_DIR}/cvs"
SVNDIR = "${CO_DIR}/svn"
--
1.7.4.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH] bitbake.conf: define shared PERSISTENT_DIR 2011-02-24 15:25 [PATCH] bitbake.conf: define shared PERSISTENT_DIR Martin Jansa @ 2011-02-24 16:45 ` Martin Jansa 2011-02-25 15:27 ` [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR Martin Jansa 2011-03-21 9:52 ` [PATCH] bitbake.conf: define shared PERSISTENT_DIR Martin Jansa 1 sibling, 1 reply; 9+ messages in thread From: Martin Jansa @ 2011-02-24 16:45 UTC (permalink / raw) To: openembedded-devel [-- Attachment #1: Type: text/plain, Size: 3691 bytes --] On Thu, Feb 24, 2011 at 04:25:56PM +0100, Martin Jansa wrote: > * import from poky > commit 6a11cf7dfe930461a6660e3b783b546fa2634900 > Author: Richard Purdie <rpurdie@linux.intel.com> > Date: Sun May 31 22:23:30 2009 +0100 > * we had CACHE var pointing to machine specific dir since > commit caf077679022f37ce55d758101f130e4e93bd7b5 > Author: Richard Purdie <rpurdie@rpsys.net> > Date: Mon Nov 20 17:40:20 2006 +0000 > > * current bitbake is looking for cache dir like this: > cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or > bb.data.getVar("CACHE", d, True)) > which without PERSISTENT_DIR defined picks machine-specific CACHE dir > so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch > in same builddir :/ > > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> > --- > conf/bitbake.conf | 2 ++ > 1 files changed, 2 insertions(+), 0 deletions(-) > > diff --git a/conf/bitbake.conf b/conf/bitbake.conf > index dcc5c70..53bb7b7 100644 > --- a/conf/bitbake.conf > +++ b/conf/bitbake.conf > @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}" > > TMPDIR = "${TOPDIR}/tmp" > CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}" > +# The persistent cache should be shared by all builds > +PERSISTENT_DIR = "${TMPDIR}/cache" Or even better we can move it outside ${TMPDIR} and add sanity check to warn user that he still has ${CACHE}/bb_persist_data.sqlite3 and should move it to ${PERSISTENT_DIR}/bb_persist_data.sqlite3. (I'll send such patch if requested). But while thinking about it I found that there are probably other issues with that too. Machine-specific persistent cache was rebuilding packages with FEED_ARCH, which works ok only as long as all machines started with same cache and all machines are always built (so _count are incremented consistently in all caches). Shared persistent cache is much better, but still fails if some machine is overriding SRCREV_machine ie in u-boot_git it's quite common with same git SRC_URI and same branch (so key in persistent cache is the same, but _rev is different so _count is incremented twice every time this _machine is built. But overriding SRCREV is less common than multiple machines building _same_ package with FEED_ARCH or even "all". It could be maybe improved if we add actual PACKAGE_ARCH to key, but I think that's not available while bitbake is parsing recipe (ie before calling utils.is_machine_specific and changing PACKAGE_ARCH to MACHINE_ARCH. This improved cache would provide consistent _count, but will still fail on target. ie when SRCREV for FEED_ARCH was changed twice and for om-gta02 just once: Package: frameworkd-config Version: 1:0.9.5.9+gitr2+e31fa77906981a56be2271246693fd929c973abf-r12.6 Architecture: armv4t will be preferred instead of Package: frameworkd-config Version: 1:0.9.5.9+gitr1+aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa-r12.6 Architecture: om-gta02 even with: SHR root@gjama / $ cat /etc/opkg/arch.conf .. arch armv4t 26 arch om-gta02 31 because afaik opkg prefers newer version before higher arch priority. I don't see how to fix this in all corner cases :/. But shared PERSISTENT_DIR outside TMPDIR seems much better than machine specific in CACHE. BTW: I'm quite frustrated lately to see that OE is great only for 99%, but last 1% of corner cases like this and ie substrings in packages with debian.bbclass are quite hard to find/debug/explain/fix :(. Regards, -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR 2011-02-24 16:45 ` Martin Jansa @ 2011-02-25 15:27 ` Martin Jansa 2011-03-03 7:27 ` Martin Jansa 0 siblings, 1 reply; 9+ messages in thread From: Martin Jansa @ 2011-02-25 15:27 UTC (permalink / raw) To: openembedded-devel * import from poky 6a11cf7dfe930461a6660e3b783b546fa2634900 * we had CACHE var pointing to machine specific dir since 2006 caf077679022f37ce55d758101f130e4e93bd7b5 * current bitbake is looking for cache dir like this: cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or bb.data.getVar("CACHE", d, True)) which without PERSISTENT_DIR defined picks machine-specific CACHE dir so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch in same builddir :/ Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> --- classes/sanity.bbclass | 12 ++++++++++++ conf/bitbake.conf | 2 ++ conf/sanity.conf | 1 + 3 files changed, 15 insertions(+), 0 deletions(-) diff --git a/classes/sanity.bbclass b/classes/sanity.bbclass index 1e6b170..ffe3a49 100644 --- a/classes/sanity.bbclass +++ b/classes/sanity.bbclass @@ -189,6 +189,18 @@ def check_sanity(e): if messages != "": raise_sanity_error(messages) + # + # Check there isn't old persistent cache + # + cache = data.getVar('CACHE', e.data, True) + persistent_dir = data.getVar('PERSISTENT_DIR', e.data, True) + persistent_cache_filename = data.getVar('SANITY_PERSIST_DATA_FILE', e.data, True) + if os.path.exists(cache + '/' + persistent_cache_filename): + messages = messages + "Error, persistent cache file '%s' exists in old location '%s', please migrate it to new location '%s' and merge them together if you have one for each MACHINE.\n" % (persistent_cache_filename, cache, persistent_dir) + + if messages != "": + raise_sanity_error(messages) + python check_sanity_eventhandler() { if isinstance(e, bb.event.BuildStarted): check_sanity(e) diff --git a/conf/bitbake.conf b/conf/bitbake.conf index 88ad2ba..0c11cdb 100644 --- a/conf/bitbake.conf +++ b/conf/bitbake.conf @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}" TMPDIR = "${TOPDIR}/tmp" CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}" +# The persistent cache should be shared by all builds +PERSISTENT_DIR = "${TOPDIR}/conf/persistent" CO_DIR = "${DL_DIR}" CVSDIR = "${CO_DIR}/cvs" SVNDIR = "${CO_DIR}/svn" diff --git a/conf/sanity.conf b/conf/sanity.conf index e9902f6..c1ac332 100644 --- a/conf/sanity.conf +++ b/conf/sanity.conf @@ -7,5 +7,6 @@ BB_MIN_VERSION = "1.10.2" SANITY_ABIFILE = "${TMPDIR}/abi_version" SANITY_PRFILE = "${TMPDIR}/distro_pr" +SANITY_PERSIST_DATA_FILE = "bb_persist_data.sqlite3" INHERIT += "sanity" -- 1.7.4.1 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR 2011-02-25 15:27 ` [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR Martin Jansa @ 2011-03-03 7:27 ` Martin Jansa 2011-03-03 14:38 ` Tom Rini 0 siblings, 1 reply; 9+ messages in thread From: Martin Jansa @ 2011-03-03 7:27 UTC (permalink / raw) To: openembedded-devel [-- Attachment #1: Type: text/plain, Size: 2879 bytes --] On Fri, Feb 25, 2011 at 04:27:16PM +0100, Martin Jansa wrote: > * import from poky 6a11cf7dfe930461a6660e3b783b546fa2634900 > * we had CACHE var pointing to machine specific dir since 2006 > caf077679022f37ce55d758101f130e4e93bd7b5 > * current bitbake is looking for cache dir like this: > cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or > bb.data.getVar("CACHE", d, True)) > which without PERSISTENT_DIR defined picks machine-specific CACHE dir > so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch > in same builddir :/ ping? > > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> > --- > classes/sanity.bbclass | 12 ++++++++++++ > conf/bitbake.conf | 2 ++ > conf/sanity.conf | 1 + > 3 files changed, 15 insertions(+), 0 deletions(-) > > diff --git a/classes/sanity.bbclass b/classes/sanity.bbclass > index 1e6b170..ffe3a49 100644 > --- a/classes/sanity.bbclass > +++ b/classes/sanity.bbclass > @@ -189,6 +189,18 @@ def check_sanity(e): > if messages != "": > raise_sanity_error(messages) > > + # > + # Check there isn't old persistent cache > + # > + cache = data.getVar('CACHE', e.data, True) > + persistent_dir = data.getVar('PERSISTENT_DIR', e.data, True) > + persistent_cache_filename = data.getVar('SANITY_PERSIST_DATA_FILE', e.data, True) > + if os.path.exists(cache + '/' + persistent_cache_filename): > + messages = messages + "Error, persistent cache file '%s' exists in old location '%s', please migrate it to new location '%s' and merge them together if you have one for each MACHINE.\n" % (persistent_cache_filename, cache, persistent_dir) > + > + if messages != "": > + raise_sanity_error(messages) > + > python check_sanity_eventhandler() { > if isinstance(e, bb.event.BuildStarted): > check_sanity(e) > diff --git a/conf/bitbake.conf b/conf/bitbake.conf > index 88ad2ba..0c11cdb 100644 > --- a/conf/bitbake.conf > +++ b/conf/bitbake.conf > @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}" > > TMPDIR = "${TOPDIR}/tmp" > CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}" > +# The persistent cache should be shared by all builds > +PERSISTENT_DIR = "${TOPDIR}/conf/persistent" > CO_DIR = "${DL_DIR}" > CVSDIR = "${CO_DIR}/cvs" > SVNDIR = "${CO_DIR}/svn" > diff --git a/conf/sanity.conf b/conf/sanity.conf > index e9902f6..c1ac332 100644 > --- a/conf/sanity.conf > +++ b/conf/sanity.conf > @@ -7,5 +7,6 @@ BB_MIN_VERSION = "1.10.2" > > SANITY_ABIFILE = "${TMPDIR}/abi_version" > SANITY_PRFILE = "${TMPDIR}/distro_pr" > +SANITY_PERSIST_DATA_FILE = "bb_persist_data.sqlite3" > > INHERIT += "sanity" > -- > 1.7.4.1 > -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR 2011-03-03 7:27 ` Martin Jansa @ 2011-03-03 14:38 ` Tom Rini 2011-03-04 0:11 ` Richard Purdie 0 siblings, 1 reply; 9+ messages in thread From: Tom Rini @ 2011-03-03 14:38 UTC (permalink / raw) To: openembedded-devel, Richard Purdie On 03/03/2011 12:27 AM, Martin Jansa wrote: > On Fri, Feb 25, 2011 at 04:27:16PM +0100, Martin Jansa wrote: >> * import from poky 6a11cf7dfe930461a6660e3b783b546fa2634900 >> * we had CACHE var pointing to machine specific dir since 2006 >> caf077679022f37ce55d758101f130e4e93bd7b5 >> * current bitbake is looking for cache dir like this: >> cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or >> bb.data.getVar("CACHE", d, True)) >> which without PERSISTENT_DIR defined picks machine-specific CACHE dir >> so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch >> in same builddir :/ > > ping? > I'd like Richards comments here. >> >> Signed-off-by: Martin Jansa<Martin.Jansa@gmail.com> >> --- >> classes/sanity.bbclass | 12 ++++++++++++ >> conf/bitbake.conf | 2 ++ >> conf/sanity.conf | 1 + >> 3 files changed, 15 insertions(+), 0 deletions(-) >> >> diff --git a/classes/sanity.bbclass b/classes/sanity.bbclass >> index 1e6b170..ffe3a49 100644 >> --- a/classes/sanity.bbclass >> +++ b/classes/sanity.bbclass >> @@ -189,6 +189,18 @@ def check_sanity(e): >> if messages != "": >> raise_sanity_error(messages) >> >> + # >> + # Check there isn't old persistent cache >> + # >> + cache = data.getVar('CACHE', e.data, True) >> + persistent_dir = data.getVar('PERSISTENT_DIR', e.data, True) >> + persistent_cache_filename = data.getVar('SANITY_PERSIST_DATA_FILE', e.data, True) >> + if os.path.exists(cache + '/' + persistent_cache_filename): >> + messages = messages + "Error, persistent cache file '%s' exists in old location '%s', please migrate it to new location '%s' and merge them together if you have one for each MACHINE.\n" % (persistent_cache_filename, cache, persistent_dir) >> + >> + if messages != "": >> + raise_sanity_error(messages) >> + >> python check_sanity_eventhandler() { >> if isinstance(e, bb.event.BuildStarted): >> check_sanity(e) >> diff --git a/conf/bitbake.conf b/conf/bitbake.conf >> index 88ad2ba..0c11cdb 100644 >> --- a/conf/bitbake.conf >> +++ b/conf/bitbake.conf >> @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}" >> >> TMPDIR = "${TOPDIR}/tmp" >> CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}" >> +# The persistent cache should be shared by all builds >> +PERSISTENT_DIR = "${TOPDIR}/conf/persistent" >> CO_DIR = "${DL_DIR}" >> CVSDIR = "${CO_DIR}/cvs" >> SVNDIR = "${CO_DIR}/svn" >> diff --git a/conf/sanity.conf b/conf/sanity.conf >> index e9902f6..c1ac332 100644 >> --- a/conf/sanity.conf >> +++ b/conf/sanity.conf >> @@ -7,5 +7,6 @@ BB_MIN_VERSION = "1.10.2" >> >> SANITY_ABIFILE = "${TMPDIR}/abi_version" >> SANITY_PRFILE = "${TMPDIR}/distro_pr" >> +SANITY_PERSIST_DATA_FILE = "bb_persist_data.sqlite3" >> >> INHERIT += "sanity" >> -- >> 1.7.4.1 >> > > > > _______________________________________________ > Openembedded-devel mailing list > Openembedded-devel@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel -- Tom Rini Mentor Graphics Corporation ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR 2011-03-03 14:38 ` Tom Rini @ 2011-03-04 0:11 ` Richard Purdie 2011-03-04 6:10 ` Martin Jansa 2011-03-04 8:58 ` Martin Jansa 0 siblings, 2 replies; 9+ messages in thread From: Richard Purdie @ 2011-03-04 0:11 UTC (permalink / raw) To: Tom Rini; +Cc: openembedded-devel On Thu, 2011-03-03 at 07:38 -0700, Tom Rini wrote: > On 03/03/2011 12:27 AM, Martin Jansa wrote: > > On Fri, Feb 25, 2011 at 04:27:16PM +0100, Martin Jansa wrote: > >> * import from poky 6a11cf7dfe930461a6660e3b783b546fa2634900 > >> * we had CACHE var pointing to machine specific dir since 2006 > >> caf077679022f37ce55d758101f130e4e93bd7b5 > >> * current bitbake is looking for cache dir like this: > >> cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or > >> bb.data.getVar("CACHE", d, True)) > >> which without PERSISTENT_DIR defined picks machine-specific CACHE dir > >> so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch > >> in same builddir :/ > > > > ping? > > > > I'd like Richards comments here. The intent was always to have CACHE be machine specific and PERSISTENT_DIR not be machine specific. This means all builds share the idea of revision numbers yet the main cache is machine specific so you don't hit reparses when switching machines unnecessarily. So yes, I think the change is valid, I hadn't realised OE didn't have that change and I'm happy to ack it... Cheers, Richard > >> > >> Signed-off-by: Martin Jansa<Martin.Jansa@gmail.com> > >> --- > >> classes/sanity.bbclass | 12 ++++++++++++ > >> conf/bitbake.conf | 2 ++ > >> conf/sanity.conf | 1 + > >> 3 files changed, 15 insertions(+), 0 deletions(-) > >> > >> diff --git a/classes/sanity.bbclass b/classes/sanity.bbclass > >> index 1e6b170..ffe3a49 100644 > >> --- a/classes/sanity.bbclass > >> +++ b/classes/sanity.bbclass > >> @@ -189,6 +189,18 @@ def check_sanity(e): > >> if messages != "": > >> raise_sanity_error(messages) > >> > >> + # > >> + # Check there isn't old persistent cache > >> + # > >> + cache = data.getVar('CACHE', e.data, True) > >> + persistent_dir = data.getVar('PERSISTENT_DIR', e.data, True) > >> + persistent_cache_filename = data.getVar('SANITY_PERSIST_DATA_FILE', e.data, True) > >> + if os.path.exists(cache + '/' + persistent_cache_filename): > >> + messages = messages + "Error, persistent cache file '%s' exists in old location '%s', please migrate it to new location '%s' and merge them together if you have one for each MACHINE.\n" % (persistent_cache_filename, cache, persistent_dir) > >> + > >> + if messages != "": > >> + raise_sanity_error(messages) > >> + > >> python check_sanity_eventhandler() { > >> if isinstance(e, bb.event.BuildStarted): > >> check_sanity(e) > >> diff --git a/conf/bitbake.conf b/conf/bitbake.conf > >> index 88ad2ba..0c11cdb 100644 > >> --- a/conf/bitbake.conf > >> +++ b/conf/bitbake.conf > >> @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}" > >> > >> TMPDIR = "${TOPDIR}/tmp" > >> CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}" > >> +# The persistent cache should be shared by all builds > >> +PERSISTENT_DIR = "${TOPDIR}/conf/persistent" > >> CO_DIR = "${DL_DIR}" > >> CVSDIR = "${CO_DIR}/cvs" > >> SVNDIR = "${CO_DIR}/svn" > >> diff --git a/conf/sanity.conf b/conf/sanity.conf > >> index e9902f6..c1ac332 100644 > >> --- a/conf/sanity.conf > >> +++ b/conf/sanity.conf > >> @@ -7,5 +7,6 @@ BB_MIN_VERSION = "1.10.2" > >> > >> SANITY_ABIFILE = "${TMPDIR}/abi_version" > >> SANITY_PRFILE = "${TMPDIR}/distro_pr" > >> +SANITY_PERSIST_DATA_FILE = "bb_persist_data.sqlite3" > >> > >> INHERIT += "sanity" > >> -- > >> 1.7.4.1 > >> > > > > > > > > _______________________________________________ > > Openembedded-devel mailing list > > Openembedded-devel@lists.openembedded.org > > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel > > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR 2011-03-04 0:11 ` Richard Purdie @ 2011-03-04 6:10 ` Martin Jansa 2011-03-04 8:58 ` Martin Jansa 1 sibling, 0 replies; 9+ messages in thread From: Martin Jansa @ 2011-03-04 6:10 UTC (permalink / raw) To: openembedded-devel [-- Attachment #1: Type: text/plain, Size: 4916 bytes --] On Fri, Mar 04, 2011 at 12:11:46AM +0000, Richard Purdie wrote: > On Thu, 2011-03-03 at 07:38 -0700, Tom Rini wrote: > > On 03/03/2011 12:27 AM, Martin Jansa wrote: > > > On Fri, Feb 25, 2011 at 04:27:16PM +0100, Martin Jansa wrote: > > >> * import from poky 6a11cf7dfe930461a6660e3b783b546fa2634900 > > >> * we had CACHE var pointing to machine specific dir since 2006 > > >> caf077679022f37ce55d758101f130e4e93bd7b5 > > >> * current bitbake is looking for cache dir like this: > > >> cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or > > >> bb.data.getVar("CACHE", d, True)) > > >> which without PERSISTENT_DIR defined picks machine-specific CACHE dir > > >> so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch > > >> in same builddir :/ > > > > > > ping? > > > > > > > I'd like Richards comments here. > > The intent was always to have CACHE be machine specific and > PERSISTENT_DIR not be machine specific. This means all builds share the > idea of revision numbers yet the main cache is machine specific so you > don't hit reparses when switching machines unnecessarily. BTW: with bitbake master and shared CACHE on one system I noticed that if I set MACHINE/DISTRO only in shell environment (with correct BB_ENV_EXTRAWHITE), then cache is not invalidated and metadata reparsed, resulting in broken builds where parts were respecting ${MACHINE} from shell and some parts had different machine from previous build. IIRC I noticed it on different machine listed in build summary and then another one in filename of machine-specific ipk. Removing cache after switching MACHINE in shell env worked fine. Regards, > So yes, I think the change is valid, I hadn't realised OE didn't have > that change and I'm happy to ack it... > > Cheers, > > Richard > > > > >> > > >> Signed-off-by: Martin Jansa<Martin.Jansa@gmail.com> > > >> --- > > >> classes/sanity.bbclass | 12 ++++++++++++ > > >> conf/bitbake.conf | 2 ++ > > >> conf/sanity.conf | 1 + > > >> 3 files changed, 15 insertions(+), 0 deletions(-) > > >> > > >> diff --git a/classes/sanity.bbclass b/classes/sanity.bbclass > > >> index 1e6b170..ffe3a49 100644 > > >> --- a/classes/sanity.bbclass > > >> +++ b/classes/sanity.bbclass > > >> @@ -189,6 +189,18 @@ def check_sanity(e): > > >> if messages != "": > > >> raise_sanity_error(messages) > > >> > > >> + # > > >> + # Check there isn't old persistent cache > > >> + # > > >> + cache = data.getVar('CACHE', e.data, True) > > >> + persistent_dir = data.getVar('PERSISTENT_DIR', e.data, True) > > >> + persistent_cache_filename = data.getVar('SANITY_PERSIST_DATA_FILE', e.data, True) > > >> + if os.path.exists(cache + '/' + persistent_cache_filename): > > >> + messages = messages + "Error, persistent cache file '%s' exists in old location '%s', please migrate it to new location '%s' and merge them together if you have one for each MACHINE.\n" % (persistent_cache_filename, cache, persistent_dir) > > >> + > > >> + if messages != "": > > >> + raise_sanity_error(messages) > > >> + > > >> python check_sanity_eventhandler() { > > >> if isinstance(e, bb.event.BuildStarted): > > >> check_sanity(e) > > >> diff --git a/conf/bitbake.conf b/conf/bitbake.conf > > >> index 88ad2ba..0c11cdb 100644 > > >> --- a/conf/bitbake.conf > > >> +++ b/conf/bitbake.conf > > >> @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}" > > >> > > >> TMPDIR = "${TOPDIR}/tmp" > > >> CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}" > > >> +# The persistent cache should be shared by all builds > > >> +PERSISTENT_DIR = "${TOPDIR}/conf/persistent" > > >> CO_DIR = "${DL_DIR}" > > >> CVSDIR = "${CO_DIR}/cvs" > > >> SVNDIR = "${CO_DIR}/svn" > > >> diff --git a/conf/sanity.conf b/conf/sanity.conf > > >> index e9902f6..c1ac332 100644 > > >> --- a/conf/sanity.conf > > >> +++ b/conf/sanity.conf > > >> @@ -7,5 +7,6 @@ BB_MIN_VERSION = "1.10.2" > > >> > > >> SANITY_ABIFILE = "${TMPDIR}/abi_version" > > >> SANITY_PRFILE = "${TMPDIR}/distro_pr" > > >> +SANITY_PERSIST_DATA_FILE = "bb_persist_data.sqlite3" > > >> > > >> INHERIT += "sanity" > > >> -- > > >> 1.7.4.1 > > >> > > > > > > > > > > > > _______________________________________________ > > > Openembedded-devel mailing list > > > Openembedded-devel@lists.openembedded.org > > > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel > > > > > > > > _______________________________________________ > Openembedded-devel mailing list > Openembedded-devel@lists.openembedded.org > http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR 2011-03-04 0:11 ` Richard Purdie 2011-03-04 6:10 ` Martin Jansa @ 2011-03-04 8:58 ` Martin Jansa 1 sibling, 0 replies; 9+ messages in thread From: Martin Jansa @ 2011-03-04 8:58 UTC (permalink / raw) To: openembedded-devel [-- Attachment #1: Type: text/plain, Size: 1515 bytes --] On Fri, Mar 04, 2011 at 12:11:46AM +0000, Richard Purdie wrote: > On Thu, 2011-03-03 at 07:38 -0700, Tom Rini wrote: > > On 03/03/2011 12:27 AM, Martin Jansa wrote: > > > On Fri, Feb 25, 2011 at 04:27:16PM +0100, Martin Jansa wrote: > > >> * import from poky 6a11cf7dfe930461a6660e3b783b546fa2634900 > > >> * we had CACHE var pointing to machine specific dir since 2006 > > >> caf077679022f37ce55d758101f130e4e93bd7b5 > > >> * current bitbake is looking for cache dir like this: > > >> cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or > > >> bb.data.getVar("CACHE", d, True)) > > >> which without PERSISTENT_DIR defined picks machine-specific CACHE dir > > >> so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch > > >> in same builddir :/ > > > > > > ping? > > > > > > > I'd like Richards comments here. > > The intent was always to have CACHE be machine specific and > PERSISTENT_DIR not be machine specific. This means all builds share the > idea of revision numbers yet the main cache is machine specific so you > don't hit reparses when switching machines unnecessarily. > > So yes, I think the change is valid, I hadn't realised OE didn't have > that change and I'm happy to ack it... Can I get at least one more ACK? And if we all agree on new location "${TOPDIR}/conf/persistent" then I would send similar patch to oe-core and move it too Regards, -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH] bitbake.conf: define shared PERSISTENT_DIR 2011-02-24 15:25 [PATCH] bitbake.conf: define shared PERSISTENT_DIR Martin Jansa 2011-02-24 16:45 ` Martin Jansa @ 2011-03-21 9:52 ` Martin Jansa 1 sibling, 0 replies; 9+ messages in thread From: Martin Jansa @ 2011-03-21 9:52 UTC (permalink / raw) To: openembedded-devel [-- Attachment #1: Type: text/plain, Size: 1959 bytes --] On Thu, Feb 24, 2011 at 04:25:56PM +0100, Martin Jansa wrote: > * import from poky > commit 6a11cf7dfe930461a6660e3b783b546fa2634900 > Author: Richard Purdie <rpurdie@linux.intel.com> > Date: Sun May 31 22:23:30 2009 +0100 > * we had CACHE var pointing to machine specific dir since > commit caf077679022f37ce55d758101f130e4e93bd7b5 > Author: Richard Purdie <rpurdie@rpsys.net> > Date: Mon Nov 20 17:40:20 2006 +0000 > > * current bitbake is looking for cache dir like this: > cachedir = (bb.data.getVar("PERSISTENT_DIR", d, True) or > bb.data.getVar("CACHE", d, True)) > which without PERSISTENT_DIR defined picks machine-specific CACHE dir > so LOCALCOUNT numbers in SRCPV keeps rolling with every MACHINE switch > in same builddir :/ > > Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> ping? 2nd version which moved it outside TMPDIR [oe] [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR was sort of rejected: http://lists.linuxtogo.org/pipermail/openembedded-core/2011-March/000298.html but this part without moving it outside TMPDIR is still valid and I'll push it after month of review if nobody objects :). > --- > conf/bitbake.conf | 2 ++ > 1 files changed, 2 insertions(+), 0 deletions(-) > > diff --git a/conf/bitbake.conf b/conf/bitbake.conf > index dcc5c70..53bb7b7 100644 > --- a/conf/bitbake.conf > +++ b/conf/bitbake.conf > @@ -294,6 +294,8 @@ FILESDIR = "${@bb.which(d.getVar('FILESPATH', 1), '.')}" > > TMPDIR = "${TOPDIR}/tmp" > CACHE = "${TMPDIR}/cache${@['', '/' + str(bb.data.getVar('MACHINE', d, 1))][bool(bb.data.getVar('MACHINE', d, 1))]}" > +# The persistent cache should be shared by all builds > +PERSISTENT_DIR = "${TMPDIR}/cache" > CO_DIR = "${DL_DIR}" > CVSDIR = "${CO_DIR}/cvs" > SVNDIR = "${CO_DIR}/svn" > -- > 1.7.4.1 > -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2011-03-21 9:54 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2011-02-24 15:25 [PATCH] bitbake.conf: define shared PERSISTENT_DIR Martin Jansa 2011-02-24 16:45 ` Martin Jansa 2011-02-25 15:27 ` [PATCH] bitbake.conf: define shared PERSISTENT_DIR and point it outside TMPDIR Martin Jansa 2011-03-03 7:27 ` Martin Jansa 2011-03-03 14:38 ` Tom Rini 2011-03-04 0:11 ` Richard Purdie 2011-03-04 6:10 ` Martin Jansa 2011-03-04 8:58 ` Martin Jansa 2011-03-21 9:52 ` [PATCH] bitbake.conf: define shared PERSISTENT_DIR Martin Jansa
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.