* [PATCH 1/3] bbclass/sstate_readonly approach 1: add prefuncs to SSTATETASKS
2014-08-06 7:15 [PATCH 0/3][RFC] only allowed sstate-cache objects are allowed in a build (read-only sstate-cache?) Hongxu Jia
@ 2014-08-06 7:15 ` Hongxu Jia
2014-08-06 7:16 ` [PATCH 2/3] bbclass/sstate_readonly_2 approach 2: add event handler at TaskStarted time Hongxu Jia
` (2 subsequent siblings)
3 siblings, 0 replies; 5+ messages in thread
From: Hongxu Jia @ 2014-08-06 7:15 UTC (permalink / raw)
To: openembedded-core, mark.hatle, richard.purdie; +Cc: saul.wold
Add prefuncs to SSTATETASKS to handle the sstate-cache read-only cheking.
If read-only sstate-cache enable, and the building recipe not in the
${SSTATECACHE_WHITELIST}, it trigered an instant error.
Flaws:
1. For each SSTATETASKS task, there is a read-only sstate-cache checking,
it increased the build loads.
2. The instant error breaks the building randomly while there are multible
recipes not in the ${SSTATECACHE_WHITELIST}, and also could not list all
the missing recipes which sould be added to ${SSTATECACHE_WHITELIST}.
3. The checking time is postponed to SSTATETASKS, take db-native for example:
if you tweak db-native's do_configure, the checking will occur at
do_populate_sysroot time. The tasks 'do_compile', 'do_install' has
been invoked.
...
$ bitbake db
ERROR: Read-only sstate-cache is enabled, the build of gettext-minimal-native
did not come from sstate-cache. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to build from source
ERROR: Function failed: sstate_readonly_check
ERROR: Logfile of failure stored in: tmp/work/x86_64-linux/gettext-minimal-native/0.18.3.2-r0/temp/log.do_populate_sysroot.23801
ERROR: Task 155 (poky/meta/recipes-core/gettext/gettext-minimal-native_0.18.3.2.bb, do_populate_sysroot) failed with exit code '1'
NOTE: Tasks Summary: Attempted 84 tasks of which 49 didn't need to be rerun and 1 failed.
...
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
meta/classes/sstate_readonly.bbclass | 38 ++++++++++++++++++++++++++++++++++++
1 file changed, 38 insertions(+)
create mode 100644 meta/classes/sstate_readonly.bbclass
diff --git a/meta/classes/sstate_readonly.bbclass b/meta/classes/sstate_readonly.bbclass
new file mode 100644
index 0000000..88b1c39
--- /dev/null
+++ b/meta/classes/sstate_readonly.bbclass
@@ -0,0 +1,38 @@
+# 1) If ${SSTATECACHE_WHITELIST} is "", it means read-only sstate-cache
+# disabled;
+#
+# 2) If read-only sstate-cache enabled and the recipe's ${PN} not listed
+# in ${SSTATECACHE_WHITELIST}, the build from source will triger an
+# instant error;
+SSTATECACHE_WHITELIST ?= ""
+
+python () {
+ unique_tasks = set((d.getVar('SSTATETASKS', True) or "").split())
+ d.setVar('SSTATETASKS', " ".join(unique_tasks))
+ for task in unique_tasks:
+ d.prependVarFlag(task, 'prefuncs', "sstate_readonly_check")
+}
+
+_sstate_readonly_check[vardepsexclude] += "SSTATECACHE_WHITELIST"
+def _sstate_readonly_check(d):
+ whitelist = d.getVar('SSTATECACHE_WHITELIST', True) or ""
+ if whitelist != "":
+ pn = d.getVar('PN', True)
+ if pn not in whitelist.split():
+ msg = 'Read-only sstate-cache is enabled, the build of %s\n' % pn
+ msg += 'did not come from sstate-cache. Only the recipe listed in\n'
+ msg += 'SSTATECACHE_WHITELIST is allowed to build from source'
+ bb.fatal(msg)
+
+python sstate_readonly_check(){
+ _sstate_readonly_check(d)
+}
+
+python do_cleansstate_prepend() {
+ _sstate_readonly_check(d)
+}
+
+python do_cleanall_prepend() {
+ _sstate_readonly_check(d)
+}
+
--
1.9.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH 2/3] bbclass/sstate_readonly_2 approach 2: add event handler at TaskStarted time
2014-08-06 7:15 [PATCH 0/3][RFC] only allowed sstate-cache objects are allowed in a build (read-only sstate-cache?) Hongxu Jia
2014-08-06 7:15 ` [PATCH 1/3] bbclass/sstate_readonly approach 1: add prefuncs to SSTATETASKS Hongxu Jia
@ 2014-08-06 7:16 ` Hongxu Jia
2014-08-06 7:16 ` [PATCH 3/3] bbclass/sstate approach 3: add checking in the return path of sstate_checkhashes Hongxu Jia
2014-08-20 10:32 ` [PATCH 0/3][RFC] only allowed sstate-cache objects are allowed in a build (read-only sstate-cache?) Hongxu Jia
3 siblings, 0 replies; 5+ messages in thread
From: Hongxu Jia @ 2014-08-06 7:16 UTC (permalink / raw)
To: openembedded-core, mark.hatle, richard.purdie; +Cc: saul.wold
Add event handler at TaskStarted time to handle the sstate-cache
read-only cheking. If read-only sstate-cache enable, and the building recipe
not in the ${SSTATECACHE_WHITELIST}, it trigered an instant error.
Flaws:
1. For every task, there is a read-only sstate-cache checking, it increased
the build loads.
2. The instant error breaks the building randomly while there are multible
recipes not in the ${SSTATECACHE_WHITELIST}, and also could not list all
the missing recipes which sould be added to ${SSTATECACHE_WHITELIST}.
...
$ bitbake db
ERROR:
Read-only sstate-cache is enabled, the build of ncurses-native
did not come from sstate-cache. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to build from source
ERROR: Execution of event handler 'sstate_readonly_eventhandler' failed
ERROR: Task 350 (virtual:native:poky/meta/recipes-core/ncurses/ncurses_5.9.bb, do_fetch) failed with exit code '1'
ERROR:
Read-only sstate-cache is enabled, the build of gcc-cross-initial-i586
did not come from sstate-cache. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to build from source
ERROR: Execution of event handler 'sstate_readonly_eventhandler' failed
ERROR: Task 290 (poky/meta/recipes-devtools/gcc/gcc-cross-initial_4.9.bb, do_fetch) failed with exit code '1'
...
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
meta/classes/sstate_readonly_2.bbclass | 25 +++++++++++++++++++++++++
1 file changed, 25 insertions(+)
create mode 100644 meta/classes/sstate_readonly_2.bbclass
diff --git a/meta/classes/sstate_readonly_2.bbclass b/meta/classes/sstate_readonly_2.bbclass
new file mode 100644
index 0000000..871b56f
--- /dev/null
+++ b/meta/classes/sstate_readonly_2.bbclass
@@ -0,0 +1,25 @@
+# 1) If ${SSTATECACHE_WHITELIST} is "", it means read-only sstate-cache
+# disabled;
+#
+# 2) If read-only sstate-cache enabled and the recipe's ${PN} not listed
+# in ${SSTATECACHE_WHITELIST}, the build from source will triger an
+# instant error;
+SSTATECACHE_WHITELIST ?= ""
+
+addhandler sstate_readonly_eventhandler
+sstate_readonly_eventhandler[eventmask] = "bb.build.TaskStarted"
+python sstate_readonly_eventhandler() {
+ d = e.data
+ whitelist = d.getVar('SSTATECACHE_WHITELIST', True) or ""
+ if whitelist != "":
+ taskname = d.getVar('BB_CURRENTTASK', True)
+ if not taskname.startswith('do_'):
+ taskname = 'do_%s' % taskname
+ if not taskname.endswith("_setscene") and taskname != "do_setscene":
+ pn = d.getVar('PN', True)
+ if pn not in whitelist.split():
+ msg = '\nRead-only sstate-cache is enabled, the build of %s\n' % pn
+ msg += 'did not come from sstate-cache. Only the recipe listed in\n'
+ msg += 'SSTATECACHE_WHITELIST is allowed to build from source'
+ bb.msg.fatal('sstate', msg)
+}
--
1.9.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH 3/3] bbclass/sstate approach 3: add checking in the return path of sstate_checkhashes
2014-08-06 7:15 [PATCH 0/3][RFC] only allowed sstate-cache objects are allowed in a build (read-only sstate-cache?) Hongxu Jia
2014-08-06 7:15 ` [PATCH 1/3] bbclass/sstate_readonly approach 1: add prefuncs to SSTATETASKS Hongxu Jia
2014-08-06 7:16 ` [PATCH 2/3] bbclass/sstate_readonly_2 approach 2: add event handler at TaskStarted time Hongxu Jia
@ 2014-08-06 7:16 ` Hongxu Jia
2014-08-20 10:32 ` [PATCH 0/3][RFC] only allowed sstate-cache objects are allowed in a build (read-only sstate-cache?) Hongxu Jia
3 siblings, 0 replies; 5+ messages in thread
From: Hongxu Jia @ 2014-08-06 7:16 UTC (permalink / raw)
To: openembedded-core, mark.hatle, richard.purdie; +Cc: saul.wold
In the sstate-cache code, add a checking in the return path of
sstate_checkhashes. If read-only sstate-cache enable, and the
recipe's ${PN} not in the ${SSTATECACHE_WHITELIST}, it trigered
an instant error.
Flaws:
1. We should manually unlock the bitbake lock which the instant error
msg will exit the build immediately.
...
$ bitbake db
ERROR: Read-only sstate-cache is enabled, the build of
"db rpm-native gcc-runtime eglibc linux-libc-headers libgcc"
did not come from sstate-cache. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to build from source
Summary: There was 1 ERROR message shown, returning a non-zero exit code.
...
Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
meta/classes/sstate.bbclass | 40 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 0d3940e..17cd109 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -41,6 +41,14 @@ EXTRA_STAGING_FIXMES ?= ""
sstate_create_package[dirs] = "${SSTATE_BUILDDIR}"
sstate_unpack_package[dirs] = "${SSTATE_INSTDIR}"
+# 1) If ${SSTATECACHE_WHITELIST} is "", it means read-only sstate-cache
+# disabled;
+#
+# 2) If read-only sstate-cache enabled and the recipe's ${PN} not listed
+# in ${SSTATECACHE_WHITELIST}, the build from source will triger an
+# instant error;
+SSTATECACHE_WHITELIST ?= ""
+
python () {
if bb.data.inherits_class('native', d):
d.setVar('SSTATE_PKGARCH', d.getVar('BUILD_ARCH'))
@@ -382,6 +390,15 @@ sstate_clean[vardepsexclude] = "SSTATE_MANFILEPREFIX"
CLEANFUNCS += "sstate_cleanall"
python sstate_cleanall() {
+ whitelist = d.getVar('SSTATECACHE_WHITELIST', True) or ""
+ if whitelist:
+ pn = d.getVar('PN', True)
+ if pn not in whitelist.split():
+ msg = 'Read-only sstate-cache is enabled, the clean of \n'
+ msg += '%s is not allowed. Only the recipe listed in\n' % pn
+ msg += 'SSTATECACHE_WHITELIST is allowed to clean sstate-cache'
+ bb.fatal(msg)
+
bb.note("Removing shared state for package %s" % d.getVar('PN', True))
manifest_dir = d.getVar('SSTATE_MANIFESTS', True)
@@ -704,6 +721,29 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
evdata['found'].append( (sq_fn[task], sq_task[task], sq_hash[task], sstatefile ) )
bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d)
+ whitelist = d.getVar('SSTATECACHE_WHITELIST', True) or ""
+ if whitelist:
+ missed_pn = []
+ for task in missed:
+ fn = sq_fn[task]
+ data = bb.cache.Cache.loadDataFull(fn, '', d)
+ pn = data.getVar('PN', True) or ""
+ if pn and pn not in missed_pn:
+ missed_pn.append(pn)
+
+ if missed_pn:
+ blacklist = [pn for pn in missed_pn if pn not in whitelist.split()]
+ if blacklist:
+ # We should manually unlock the bitbake lock, because the fatal
+ # msg will exit the build immediately.
+ lockfile = d.expand("${TOPDIR}/bitbake.lock")
+ os.unlink(lockfile)
+ msg = 'Read-only sstate-cache is enabled, the build of \n'
+ msg += '"' + ' '.join(blacklist) + '"\n'
+ msg += 'did not come from sstate-cache. Only the recipe listed in\n'
+ msg += 'SSTATECACHE_WHITELIST is allowed to build from source'
+ bb.msg.fatal('sstate', msg)
+
return ret
BB_SETSCENE_DEPVALID = "setscene_depvalid"
--
1.9.1
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH 0/3][RFC] only allowed sstate-cache objects are allowed in a build (read-only sstate-cache?)
2014-08-06 7:15 [PATCH 0/3][RFC] only allowed sstate-cache objects are allowed in a build (read-only sstate-cache?) Hongxu Jia
` (2 preceding siblings ...)
2014-08-06 7:16 ` [PATCH 3/3] bbclass/sstate approach 3: add checking in the return path of sstate_checkhashes Hongxu Jia
@ 2014-08-20 10:32 ` Hongxu Jia
3 siblings, 0 replies; 5+ messages in thread
From: Hongxu Jia @ 2014-08-20 10:32 UTC (permalink / raw)
To: openembedded-core, mark.hatle, richard.purdie; +Cc: saul.wold
Ping
//Hongxu
On 08/06/2014 03:15 PM, Hongxu Jia wrote:
> Issue description:
>
> The issue is that the developer who demand only the "new" software
> they write is allowed to be compiled from source, they only want to
> reuse binaries from an existed sstate-cache, if the developer makes
> a change that triggers a rebuild, it should be an instant error.
>
> The purpose of this is for the sstate-cache to check if the item
> exists or not. If it doesn't the item needs to be in a whitelist
> or we need to fail.
>
> I dig into three approaches to implement, and send them to oe-core
> list to see if the community has a preference as to the approach,
> and any additional comments.
>
> In these approaches, I'm not sure we should protect the 'clean' or
> not. Since it's ok to clean the sstate-cache, as long as a mirror
> fetch will pull it back down. Any suggestion is welcomed.
>
> If you have any better ideas, please don't hesitate to share with us.
>
> Test steps:
>
> 1) For approach 1
> INHERIT += 'sstate_readonly'
>
> For approach 2
> INHERIT += 'sstate_readonly_2'
>
> 2) Create a sstate cache for testing
> $ bitbake db
>
> 3) Add gzip to SSTATECACHE_WHITELIST, it enabled read-only sstate-cache,
> vim local.conf
> ...
> SSTATECACHE_WHITELIST = 'gzip'
> ...
>
> 4) Remove tmp dir and build db from sstate-cache succeed
> $ mv tmp tmp-back && bitbake db
>
> 5) tweak db's do_configure task by adding comments
> --- a/meta/recipes-support/db/db_6.0.30.bb
> +++ b/meta/recipes-support/db/db_6.0.30.bb
> @@ -27,6 +27,8 @@ LIC_FILES_CHKSUM = "file://../LICENSE;md5=1ec8b0b17cc31513fe35ab10716f8490"
> @@ -82,6 +84,7 @@ do_configure() {
> gnu-configize --force ${S}
> export STRIP="true"
> oe_runconf
> + echo "hello"
> }
>
> 6) build db and there is a build failure
> $ bitbake db
>
> 7) clean db and there is a clean failure
> bitbake db -ccleansstate
>
> 8) Append the missing recipe's PN to SSTATECACHE_WHITELIST
> vim local.conf
> ...
> SSTATECACHE_WHITELIST = 'gzip db rpm-native gcc-runtime eglibc linux-libc-headers libgcc'
> ...
>
> 9) Build db and clean db succeed
> $ bitbake db && bitbake db -ccleansstate
>
> //Hongxu
>
> The following changes since commit 870bb8d35547b8313b3a487d7e8b914ab9470e64:
>
> local.conf.sample.extended: fix example for EXTRA_USERS_PARAMS (2014-08-04 17:38:24 +0100)
>
> are available in the git repository at:
>
> git://git.pokylinux.org/poky-contrib hongxu/readonly-sstatecache
> http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/readonly-sstatecache
>
> Hongxu Jia (3):
> bbclass/sstate_readonly approach 1: add prefuncs to SSTATETASKS
> bbclass/sstate_readonly_2 approach 2: add event handler at TaskStarted
> time
> bbclass/sstate approach 3: add checking in the return path of
> sstate_checkhashes
>
> meta/classes/sstate.bbclass | 40 ++++++++++++++++++++++++++++++++++
> meta/classes/sstate_readonly.bbclass | 38 ++++++++++++++++++++++++++++++++
> meta/classes/sstate_readonly_2.bbclass | 25 +++++++++++++++++++++
> 3 files changed, 103 insertions(+)
> create mode 100644 meta/classes/sstate_readonly.bbclass
> create mode 100644 meta/classes/sstate_readonly_2.bbclass
>
^ permalink raw reply [flat|nested] 5+ messages in thread