[PATCH 0/3][RFC] only allowed sstate-cache objects are allowed in a build (read-only sstate-cache?)

[PATCH 0/3][RFC] only allowed sstate-cache objects are allowed in a build (read-only sstate-cache?)

[Hongxu Jia]

Issue description:

The issue is that the developer who demand only the "new" software
they write is allowed to be compiled from source, they only want to
reuse binaries from an existed sstate-cache, if the developer makes
a change that triggers a rebuild, it should be an instant error. 

The purpose of this is for the sstate-cache to check if the item
exists or not. If it doesn't the item needs to be in a whitelist
or we need to fail.

I dig into three approaches to implement, and send them to oe-core
list to see if the community has a preference as to the approach,
and any additional comments.

In these approaches, I'm not sure we should protect the 'clean' or
not. Since it's ok to clean the sstate-cache, as long as a mirror
fetch will pull it back down. Any suggestion is welcomed.

If you have any better ideas, please don't hesitate to share with us.

Test steps:

1) For approach 1
   INHERIT += 'sstate_readonly'

   For approach 2
   INHERIT += 'sstate_readonly_2'

2) Create a sstate cache for testing
$ bitbake db

3) Add gzip to SSTATECACHE_WHITELIST, it enabled read-only sstate-cache,
vim local.conf
...
SSTATECACHE_WHITELIST = 'gzip'
...

4) Remove tmp dir and build db from sstate-cache succeed
$ mv tmp tmp-back && bitbake db

5) tweak db's do_configure task by adding comments
--- a/meta/recipes-support/db/db_6.0.30.bb
+++ b/meta/recipes-support/db/db_6.0.30.bb
@@ -27,6 +27,8 @@ LIC_FILES_CHKSUM = "file://../LICENSE;md5=1ec8b0b17cc31513fe35ab10716f8490"
@@ -82,6 +84,7 @@ do_configure() {
        gnu-configize --force ${S}
        export STRIP="true"
        oe_runconf
+       echo "hello"
 }

6) build db and there is a build failure
$ bitbake db

7) clean db and there is a clean failure
bitbake db -ccleansstate

8) Append the missing recipe's PN to SSTATECACHE_WHITELIST
vim local.conf
...
SSTATECACHE_WHITELIST = 'gzip db rpm-native gcc-runtime eglibc linux-libc-headers libgcc'
...

9) Build db and clean db succeed
$ bitbake db && bitbake db -ccleansstate

//Hongxu

The following changes since commit 870bb8d35547b8313b3a487d7e8b914ab9470e64:

  local.conf.sample.extended: fix example for EXTRA_USERS_PARAMS (2014-08-04 17:38:24 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib hongxu/readonly-sstatecache
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/readonly-sstatecache

Hongxu Jia (3):
  bbclass/sstate_readonly approach 1: add prefuncs to SSTATETASKS
  bbclass/sstate_readonly_2 approach 2: add event handler at TaskStarted
    time
  bbclass/sstate approach 3: add checking in the return path of
    sstate_checkhashes

 meta/classes/sstate.bbclass            | 40 ++++++++++++++++++++++++++++++++++
 meta/classes/sstate_readonly.bbclass   | 38 ++++++++++++++++++++++++++++++++
 meta/classes/sstate_readonly_2.bbclass | 25 +++++++++++++++++++++
 3 files changed, 103 insertions(+)
 create mode 100644 meta/classes/sstate_readonly.bbclass
 create mode 100644 meta/classes/sstate_readonly_2.bbclass

-- 
1.9.1

[PATCH 1/3] bbclass/sstate_readonly approach 1: add prefuncs to SSTATETASKS

[Hongxu Jia]

Add prefuncs to SSTATETASKS to handle the sstate-cache read-only cheking.
If read-only sstate-cache enable, and the building recipe not in the
${SSTATECACHE_WHITELIST}, it trigered an instant error.

Flaws:
1. For each SSTATETASKS task, there is a read-only sstate-cache checking,
   it increased the build loads.

2. The instant error breaks the building randomly while there are multible
   recipes not in the ${SSTATECACHE_WHITELIST}, and also could not list all
   the missing recipes which sould be added to ${SSTATECACHE_WHITELIST}.

3. The checking time is postponed to SSTATETASKS, take db-native for example:
   if you tweak db-native's do_configure, the checking will occur at
   do_populate_sysroot time. The tasks 'do_compile', 'do_install' has
   been invoked.

...
$ bitbake db
ERROR: Read-only sstate-cache is enabled, the build of gettext-minimal-native
did not come from sstate-cache. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to build from source
ERROR: Function failed: sstate_readonly_check
ERROR: Logfile of failure stored in: tmp/work/x86_64-linux/gettext-minimal-native/0.18.3.2-r0/temp/log.do_populate_sysroot.23801
ERROR: Task 155 (poky/meta/recipes-core/gettext/gettext-minimal-native_0.18.3.2.bb, do_populate_sysroot) failed with exit code '1'
NOTE: Tasks Summary: Attempted 84 tasks of which 49 didn't need to be rerun and 1 failed.
...

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/sstate_readonly.bbclass | 38 ++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100644 meta/classes/sstate_readonly.bbclass

diff --git a/meta/classes/sstate_readonly.bbclass b/meta/classes/sstate_readonly.bbclass
new file mode 100644
index 0000000..88b1c39
--- /dev/null
+++ b/meta/classes/sstate_readonly.bbclass
@@ -0,0 +1,38 @@
+# 1) If ${SSTATECACHE_WHITELIST} is "", it means read-only sstate-cache
+#    disabled;
+#
+# 2) If read-only sstate-cache enabled and the recipe's ${PN} not listed
+#    in ${SSTATECACHE_WHITELIST}, the build from source will triger an
+#    instant error;
+SSTATECACHE_WHITELIST ?= ""
+
+python () {
+    unique_tasks = set((d.getVar('SSTATETASKS', True) or "").split())
+    d.setVar('SSTATETASKS', " ".join(unique_tasks))
+    for task in unique_tasks:
+        d.prependVarFlag(task, 'prefuncs', "sstate_readonly_check")
+}
+
+_sstate_readonly_check[vardepsexclude] += "SSTATECACHE_WHITELIST"
+def _sstate_readonly_check(d):
+    whitelist = d.getVar('SSTATECACHE_WHITELIST', True) or ""
+    if whitelist != "":
+        pn = d.getVar('PN', True)
+        if pn not in whitelist.split():
+            msg =  'Read-only sstate-cache is enabled, the build of %s\n' % pn
+            msg += 'did not come from sstate-cache. Only the recipe listed in\n'
+            msg += 'SSTATECACHE_WHITELIST is allowed to build from source'
+            bb.fatal(msg)
+
+python sstate_readonly_check(){
+    _sstate_readonly_check(d)
+}
+
+python do_cleansstate_prepend() {
+        _sstate_readonly_check(d)
+}
+
+python do_cleanall_prepend() {
+    _sstate_readonly_check(d)
+}
+
-- 
1.9.1

[PATCH 2/3] bbclass/sstate_readonly_2 approach 2: add event handler at TaskStarted time

[Hongxu Jia]

Add event handler at TaskStarted time to handle the sstate-cache
read-only cheking. If read-only sstate-cache enable, and the building recipe
not in the ${SSTATECACHE_WHITELIST}, it trigered an instant error.

Flaws:
1. For every task, there is a read-only sstate-cache checking, it increased
   the build loads.

2. The instant error breaks the building randomly while there are multible
   recipes not in the ${SSTATECACHE_WHITELIST}, and also could not list all
   the missing recipes which sould be added to ${SSTATECACHE_WHITELIST}.

...
$ bitbake db
ERROR:
Read-only sstate-cache is enabled, the build of ncurses-native
did not come from sstate-cache. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to build from source
ERROR: Execution of event handler 'sstate_readonly_eventhandler' failed
ERROR: Task 350 (virtual:native:poky/meta/recipes-core/ncurses/ncurses_5.9.bb, do_fetch) failed with exit code '1'
ERROR:
Read-only sstate-cache is enabled, the build of gcc-cross-initial-i586
did not come from sstate-cache. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to build from source
ERROR: Execution of event handler 'sstate_readonly_eventhandler' failed
ERROR: Task 290 (poky/meta/recipes-devtools/gcc/gcc-cross-initial_4.9.bb, do_fetch) failed with exit code '1'
...

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/sstate_readonly_2.bbclass | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)
 create mode 100644 meta/classes/sstate_readonly_2.bbclass

diff --git a/meta/classes/sstate_readonly_2.bbclass b/meta/classes/sstate_readonly_2.bbclass
new file mode 100644
index 0000000..871b56f
--- /dev/null
+++ b/meta/classes/sstate_readonly_2.bbclass
@@ -0,0 +1,25 @@
+# 1) If ${SSTATECACHE_WHITELIST} is "", it means read-only sstate-cache
+#    disabled;
+#
+# 2) If read-only sstate-cache enabled and the recipe's ${PN} not listed
+#    in ${SSTATECACHE_WHITELIST}, the build from source will triger an
+#    instant error;
+SSTATECACHE_WHITELIST ?= ""
+
+addhandler sstate_readonly_eventhandler
+sstate_readonly_eventhandler[eventmask] = "bb.build.TaskStarted"
+python sstate_readonly_eventhandler() {
+    d = e.data
+    whitelist = d.getVar('SSTATECACHE_WHITELIST', True) or ""
+    if whitelist != "":
+        taskname = d.getVar('BB_CURRENTTASK', True)
+        if not taskname.startswith('do_'):
+            taskname = 'do_%s' % taskname
+        if not taskname.endswith("_setscene") and taskname != "do_setscene":
+            pn = d.getVar('PN', True)
+            if pn not in whitelist.split():
+                msg =  '\nRead-only sstate-cache is enabled, the build of %s\n' % pn
+                msg += 'did not come from sstate-cache. Only the recipe listed in\n'
+                msg += 'SSTATECACHE_WHITELIST is allowed to build from source'
+                bb.msg.fatal('sstate', msg)
+}
-- 
1.9.1

[PATCH 3/3] bbclass/sstate approach 3: add checking in the return path of sstate_checkhashes

[Hongxu Jia]

In the sstate-cache code, add a checking in the return path of
sstate_checkhashes. If read-only sstate-cache enable, and the
recipe's ${PN} not in the ${SSTATECACHE_WHITELIST}, it trigered
an instant error.

Flaws:
1. We should manually unlock the bitbake lock which the instant error
   msg will exit the build immediately.

...
$ bitbake db
ERROR: Read-only sstate-cache is enabled, the build of
"db rpm-native gcc-runtime eglibc linux-libc-headers libgcc"
did not come from sstate-cache. Only the recipe listed in
SSTATECACHE_WHITELIST is allowed to build from source

Summary: There was 1 ERROR message shown, returning a non-zero exit code.
...

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/sstate.bbclass | 40 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 40 insertions(+)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index 0d3940e..17cd109 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -41,6 +41,14 @@ EXTRA_STAGING_FIXMES ?= ""
 sstate_create_package[dirs] = "${SSTATE_BUILDDIR}"
 sstate_unpack_package[dirs] = "${SSTATE_INSTDIR}"
 
+# 1) If ${SSTATECACHE_WHITELIST} is "", it means read-only sstate-cache
+#    disabled;
+#
+# 2) If read-only sstate-cache enabled and the recipe's ${PN} not listed
+#    in ${SSTATECACHE_WHITELIST}, the build from source will triger an
+#    instant error;
+SSTATECACHE_WHITELIST ?= ""
+
 python () {
     if bb.data.inherits_class('native', d):
         d.setVar('SSTATE_PKGARCH', d.getVar('BUILD_ARCH'))
@@ -382,6 +390,15 @@ sstate_clean[vardepsexclude] = "SSTATE_MANFILEPREFIX"
 CLEANFUNCS += "sstate_cleanall"
 
 python sstate_cleanall() {
+    whitelist = d.getVar('SSTATECACHE_WHITELIST', True) or ""
+    if whitelist:
+        pn = d.getVar('PN', True)
+        if pn not in whitelist.split():
+            msg =  'Read-only sstate-cache is enabled, the clean of \n'
+            msg += '%s is not allowed. Only the recipe listed in\n' % pn
+            msg += 'SSTATECACHE_WHITELIST is allowed to clean sstate-cache'
+            bb.fatal(msg)
+
     bb.note("Removing shared state for package %s" % d.getVar('PN', True))
 
     manifest_dir = d.getVar('SSTATE_MANIFESTS', True)
@@ -704,6 +721,29 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash, sq_hashfn, d):
             evdata['found'].append( (sq_fn[task], sq_task[task], sq_hash[task], sstatefile ) )
         bb.event.fire(bb.event.MetadataEvent("MissedSstate", evdata), d)
 
+    whitelist = d.getVar('SSTATECACHE_WHITELIST', True) or ""
+    if whitelist:
+        missed_pn = []
+        for task in missed:
+            fn = sq_fn[task]
+            data = bb.cache.Cache.loadDataFull(fn, '', d)
+            pn = data.getVar('PN', True) or ""
+            if pn and pn not in missed_pn:
+                missed_pn.append(pn)
+
+        if missed_pn:
+            blacklist = [pn for pn in missed_pn if pn not in whitelist.split()]
+            if blacklist:
+                # We should manually unlock the bitbake lock, because the fatal
+                # msg will exit the build immediately.
+                lockfile = d.expand("${TOPDIR}/bitbake.lock")
+                os.unlink(lockfile)
+                msg =  'Read-only sstate-cache is enabled, the build of \n'
+                msg += '"' + ' '.join(blacklist) + '"\n'
+                msg += 'did not come from sstate-cache. Only the recipe listed in\n'
+                msg += 'SSTATECACHE_WHITELIST is allowed to build from source'
+                bb.msg.fatal('sstate', msg)
+
     return ret
 
 BB_SETSCENE_DEPVALID = "setscene_depvalid"
-- 
1.9.1

Re: [PATCH 0/3][RFC] only allowed sstate-cache objects are allowed in a build (read-only sstate-cache?)

[Hongxu Jia]

Ping

//Hongxu

On 08/06/2014 03:15 PM, Hongxu Jia wrote:
> Issue description:
>
> The issue is that the developer who demand only the "new" software
> they write is allowed to be compiled from source, they only want to
> reuse binaries from an existed sstate-cache, if the developer makes
> a change that triggers a rebuild, it should be an instant error.
>
> The purpose of this is for the sstate-cache to check if the item
> exists or not. If it doesn't the item needs to be in a whitelist
> or we need to fail.
>
> I dig into three approaches to implement, and send them to oe-core
> list to see if the community has a preference as to the approach,
> and any additional comments.
>
> In these approaches, I'm not sure we should protect the 'clean' or
> not. Since it's ok to clean the sstate-cache, as long as a mirror
> fetch will pull it back down. Any suggestion is welcomed.
>
> If you have any better ideas, please don't hesitate to share with us.
>
> Test steps:
>
> 1) For approach 1
>     INHERIT += 'sstate_readonly'
>
>     For approach 2
>     INHERIT += 'sstate_readonly_2'
>
> 2) Create a sstate cache for testing
> $ bitbake db
>
> 3) Add gzip to SSTATECACHE_WHITELIST, it enabled read-only sstate-cache,
> vim local.conf
> ...
> SSTATECACHE_WHITELIST = 'gzip'
> ...
>
> 4) Remove tmp dir and build db from sstate-cache succeed
> $ mv tmp tmp-back && bitbake db
>
> 5) tweak db's do_configure task by adding comments
> --- a/meta/recipes-support/db/db_6.0.30.bb
> +++ b/meta/recipes-support/db/db_6.0.30.bb
> @@ -27,6 +27,8 @@ LIC_FILES_CHKSUM = "file://../LICENSE;md5=1ec8b0b17cc31513fe35ab10716f8490"
> @@ -82,6 +84,7 @@ do_configure() {
>          gnu-configize --force ${S}
>          export STRIP="true"
>          oe_runconf
> +       echo "hello"
>   }
>
> 6) build db and there is a build failure
> $ bitbake db
>
> 7) clean db and there is a clean failure
> bitbake db -ccleansstate
>
> 8) Append the missing recipe's PN to SSTATECACHE_WHITELIST
> vim local.conf
> ...
> SSTATECACHE_WHITELIST = 'gzip db rpm-native gcc-runtime eglibc linux-libc-headers libgcc'
> ...
>
> 9) Build db and clean db succeed
> $ bitbake db && bitbake db -ccleansstate
>
> //Hongxu
>
> The following changes since commit 870bb8d35547b8313b3a487d7e8b914ab9470e64:
>
>    local.conf.sample.extended: fix example for EXTRA_USERS_PARAMS (2014-08-04 17:38:24 +0100)
>
> are available in the git repository at:
>
>    git://git.pokylinux.org/poky-contrib hongxu/readonly-sstatecache
>    http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/readonly-sstatecache
>
> Hongxu Jia (3):
>    bbclass/sstate_readonly approach 1: add prefuncs to SSTATETASKS
>    bbclass/sstate_readonly_2 approach 2: add event handler at TaskStarted
>      time
>    bbclass/sstate approach 3: add checking in the return path of
>      sstate_checkhashes
>
>   meta/classes/sstate.bbclass            | 40 ++++++++++++++++++++++++++++++++++
>   meta/classes/sstate_readonly.bbclass   | 38 ++++++++++++++++++++++++++++++++
>   meta/classes/sstate_readonly_2.bbclass | 25 +++++++++++++++++++++
>   3 files changed, 103 insertions(+)
>   create mode 100644 meta/classes/sstate_readonly.bbclass
>   create mode 100644 meta/classes/sstate_readonly_2.bbclass
>