[PATCH V2 0/5] improve locked down sstate

[PATCH V2 0/5] improve locked down sstate

[Hongxu Jia]

Changed in V2:
- Set SIGGEN_LOCKEDSIGS_CHECK_LEVEL default to error

- Move dump sig file event handler to a separate class

- Fix typo

- Fix overrides behaviour to remove SIGGEN_LOCKEDSIGS_i586

- Support incremental dump lockedsigs without complicate
  the structure of the lockedsigs variable.

Test Steps:

1. Prepare a locked sstat-cache by dumping lockedsigs file
1) vim local.conf
...
USER_CLASSES += "sstate_lockedsig"
SIGGEN_LOCKEDSIGS_CONFIG = "/path/to/locked-sigs.inc"
SSTATE_DIR ?= "/path/to/sstate-cache"
...

2) Build db-native from a clean environment
$ bitbake db-native

3) Back up lockedsigs file
$ mv /path/to/locked-sigs.inc  /path/to/locked-sigs.inc-back

4) Incremental build ldconfig-native
$ vim local.conf
...
require ${SIGGEN_LOCKEDSIGS_CONFIG}
...
$ bitbake ldconfig-native

5) Compare the two sig files, and check whether it is incremental dump or not
$ vimdiff /path/to/locked-sigs.inc-back /path/to/locked-sigs.inc

2. Create a new build which uses previous locked sstat-cache
1) vim local.conf
...
SSTATE_DIR ?= "/path/to/sstate-cache"
SIGGEN_LOCKEDSIGS_CONFIG = "/path/to/locked-sigs.inc"
require ${SIGGEN_LOCKEDSIGS_CONFIG}
...

2) Build db-native
$ bitbake db-native

3) Tweak db's do_configure task by adding comments
--- a/meta/recipes-support/db/db_6.0.30.bb
+++ b/meta/recipes-support/db/db_6.0.30.bb
@@ -82,6 +82,7 @@ do_configure() {
        gnu-configize --force ${S}
        export STRIP="true"
        oe_runconf
+       echo "hello"
 }

4) Build db-native and abort
$ bitbake db-native
...
ERROR: The db-native:do_configure sig (00546ab0e73b3afd50efc03f26e5f040) changed, use locked sig 9d8f2210506446e76d012d1ebd49a021 to instead
The db-native:do_compile sig (4442f979d97bdbed71a8fecbbbe41ed3) changed, use locked sig d29a7c568d10859176d733773433fe92 to instead
The db-native:do_install sig (562cb98d8eb25a7c79ded6a5359c56c1) changed, use locked sig 1e917d8d619c58987a1b2fe286e9288c to instead
The db-native:do_populate_sysroot sig (91627e36da9ac8fe489f36b67a87ed2e) changed, use locked sig f780d45dbf680e40a871820327c4c56c to instead
The db-native:do_build sig (ba289cba4a3bfda0c07aed9868fdcd5d) changed, use locked sig d9b84e5fb30011fd31bc0e7d7652fead to instead
...

5) Remove db-native sstate-cache
$ bitbake db-native -ccleansstate

6) Build db-native and abort
$ bitbake db-native
...
Locked sig is set for db-native:do_populate_sysroot (f780d45dbf680e40a871820327c4c56c) yet not in sstate cache?
...

7) Do not break the build
vim local.conf
...
SIGGEN_LOCKEDSIGS_CHECK_LEVEL = "warn"
...

8) Build db-native
$ bitbake db-native
...
WARNING Locked sig is set for db-native:do_populate_sysroot (f780d45dbf680e40a871820327c4c56c) yet not in sstate cache?
...

//Hongxu

The following changes since commit 7d80f8e9468253496a7097685aac8f468940a9c5:

  bitbake: data_smart: Clarify what 'computed' means in the data store history context (2014-09-16 22:15:24 +0100)

are available in the git repository at:

  git://git.pokylinux.org/poky-contrib hongxu/locked-sstate
  http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=hongxu/locked-sstate

Hongxu Jia (5):
  sstate: set SIGGEN_LOCKEDSIGS_CHECK_LEVEL default to error
  sstate_lockedsig.bbclass: add event handler to dump lockedsigs at
    recipe building time
  sstatesig.py: fix typo of locke sstate
  sstatesig: fix overrides behaviour to remove SIGGEN_LOCKEDSIGS_i586
  sstatesig: incremental dump lockedsigs

git@opal.openembedded.org (4):
  Richard Purdie : sstatesig/sstate: Add support for locked down sstate
    cache usage
  Richard Purdie : sstatesig: Improve to handle locking of multiple
    machines
  Randy Witt : sstatesig.py: Replace '_'s with '-' s in
    SSTATE_LOCKEDSIGS* names.
  Hongxu Jia : sstatesig: Improve the support for locked down sstate
    cache usage

 meta/classes/sstate.bbclass           |   5 ++
 meta/classes/sstate_lockedsig.bbclass |  12 ++++
 meta/lib/oe/sstatesig.py              | 106 ++++++++++++++++++++++++++++++++++
 scripts/gen-lockedsig-cache           |  40 +++++++++++++
 4 files changed, 163 insertions(+)
 create mode 100644 meta/classes/sstate_lockedsig.bbclass
 create mode 100755 scripts/gen-lockedsig-cache

-- 
1.9.1

[PATCH 5/9] sstate: set SIGGEN_LOCKEDSIGS_CHECK_LEVEL default to error

[Hongxu Jia]

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/sstate.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index d3e251c..4057c8c 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -39,6 +39,8 @@ SSTATEPOSTUNPACKFUNCS = "sstate_hardcode_path_unpack"
 SSTATEPOSTINSTFUNCS = ""
 EXTRA_STAGING_FIXMES ?= ""
 
+SIGGEN_LOCKEDSIGS_CHECK_LEVEL ?= 'error'
+
 # Specify dirs in which the shell function is executed and don't use ${B}
 # as default dirs to avoid possible race about ${B} with other task.
 sstate_create_package[dirs] = "${SSTATE_BUILDDIR}"
-- 
1.9.1

[PATCH 6/9] sstate_lockedsig.bbclass: add event handler to dump lockedsigs at recipe building time

[Hongxu Jia]

While the class in use, it adds a event handler at 'bb.event.BuildCompleted',
so after the build completed, it will dump the lockedsigs files.

Use SIGGEN_LOCKEDSIGS_CONFIG variable controls where to dump the lockedsigs file,
the default is placed into ${SSTATE_DIR}/locked-sigs.inc.

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/classes/sstate_lockedsig.bbclass | 12 ++++++++++++
 1 file changed, 12 insertions(+)
 create mode 100644 meta/classes/sstate_lockedsig.bbclass

diff --git a/meta/classes/sstate_lockedsig.bbclass b/meta/classes/sstate_lockedsig.bbclass
new file mode 100644
index 0000000..ea88260
--- /dev/null
+++ b/meta/classes/sstate_lockedsig.bbclass
@@ -0,0 +1,12 @@
+SIGGEN_LOCKEDSIGS_CONFIG ?= "${SSTATE_DIR}/locked-sigs.inc"
+addhandler sstate_dump_lockedsig
+sstate_dump_lockedsig[eventmask] = "bb.event.BuildCompleted"
+python sstate_dump_lockedsig() {
+    d = e.data
+    if e.getFailures():
+        return
+
+    if hasattr(bb.parse.siggen, "dump_lockedsigs"):
+        lockedsigs = d.getVar('SIGGEN_LOCKEDSIGS_CONFIG', True)
+        bb.parse.siggen.dump_lockedsigs(lockedsigs)
+}
-- 
1.9.1

[PATCH 7/9] sstatesig.py: fix typo of locke sstate

[Hongxu Jia]

We used sigfile to instead of where_to_dump. Fix the missing one.

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/lib/oe/sstatesig.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index df325d0..65dbac1 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -138,7 +138,7 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
         super(bb.siggen.SignatureGeneratorBasicHash, self).dump_sigtask(fn, task, stampbase, runtime)
 
     def dump_lockedsigs(self, sigfile=None):
-        if not where_to_dump:
+        if not sigfile:
             sigfile = os.getcwd() + "/locked-sigs.inc"
 
         bb.plain("Writing locked sigs to %s" % sigfile)
-- 
1.9.1

[PATCH 8/9] sstatesig: fix overrides behaviour to remove SIGGEN_LOCKEDSIGS_i586

[Hongxu Jia]

Require a sig file which SIGGEN_LOCKEDSIGS_i586 is not null, but
the actual SIGGEN_LOCKEDSIGS_i586 is null.
Invoking 'bitbake -e' and we got:
...
 3935 # $SIGGEN_LOCKEDSIGS_i586 [2 operations]
 3936 #   set /path/to/locked-sigs.inc:8576
 3938 #   del data_smart.py:406 [finalize]
 3939 #     ""
...

It was caused by the following commit:
...
(Bitbake rev: 899d45b90061eb3cf3e71029072eee42cd80930c)
Author: Richard Purdie <richard.purdie@linuxfoundation.org>
Date:   Tue May 31 23:52:50 2011 +0100

    bitbake/data_smart: Change overrides behaviour to remove expanded
      variables from the datastore
...

We add prefix 't-' to type to workaround the overrides behaviour.

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/lib/oe/sstatesig.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index 65dbac1..af7617e 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -146,7 +146,7 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
         for k in self.runtaskdeps:
             fn = k.rsplit(".",1)[0]
             t = self.lockedhashfn[fn].split(" ")[1].split(":")[5]
-            t = t.replace('_', '-')
+            t = 't-' + t.replace('_', '-')
             if t not in types:
                 types[t] = []
             types[t].append(k)
-- 
1.9.1

[PATCH 9/9] sstatesig: incremental dump lockedsigs

[Hongxu Jia]

The idea of incremental sig is:

New sig file = Old sig file (if available) + New sig items in current build.

Limit the modification within the dump_lockedsigs, and add two variables
'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.

How to config for incremental dump:
...
USER_CLASSES += "sstate_lockedsig"
SIGGEN_LOCKEDSIGS_CONFIG = "${TOPDIR}/locked-sigs.inc"
require ${SIGGEN_LOCKEDSIGS_CONFIG}
...

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
---
 meta/lib/oe/sstatesig.py | 33 +++++++++++++++++++--------------
 1 file changed, 19 insertions(+), 14 deletions(-)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index af7617e..d8cd8e6 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -93,6 +93,10 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
         self.lockedhashfn = {}
         self.machine = data.getVar("MACHINE", True)
         self.mismatch_msgs = []
+        self.lockedsigs_types = (data.getVar("SIGGEN_LOCKEDSIGS_TYPES", True) or "").split()
+        self.lockedsigs_raw = {}
+        for t in self.lockedsigs_types:
+            self.lockedsigs_raw[t] = (data.getVar("SIGGEN_LOCKEDSIGS_%s" % t, True) or "").split()
         pass
     def rundep_check(self, fn, recipename, task, dep, depname, dataCache = None):
         return sstate_rundepfilter(self, fn, recipename, task, dep, depname, dataCache)
@@ -142,28 +146,29 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
             sigfile = os.getcwd() + "/locked-sigs.inc"
 
         bb.plain("Writing locked sigs to %s" % sigfile)
-        types = {}
         for k in self.runtaskdeps:
             fn = k.rsplit(".",1)[0]
             t = self.lockedhashfn[fn].split(" ")[1].split(":")[5]
             t = 't-' + t.replace('_', '-')
-            if t not in types:
-                types[t] = []
-            types[t].append(k)
+            if t not in self.lockedsigs_types:
+                self.lockedsigs_types.append(t)
+                self.lockedsigs_raw[t] = []
+
+            pn = self.lockedpnmap[fn]
+            task = k.rsplit(".",1)[1]
+            hash = self.taskhash[k]
+            lockedsig = "%s:%s:%s" % (pn, task, hash)
+            if lockedsig not in self.lockedsigs_raw[t]:
+                self.lockedsigs_raw[t].append(lockedsig)
 
         with open(sigfile, "w") as f:
-            for t in types:
+            for t in sorted(self.lockedsigs_types):
                 f.write('SIGGEN_LOCKEDSIGS_%s = "\\\n' % t)
-                types[t].sort()
-                sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]])
-                for k in sortedk:
-                    fn = k.rsplit(".",1)[0]
-                    task = k.rsplit(".",1)[1]
-                    if k not in self.taskhash:
-                        continue
-                    f.write("    " + self.lockedpnmap[fn] + ":" + task + ":" + self.taskhash[k] + " \\\n")
+                for ls in sorted(self.lockedsigs_raw[t]):
+                    f.write("    " + ls + " \\\n")
                 f.write('    "\n')
-            f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys())))
+            f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' %
+                    (self.machine, " ".join(sorted(self.lockedsigs_types))))
 
     def checkhashes(self, missed, ret, sq_fn, sq_task, sq_hash, sq_hashfn, d):
         checklevel = d.getVar("SIGGEN_LOCKEDSIGS_CHECK_LEVEL", True)
-- 
1.9.1

Re: [PATCH 9/9] sstatesig: incremental dump lockedsigs

[Richard Purdie]

On Wed, 2014-09-17 at 16:08 +0800, Hongxu Jia wrote:
> The idea of incremental sig is:
> 
> New sig file = Old sig file (if available) + New sig items in current build.
> 
> Limit the modification within the dump_lockedsigs, and add two variables
> 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.
> 
> How to config for incremental dump:
> ...
> USER_CLASSES += "sstate_lockedsig"
> SIGGEN_LOCKEDSIGS_CONFIG = "${TOPDIR}/locked-sigs.inc"
> require ${SIGGEN_LOCKEDSIGS_CONFIG}
> ...
> 
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>

I'm fine with the idea in principle. Why can't we do something like:

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index af7617e..1da282e 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py
@@ -158,10 +158,14 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
                 sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]])
                 for k in sortedk:
                     fn = k.rsplit(".",1)[0]
+                    pn = self.lockedpnmap[fn]
                     task = k.rsplit(".",1)[1]
                     if k not in self.taskhash:
                         continue
-                    f.write("    " + self.lockedpnmap[fn] + ":" + task + ":" + self.taskhash[k] + " \\\n")
+                    if pn in self.lockedsigs and task in self.lockedsigs[pn] and self.hashtask[k] == self.lockedsigs[pn][task]:
+                        continue
+                    sigentry = pn + ":" + task + ":" + self.taskhash[k]
+                    f.write("    " + sigentry + " \\\n")
                 f.write('    "\n')
             f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys())))
 

which is substantially simpler though?

Cheers,

Richard

Re: [PATCH 9/9] sstatesig: incremental dump lockedsigs

[Otavio Salvador]

On Wed, Sep 17, 2014 at 1:16 PM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Wed, 2014-09-17 at 16:08 +0800, Hongxu Jia wrote:
>> The idea of incremental sig is:
>>
>> New sig file = Old sig file (if available) + New sig items in current build.
>>
>> Limit the modification within the dump_lockedsigs, and add two variables
>> 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.
>>
>> How to config for incremental dump:
>> ...
>> USER_CLASSES += "sstate_lockedsig"
>> SIGGEN_LOCKEDSIGS_CONFIG = "${TOPDIR}/locked-sigs.inc"
>> require ${SIGGEN_LOCKEDSIGS_CONFIG}
>> ...
>>
>> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>
> I'm fine with the idea in principle. Why can't we do something like:
...
> which is substantially simpler though?

Wouldn't it keep accumulating locked hashes for same recipes?


-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH 9/9] sstatesig: incremental dump lockedsigs

[Hongxu Jia]

On 09/18/2014 12:21 AM, Otavio Salvador wrote:
> On Wed, Sep 17, 2014 at 1:16 PM, Richard Purdie
> <richard.purdie@linuxfoundation.org> wrote:
>> On Wed, 2014-09-17 at 16:08 +0800, Hongxu Jia wrote:
>>> The idea of incremental sig is:
>>>
>>> New sig file = Old sig file (if available) + New sig items in current build.
>>>
>>> Limit the modification within the dump_lockedsigs, and add two variables
>>> 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.
>>>
>>> How to config for incremental dump:
>>> ...
>>> USER_CLASSES += "sstate_lockedsig"
>>> SIGGEN_LOCKEDSIGS_CONFIG = "${TOPDIR}/locked-sigs.inc"
>>> require ${SIGGEN_LOCKEDSIGS_CONFIG}
>>> ...
>>>
>>> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
>> I'm fine with the idea in principle. Why can't we do something like:
> ...
>> which is substantially simpler though?
> Wouldn't it keep accumulating locked hashes for same recipes?

Of course not, checking added before write to file
...

+                    if pn in self.lockedsigs and task in self.lockedsigs[pn] and self.hashtask[k] == self.lockedsigs[pn][task]:
+                        continue

...

But there are still accumulating empty SIGGEN_LOCKEDSIGS added, such as:
...
SIGGEN_LOCKEDSIGS_t-x86-64 += "\
     "
SIGGEN_LOCKEDSIGS_TYPES_qemux86 += "t-x86-64"
...

I will fix it in V3

//Hongxu

>
>

Re: [PATCH 9/9] sstatesig: incremental dump lockedsigs

[Hongxu Jia]

On 09/18/2014 12:16 AM, Richard Purdie wrote:
> On Wed, 2014-09-17 at 16:08 +0800, Hongxu Jia wrote:
>> The idea of incremental sig is:
>>
>> New sig file = Old sig file (if available) + New sig items in current build.
>>
>> Limit the modification within the dump_lockedsigs, and add two variables
>> 'self.lockedsigs_types' and 'self.lockedsigs_raw' keep old sig file.
>>
>> How to config for incremental dump:
>> ...
>> USER_CLASSES += "sstate_lockedsig"
>> SIGGEN_LOCKEDSIGS_CONFIG = "${TOPDIR}/locked-sigs.inc"
>> require ${SIGGEN_LOCKEDSIGS_CONFIG}
>> ...
>>
>> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> I'm fine with the idea in principle. Why can't we do something like:
>
> diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
> index af7617e..1da282e 100644
> --- a/meta/lib/oe/sstatesig.py
> +++ b/meta/lib/oe/sstatesig.py
> @@ -158,10 +158,14 @@ class SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
>                   sortedk = sorted(types[t], key=lambda k: self.lockedpnmap[k.rsplit(".",1)[0]])
>                   for k in sortedk:
>                       fn = k.rsplit(".",1)[0]
> +                    pn = self.lockedpnmap[fn]
>                       task = k.rsplit(".",1)[1]
>                       if k not in self.taskhash:
>                           continue
> -                    f.write("    " + self.lockedpnmap[fn] + ":" + task + ":" + self.taskhash[k] + " \\\n")
> +                    if pn in self.lockedsigs and task in self.lockedsigs[pn] and self.hashtask[k] == self.lockedsigs[pn][task]:
> +                        continue
> +                    sigentry = pn + ":" + task + ":" + self.taskhash[k]
> +                    f.write("    " + sigentry + " \\\n")
>                   f.write('    "\n')
>               f.write('SIGGEN_LOCKEDSIGS_TYPES_%s = "%s"' % (self.machine, " ".join(types.keys())))
>   
>
> which is substantially simpler though?

Agree, thanks for the improvement, I will send V3 with some trivial fixes.

//Hongxu

>
> Cheers,
>
> Richard
>
>

Re: [PATCH 6/9] sstate_lockedsig.bbclass: add event handler to dump lockedsigs at recipe building time

[Hongxu Jia]

Hi Richard,

Is this patch not acceptable, it is not merged to master or master-next.

//Hongxu

On 09/17/2014 04:08 PM, Hongxu Jia wrote:
> While the class in use, it adds a event handler at 'bb.event.BuildCompleted',
> so after the build completed, it will dump the lockedsigs files.
>
> Use SIGGEN_LOCKEDSIGS_CONFIG variable controls where to dump the lockedsigs file,
> the default is placed into ${SSTATE_DIR}/locked-sigs.inc.
>
> Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
> ---
>   meta/classes/sstate_lockedsig.bbclass | 12 ++++++++++++
>   1 file changed, 12 insertions(+)
>   create mode 100644 meta/classes/sstate_lockedsig.bbclass
>
> diff --git a/meta/classes/sstate_lockedsig.bbclass b/meta/classes/sstate_lockedsig.bbclass
> new file mode 100644
> index 0000000..ea88260
> --- /dev/null
> +++ b/meta/classes/sstate_lockedsig.bbclass
> @@ -0,0 +1,12 @@
> +SIGGEN_LOCKEDSIGS_CONFIG ?= "${SSTATE_DIR}/locked-sigs.inc"
> +addhandler sstate_dump_lockedsig
> +sstate_dump_lockedsig[eventmask] = "bb.event.BuildCompleted"
> +python sstate_dump_lockedsig() {
> +    d = e.data
> +    if e.getFailures():
> +        return
> +
> +    if hasattr(bb.parse.siggen, "dump_lockedsigs"):
> +        lockedsigs = d.getVar('SIGGEN_LOCKEDSIGS_CONFIG', True)
> +        bb.parse.siggen.dump_lockedsigs(lockedsigs)
> +}