* Re: [oe-commits] Richard Purdie : sstatesig/sstate: Add support for locked down sstate cache usage
2014-09-09 9:30 ` [oe-commits] Richard Purdie : sstatesig/sstate: Add support for locked down sstate cache usage Hongxu Jia
@ 2014-09-10 6:30 ` Hongxu Jia
0 siblings, 0 replies; 2+ messages in thread
From: Hongxu Jia @ 2014-09-10 6:30 UTC (permalink / raw)
To: Richard Purdie; +Cc: openembedded-core
On 09/09/2014 05:30 PM, Hongxu Jia wrote:
> On 09/05/2014 07:29 PM, git@opal.openembedded.org wrote:
>> Module: openembedded-core.git
>> Branch: master-next
>> Commit: a12e33a584a77df4bdd9ad6a5d1a58f4dde10317
>> URL:
>> http://git.openembedded.org/?p=openembedded-core.git&a=commit;h=a12e33a584a77df4bdd9ad6a5d1a58f4dde10317
>>
>> Author: Richard Purdie <richard.purdie@linuxfoundation.org>
>> Date: Fri Sep 5 10:40:02 2014 +0100
>>
>> sstatesig/sstate: Add support for locked down sstate cache usage
>>
>> I've been giving things some thought, specifically why sstate doesn't
>> get used more and why we have people requesting external toolchains. I'm
>> guessing the issue is that people don't like how often sstate can change
>> and the lack of an easy way to lock it down.
>>
>> Locking it down is actually quite easy so patch implements some basics
>> of how you can do this (for example to a specific toolchain). With an
>> addition like this to local.conf (or wherever):
>>
>> SIGGEN_LOCKEDSIGS = "\
>> gcc-cross:do_populate_sysroot:a8d91b35b98e1494957a2ddaf4598956 \
>> eglibc:do_populate_sysroot:13e8c68553dc61f9d67564f13b9b2d67 \
>> eglibc:do_packagedata:bfca0db1782c719d373f8636282596ee \
>> gcc-cross:do_packagedata:4b601ff4f67601395ee49c46701122f6 \
>> "
>>
>> the code at the end of the email will force the hashes to those values
>> for the recipes mentioned. The system would then find and use those
>> specific objects from the sstate cache instead of trying to build
>> anything.
>>
>> Obviously this is a little simplistic, you might need to put an override
>> against this to only apply those revisions for a specific architecture
>> for example. You'd also probably want to put code in the sstate hash
>> validation code to ensure it really did install these from sstate since
>> if it didn't you'd want to abort the build.
>>
>> This patch also implements support to add to bitbake -S which dumps the
>> locked sstate checksums for each task into a ready prepared include file
>> locked-sigs.inc (currently placed into cwd). There is a function,
>> bb.parse.siggen.dump_lockedsigs() which can be called to trigger the
>> same functionality from task space.
>>
>> A warning is added to sstate.bbclass through a call back into the siggen
>> class to warn if objects are not used from the locked cache. The
>> SIGGEN_ENFORCE_LOCKEDSIGS variable controls whether this is just a
>> warning
>> or a fatal error.
>>
>> A script is provided to generate sstate directory from a locked-sigs
>> file.
>>
>> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
>>
>> ---
>>
>> meta/classes/sstate.bbclass | 3 ++
>> meta/lib/oe/sstatesig.py | 74
>> +++++++++++++++++++++++++++++++++++++++++++++
>> scripts/gen-lockedsig-cache | 40 ++++++++++++++++++++++++
>> 3 files changed, 117 insertions(+)
>>
>> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
>> index ead829e..6316336 100644
>> --- a/meta/classes/sstate.bbclass
>> +++ b/meta/classes/sstate.bbclass
>> @@ -710,6 +710,9 @@ def sstate_checkhashes(sq_fn, sq_task, sq_hash,
>> sq_hashfn, d):
>> evdata['found'].append( (sq_fn[task], sq_task[task],
>> sq_hash[task], sstatefile ) )
>> bb.event.fire(bb.event.MetadataEvent("MissedSstate",
>> evdata), d)
>> + if hasattr(bb.parse.siggen, "checkhashes"):
>> + bb.parse.siggen.checkhashes(missed, ret, sq_fn, sq_task,
>> sq_hash, sq_hashfn, d)
>> +
>
> Hi Richard,
>
> I have investigated and tested your patches, and found out invoking
> bb.parse.siggen.checkhashes in sstate_checkhashes didn't work,
> the ret and missed will alway be empty.
>
> Once locked-sigs.inc file generated and included, taskhash will never be
> changed which is replaced from locked-sigs.inc in get_taskhash, the ret
> and missed will alway be empty.
>
> ...
> WARNING: Using db-native do_fetch 29c5815138c74ce8188637729999e4a4
> WARNING: Using quilt-native do_fetch 43ac1a25892c6c7d16e2dd36c61405d8
> ...
> WARNING: ret []
> WARNING: missed []
> ...
>
Oh, it's my fault, you means a warning is added to sstate.bbclass through
a call back into the siggen class to warn if *objects are not used from the
locked cache*.
What I tested and wanted is a warn/error while hashes changed and using
locked sig instead.
They are two different things, sorry for the misunderstanding.
> We hope bitbake could support to add hook at BB_HASHCHECK_FUNCTION,
> so the users to customize their own sstate-cache hash checking mechanism,
> (Such as sign/verify sstate-cache with pgp/gpg mechanism for security
> purpose)
>
As you mentioned, I could use BB_SIGNATURE_HANDLER to do that, sorry for
the nosiy.
//Hongxu
> //Hongxu
>
>> return ret
>> BB_SETSCENE_DEPVALID = "setscene_depvalid"
>> diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
>> index 4188873..7b860c5 100644
>> --- a/meta/lib/oe/sstatesig.py
>> +++ b/meta/lib/oe/sstatesig.py
>> @@ -61,6 +61,16 @@ def sstate_rundepfilter(siggen, fn, recipename,
>> task, dep, depname, dataCache):
>> # Default to keep dependencies
>> return True
>> +def sstate_lockedsigs(d):
>> + sigs = {}
>> + lockedsigs = (d.getVar("SIGGEN_LOCKEDSIGS", True) or "").split()
>> + for ls in lockedsigs:
>> + pn, task, h = ls.split(":", 2)
>> + if pn not in sigs:
>> + sigs[pn] = {}
>> + sigs[pn][task] = h
>> + return sigs
>> +
>> class SignatureGeneratorOEBasic(bb.siggen.SignatureGeneratorBasic):
>> name = "OEBasic"
>> def init_rundepcheck(self, data):
>> @@ -75,10 +85,74 @@ class
>> SignatureGeneratorOEBasicHash(bb.siggen.SignatureGeneratorBasicHash):
>> def init_rundepcheck(self, data):
>> self.abisaferecipes =
>> (data.getVar("SIGGEN_EXCLUDERECIPES_ABISAFE", True) or "").split()
>> self.saferecipedeps =
>> (data.getVar("SIGGEN_EXCLUDE_SAFE_RECIPE_DEPS", True) or "").split()
>> + self.lockedsigs = sstate_lockedsigs(data)
>> + self.lockedhashes = {}
>> + self.lockedpnmap = {}
>> pass
>> def rundep_check(self, fn, recipename, task, dep, depname,
>> dataCache = None):
>> return sstate_rundepfilter(self, fn, recipename, task, dep,
>> depname, dataCache)
>> + def get_taskdata(self):
>> + data = super(bb.siggen.SignatureGeneratorBasicHash,
>> self).get_taskdata()
>> + return (data, self.lockedpnmap)
>> +
>> + def set_taskdata(self, data):
>> + coredata, self.lockedpnmap = data
>> + super(bb.siggen.SignatureGeneratorBasicHash,
>> self).set_taskdata(coredata)
>> +
>> + def dump_sigs(self, dataCache, options):
>> + self.dump_lockedsigs()
>> + return super(bb.siggen.SignatureGeneratorBasicHash,
>> self).dump_sigs(dataCache, options)
>> +
>> + def get_taskhash(self, fn, task, deps, dataCache):
>> + recipename = dataCache.pkg_fn[fn]
>> + self.lockedpnmap[fn] = recipename
>> + if recipename in self.lockedsigs:
>> + if task in self.lockedsigs[recipename]:
>> + k = fn + "." + task
>> + h = self.lockedsigs[recipename][task]
>> + self.lockedhashes[k] = h
>> + self.taskhash[k] = h
>> + #bb.warn("Using %s %s %s" % (recipename, task, h))
>> + return h
>> + h = super(bb.siggen.SignatureGeneratorBasicHash,
>> self).get_taskhash(fn, task, deps, dataCache)
>> + #bb.warn("%s %s %s" % (recipename, task, h))
>> + return h
>> +
>> + def dump_sigtask(self, fn, task, stampbase, runtime):
>> + k = fn + "." + task
>> + if k in self.lockedhashes:
>> + return
>> + super(bb.siggen.SignatureGeneratorBasicHash,
>> self).dump_sigtask(fn, task, stampbase, runtime)
>> +
>> + def dump_lockedsigs(self):
>> + bb.plain("Writing locked sigs to " + os.getcwd() +
>> "/locked-sigs.inc")
>> + with open("locked-sigs.inc", "w") as f:
>> + f.write('SIGGEN_LOCKEDSIGS = "\\\n')
>> + #for fn in self.taskdeps:
>> + for k in self.runtaskdeps:
>> + #k = fn + "." + task
>> + fn = k.rsplit(".",1)[0]
>> + task = k.rsplit(".",1)[1]
>> + if k not in self.taskhash:
>> + continue
>> + f.write(" " + self.lockedpnmap[fn] + ":" +
>> task + ":" + self.taskhash[k] + " \\\n")
>> + f.write(' "\n')
>> +
>> + def checkhashes(self, missed, ret, sq_fn, sq_task, sq_hash,
>> sq_hashfn, d):
>> + enforce = (d.getVar("SIGGEN_ENFORCE_LOCKEDSIGS", True) or
>> "1") == "1"
>> + msgs = []
>> + for task in range(len(sq_fn)):
>> + if task not in ret:
>> + for pn in self.lockedsigs:
>> + if sq_hash[task] in
>> self.lockedsigs[pn].itervalues():
>> + msgs.append("Locked sig is set for %s:%s
>> (%s) yet not in sstate cache?" % (pn, sq_task[task], sq_hash[task]))
>> + if msgs and enforce:
>> + bb.fatal("\n".join(msgs))
>> + elif msgs:
>> + bb.warn("\n".join(msgs))
>> +
>> +
>> # Insert these classes into siggen's namespace so it can see and
>> select them
>> bb.siggen.SignatureGeneratorOEBasic = SignatureGeneratorOEBasic
>> bb.siggen.SignatureGeneratorOEBasicHash =
>> SignatureGeneratorOEBasicHash
>> diff --git a/scripts/gen-lockedsig-cache b/scripts/gen-lockedsig-cache
>> new file mode 100755
>> index 0000000..dfb282e
>> --- /dev/null
>> +++ b/scripts/gen-lockedsig-cache
>> @@ -0,0 +1,40 @@
>> +#!/usr/bin/env python
>> +#
>> +# gen-lockedsig-cache <locked-sigs.inc> <input-cachedir>
>> <output-cachedir>
>> +#
>> +
>> +import os
>> +import sys
>> +import glob
>> +import shutil
>> +import errno
>> +
>> +def mkdir(d):
>> + try:
>> + os.makedirs(d)
>> + except OSError as e:
>> + if e.errno != errno.EEXIST:
>> + raise e
>> +
>> +if len(sys.argv) < 3:
>> + print("Incorrect number of arguments specified")
>> + sys.exit(1)
>> +
>> +sigs = []
>> +with open(sys.argv[1]) as f:
>> + for l in f.readlines():
>> + if ":" in l:
>> + sigs.append(l.split(":")[2].split()[0])
>> +
>> +files = set()
>> +for s in sigs:
>> + p = sys.argv[2] + "/" + s[:2] + "/*" + s + "*"
>> + files |= set(glob.glob(p))
>> + p = sys.argv[2] + "/*/" + s[:2] + "/*" + s + "*"
>> + files |= set(glob.glob(p))
>> +
>> +for f in files:
>> + dst = f.replace(sys.argv[2], sys.argv[3])
>> + mkdir(os.path.dirname(dst))
>> + os.link(f, dst)
>> +
>>
>
^ permalink raw reply [flat|nested] 2+ messages in thread