* [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes @ 2020-04-02 18:08 Joshua Watt 2020-04-02 20:15 ` Richard Purdie 2020-04-02 20:42 ` Richard Purdie 0 siblings, 2 replies; 6+ messages in thread From: Joshua Watt @ 2020-04-02 18:08 UTC (permalink / raw) To: openembedded-core; +Cc: Joshua Watt Don't capture the source date epoch for recipes that are not being finalized. This prevents a virtual recipe (e.g. multilib) from attempting to read the source date epoch file from the non-virtual recipes workdir. [YOCTO #13851] Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> --- meta/classes/reproducible_build.bbclass | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass index 8da40f656a..48ec9108a5 100644 --- a/meta/classes/reproducible_build.bbclass +++ b/meta/classes/reproducible_build.bbclass @@ -172,6 +172,20 @@ python create_source_date_epoch_stamp() { } def get_source_date_epoch_value(d): + onlyfinalize = d.getVar('__ONLYFINALISE') + virtual = "%s:%s" % (d.getVar("BBEXTENDCURR") or "", d.getVar("BBEXTENDVARIANT") or "") + + if onlyfinalize == "default": + onlyfinalize = ":" + + # Only capture the source date epoch if the current virtual recipe is the + # one being finalized. This prevents looking for the SDE file in another + # recipes ${WORKDIR}, where it may or may not exist + if onlyfinalize: + if onlyfinalize != virtual: + bb.debug(2, "Skipping SOURCE_DATE_EPOCH: virtual '%s' != finalize '%s'" % (virtual, onlyfinalize)) + return 0 + cached = d.getVar('__CACHED_SOURCE_DATE_EPOCH') if cached: return cached -- 2.17.1 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes 2020-04-02 18:08 [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes Joshua Watt @ 2020-04-02 20:15 ` Richard Purdie 2020-04-02 23:04 ` Joshua Watt 2020-04-02 20:42 ` Richard Purdie 1 sibling, 1 reply; 6+ messages in thread From: Richard Purdie @ 2020-04-02 20:15 UTC (permalink / raw) To: Joshua Watt, openembedded-core On Thu, 2020-04-02 at 13:08 -0500, Joshua Watt wrote: > Don't capture the source date epoch for recipes that are not being > finalized. This prevents a virtual recipe (e.g. multilib) from > attempting to read the source date epoch file from the non-virtual > recipes workdir. > > [YOCTO #13851] > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > --- > meta/classes/reproducible_build.bbclass | 14 ++++++++++++++ > 1 file changed, 14 insertions(+) Thanks, its a start. Unfortunately it pokes or assumes far too much knowledge of bitbake internals so we'll have to come up with something neater. I'll give this some further thought. Cheers, Richard > diff --git a/meta/classes/reproducible_build.bbclass > b/meta/classes/reproducible_build.bbclass > index 8da40f656a..48ec9108a5 100644 > --- a/meta/classes/reproducible_build.bbclass > +++ b/meta/classes/reproducible_build.bbclass > @@ -172,6 +172,20 @@ python create_source_date_epoch_stamp() { > } > > def get_source_date_epoch_value(d): > + onlyfinalize = d.getVar('__ONLYFINALISE') > + virtual = "%s:%s" % (d.getVar("BBEXTENDCURR") or "", > d.getVar("BBEXTENDVARIANT") or "") > + > + if onlyfinalize == "default": > + onlyfinalize = ":" > + > + # Only capture the source date epoch if the current virtual > recipe is the > + # one being finalized. This prevents looking for the SDE file in > another > + # recipes ${WORKDIR}, where it may or may not exist > + if onlyfinalize: > + if onlyfinalize != virtual: > + bb.debug(2, "Skipping SOURCE_DATE_EPOCH: virtual '%s' != > finalize '%s'" % (virtual, onlyfinalize)) > + return 0 > + > cached = d.getVar('__CACHED_SOURCE_DATE_EPOCH') > if cached: > return cached > ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes 2020-04-02 20:15 ` Richard Purdie @ 2020-04-02 23:04 ` Joshua Watt 0 siblings, 0 replies; 6+ messages in thread From: Joshua Watt @ 2020-04-02 23:04 UTC (permalink / raw) To: Richard Purdie; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 2118 bytes --] On Thu, Apr 2, 2020, 3:15 PM Richard Purdie < richard.purdie@linuxfoundation.org> wrote: > On Thu, 2020-04-02 at 13:08 -0500, Joshua Watt wrote: > > Don't capture the source date epoch for recipes that are not being > > finalized. This prevents a virtual recipe (e.g. multilib) from > > attempting to read the source date epoch file from the non-virtual > > recipes workdir. > > > > [YOCTO #13851] > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > --- > > meta/classes/reproducible_build.bbclass | 14 ++++++++++++++ > > 1 file changed, 14 insertions(+) > > Thanks, its a start. Unfortunately it pokes or assumes far too much > knowledge of bitbake internals so we'll have to come up with something > neater. > Ya I was in a hurry, but I was imagining some sort of bb.utils.is_finalizing(d) to wrap the logic > I'll give this some further thought. > > Cheers, > > Richard > > > diff --git a/meta/classes/reproducible_build.bbclass > > b/meta/classes/reproducible_build.bbclass > > index 8da40f656a..48ec9108a5 100644 > > --- a/meta/classes/reproducible_build.bbclass > > +++ b/meta/classes/reproducible_build.bbclass > > @@ -172,6 +172,20 @@ python create_source_date_epoch_stamp() { > > } > > > > def get_source_date_epoch_value(d): > > + onlyfinalize = d.getVar('__ONLYFINALISE') > > + virtual = "%s:%s" % (d.getVar("BBEXTENDCURR") or "", > > d.getVar("BBEXTENDVARIANT") or "") > > + > > + if onlyfinalize == "default": > > + onlyfinalize = ":" > > + > > + # Only capture the source date epoch if the current virtual > > recipe is the > > + # one being finalized. This prevents looking for the SDE file in > > another > > + # recipes ${WORKDIR}, where it may or may not exist > > + if onlyfinalize: > > + if onlyfinalize != virtual: > > + bb.debug(2, "Skipping SOURCE_DATE_EPOCH: virtual '%s' != > > finalize '%s'" % (virtual, onlyfinalize)) > > + return 0 > > + > > cached = d.getVar('__CACHED_SOURCE_DATE_EPOCH') > > if cached: > > return cached > > > > [-- Attachment #2: Type: text/html, Size: 3138 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes 2020-04-02 18:08 [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes Joshua Watt 2020-04-02 20:15 ` Richard Purdie @ 2020-04-02 20:42 ` Richard Purdie 2020-04-02 23:09 ` Joshua Watt 2020-04-03 14:34 ` Joshua Watt 1 sibling, 2 replies; 6+ messages in thread From: Richard Purdie @ 2020-04-02 20:42 UTC (permalink / raw) To: Joshua Watt, openembedded-core On Thu, 2020-04-02 at 13:08 -0500, Joshua Watt wrote: > Don't capture the source date epoch for recipes that are not being > finalized. This prevents a virtual recipe (e.g. multilib) from > attempting to read the source date epoch file from the non-virtual > recipes workdir. > > [YOCTO #13851] > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > --- > meta/classes/reproducible_build.bbclass | 14 ++++++++++++++ > 1 file changed, 14 insertions(+) What puzzles me with this is that we're telling bitbake to ignore the variable for hashing purposes, yet it still goes ahead and executes it. If we decide that is a bug, we can come up with the patch that follows below, which I think should also fix this issue and perhaps some others? data/siggen: Don't expand ignored variables Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> --- bitbake/lib/bb/data.py | 4 ++-- bitbake/lib/bb/siggen.py | 2 +- bitbake/lib/bb/tests/data.py | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-) diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py index 6dc02172cb..b0683c5180 100644 --- a/bitbake/lib/bb/data.py +++ b/bitbake/lib/bb/data.py @@ -365,7 +365,7 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d): #bb.note("Variable %s references %s and calls %s" % (key, str(deps), str(execs))) #d.setVarFlag(key, "vardeps", deps) -def generate_dependencies(d): +def generate_dependencies(d, whitelist): keys = set(key for key in d if not key.startswith("__")) shelldeps = set(key for key in d.getVar("__exportlist", False) if d.getVarFlag(key, "export", False) and not d.getVarFlag(key, "unexport", False)) @@ -380,7 +380,7 @@ def generate_dependencies(d): newdeps = deps[task] seen = set() while newdeps: - nextdeps = newdeps + nextdeps = newdeps - whitelist seen |= nextdeps newdeps = set() for dep in nextdeps: diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py index 8bfc45235d..4c8d81c5da 100644 --- a/bitbake/lib/bb/siggen.py +++ b/bitbake/lib/bb/siggen.py @@ -146,7 +146,7 @@ class SignatureGeneratorBasic(SignatureGenerator): def _build_data(self, fn, d): ignore_mismatch = ((d.getVar("BB_HASH_IGNORE_MISMATCH") or '') == '1') - tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d) + tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d, self.basewhitelist) taskdeps, basehash = bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache, self.basewhitelist, fn) diff --git a/bitbake/lib/bb/tests/data.py b/bitbake/lib/bb/tests/data.py index 2b137706dd..5f195047de 100644 --- a/bitbake/lib/bb/tests/data.py +++ b/bitbake/lib/bb/tests/data.py @@ -476,7 +476,7 @@ class Contains(unittest.TestCase): class TaskHash(unittest.TestCase): def test_taskhashes(self): def gettask_bashhash(taskname, d): - tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d) + tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d, set()) taskdeps, basehash = bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache, set(), "somefile") bb.warn(str(lookupcache)) return basehash["somefile:" + taskname] ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes 2020-04-02 20:42 ` Richard Purdie @ 2020-04-02 23:09 ` Joshua Watt 2020-04-03 14:34 ` Joshua Watt 1 sibling, 0 replies; 6+ messages in thread From: Joshua Watt @ 2020-04-02 23:09 UTC (permalink / raw) To: Richard Purdie; +Cc: OE-core [-- Attachment #1: Type: text/plain, Size: 4065 bytes --] On Thu, Apr 2, 2020, 3:42 PM Richard Purdie < richard.purdie@linuxfoundation.org> wrote: > On Thu, 2020-04-02 at 13:08 -0500, Joshua Watt wrote: > > Don't capture the source date epoch for recipes that are not being > > finalized. This prevents a virtual recipe (e.g. multilib) from > > attempting to read the source date epoch file from the non-virtual > > recipes workdir. > > > > [YOCTO #13851] > > > > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> > > --- > > meta/classes/reproducible_build.bbclass | 14 ++++++++++++++ > > 1 file changed, 14 insertions(+) > > What puzzles me with this is that we're telling bitbake to ignore the > variable for hashing purposes, yet it still goes ahead and executes it. > > If we decide that is a bug, we can come up with the patch that follows > below, which I think should also fix this issue and perhaps some > others? > It definitely seems like a good thing to do, question is can we make this change so close to the release? The very rare SDE race is the only thing I know of that is fixed by this, which seems like very little pay off for a lot of risk... We could do both changes; mine now for the release, then let this one soak on master for a while and backport if all is well. > data/siggen: Don't expand ignored variables > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > --- > bitbake/lib/bb/data.py | 4 ++-- > bitbake/lib/bb/siggen.py | 2 +- > bitbake/lib/bb/tests/data.py | 2 +- > 3 files changed, 4 insertions(+), 4 deletions(-) > > diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py > index 6dc02172cb..b0683c5180 100644 > --- a/bitbake/lib/bb/data.py > +++ b/bitbake/lib/bb/data.py > @@ -365,7 +365,7 @@ def build_dependencies(key, keys, shelldeps, > varflagsexcl, d): > #bb.note("Variable %s references %s and calls %s" % (key, str(deps), > str(execs))) > #d.setVarFlag(key, "vardeps", deps) > > -def generate_dependencies(d): > +def generate_dependencies(d, whitelist): > > keys = set(key for key in d if not key.startswith("__")) > shelldeps = set(key for key in d.getVar("__exportlist", False) if > d.getVarFlag(key, "export", False) and not d.getVarFlag(key, "unexport", > False)) > @@ -380,7 +380,7 @@ def generate_dependencies(d): > newdeps = deps[task] > seen = set() > while newdeps: > - nextdeps = newdeps > + nextdeps = newdeps - whitelist > seen |= nextdeps > newdeps = set() > for dep in nextdeps: > diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py > index 8bfc45235d..4c8d81c5da 100644 > --- a/bitbake/lib/bb/siggen.py > +++ b/bitbake/lib/bb/siggen.py > @@ -146,7 +146,7 @@ class SignatureGeneratorBasic(SignatureGenerator): > def _build_data(self, fn, d): > > ignore_mismatch = ((d.getVar("BB_HASH_IGNORE_MISMATCH") or '') == > '1') > - tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d) > + tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d, > self.basewhitelist) > > taskdeps, basehash = bb.data.generate_dependency_hash(tasklist, > gendeps, lookupcache, self.basewhitelist, fn) > > diff --git a/bitbake/lib/bb/tests/data.py b/bitbake/lib/bb/tests/data.py > index 2b137706dd..5f195047de 100644 > --- a/bitbake/lib/bb/tests/data.py > +++ b/bitbake/lib/bb/tests/data.py > @@ -476,7 +476,7 @@ class Contains(unittest.TestCase): > class TaskHash(unittest.TestCase): > def test_taskhashes(self): > def gettask_bashhash(taskname, d): > - tasklist, gendeps, lookupcache = > bb.data.generate_dependencies(d) > + tasklist, gendeps, lookupcache = > bb.data.generate_dependencies(d, set()) > taskdeps, basehash = > bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache, set(), > "somefile") > bb.warn(str(lookupcache)) > return basehash["somefile:" + taskname] > > [-- Attachment #2: Type: text/html, Size: 5204 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes 2020-04-02 20:42 ` Richard Purdie 2020-04-02 23:09 ` Joshua Watt @ 2020-04-03 14:34 ` Joshua Watt 1 sibling, 0 replies; 6+ messages in thread From: Joshua Watt @ 2020-04-03 14:34 UTC (permalink / raw) To: Richard Purdie, openembedded-core On 4/2/20 3:42 PM, Richard Purdie wrote: > On Thu, 2020-04-02 at 13:08 -0500, Joshua Watt wrote: >> Don't capture the source date epoch for recipes that are not being >> finalized. This prevents a virtual recipe (e.g. multilib) from >> attempting to read the source date epoch file from the non-virtual >> recipes workdir. >> >> [YOCTO #13851] >> >> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com> >> --- >> meta/classes/reproducible_build.bbclass | 14 ++++++++++++++ >> 1 file changed, 14 insertions(+) > What puzzles me with this is that we're telling bitbake to ignore the > variable for hashing purposes, yet it still goes ahead and executes it. > > If we decide that is a bug, we can come up with the patch that follows > below, which I think should also fix this issue and perhaps some > others? > > data/siggen: Don't expand ignored variables > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > --- > bitbake/lib/bb/data.py | 4 ++-- > bitbake/lib/bb/siggen.py | 2 +- > bitbake/lib/bb/tests/data.py | 2 +- > 3 files changed, 4 insertions(+), 4 deletions(-) > > diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py > index 6dc02172cb..b0683c5180 100644 > --- a/bitbake/lib/bb/data.py > +++ b/bitbake/lib/bb/data.py > @@ -365,7 +365,7 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d): > #bb.note("Variable %s references %s and calls %s" % (key, str(deps), str(execs))) > #d.setVarFlag(key, "vardeps", deps) > > -def generate_dependencies(d): > +def generate_dependencies(d, whitelist): > > keys = set(key for key in d if not key.startswith("__")) > shelldeps = set(key for key in d.getVar("__exportlist", False) if d.getVarFlag(key, "export", False) and not d.getVarFlag(key, "unexport", False)) > @@ -380,7 +380,7 @@ def generate_dependencies(d): > newdeps = deps[task] > seen = set() > while newdeps: > - nextdeps = newdeps > + nextdeps = newdeps - whitelist > seen |= nextdeps > newdeps = set() > for dep in nextdeps: > diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py > index 8bfc45235d..4c8d81c5da 100644 > --- a/bitbake/lib/bb/siggen.py > +++ b/bitbake/lib/bb/siggen.py > @@ -146,7 +146,7 @@ class SignatureGeneratorBasic(SignatureGenerator): > def _build_data(self, fn, d): > > ignore_mismatch = ((d.getVar("BB_HASH_IGNORE_MISMATCH") or '') == '1') > - tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d) > + tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d, self.basewhitelist) > > taskdeps, basehash = bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache, self.basewhitelist, fn) > > diff --git a/bitbake/lib/bb/tests/data.py b/bitbake/lib/bb/tests/data.py > index 2b137706dd..5f195047de 100644 > --- a/bitbake/lib/bb/tests/data.py > +++ b/bitbake/lib/bb/tests/data.py > @@ -476,7 +476,7 @@ class Contains(unittest.TestCase): > class TaskHash(unittest.TestCase): > def test_taskhashes(self): > def gettask_bashhash(taskname, d): > - tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d) > + tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d, set()) > taskdeps, basehash = bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache, set(), "somefile") > bb.warn(str(lookupcache)) > return basehash["somefile:" + taskname] > LGTM Review-by: Joshua Watt <JPEWhacker@gmail.com> ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-04-03 14:34 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2020-04-02 18:08 [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes Joshua Watt 2020-04-02 20:15 ` Richard Purdie 2020-04-02 23:04 ` Joshua Watt 2020-04-02 20:42 ` Richard Purdie 2020-04-02 23:09 ` Joshua Watt 2020-04-03 14:34 ` Joshua Watt
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox