* [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes
@ 2020-04-02 18:08 Joshua Watt
2020-04-02 20:15 ` Richard Purdie
2020-04-02 20:42 ` Richard Purdie
0 siblings, 2 replies; 6+ messages in thread
From: Joshua Watt @ 2020-04-02 18:08 UTC (permalink / raw)
To: openembedded-core; +Cc: Joshua Watt
Don't capture the source date epoch for recipes that are not being
finalized. This prevents a virtual recipe (e.g. multilib) from
attempting to read the source date epoch file from the non-virtual
recipes workdir.
[YOCTO #13851]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
---
meta/classes/reproducible_build.bbclass | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index 8da40f656a..48ec9108a5 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -172,6 +172,20 @@ python create_source_date_epoch_stamp() {
}
def get_source_date_epoch_value(d):
+ onlyfinalize = d.getVar('__ONLYFINALISE')
+ virtual = "%s:%s" % (d.getVar("BBEXTENDCURR") or "", d.getVar("BBEXTENDVARIANT") or "")
+
+ if onlyfinalize == "default":
+ onlyfinalize = ":"
+
+ # Only capture the source date epoch if the current virtual recipe is the
+ # one being finalized. This prevents looking for the SDE file in another
+ # recipes ${WORKDIR}, where it may or may not exist
+ if onlyfinalize:
+ if onlyfinalize != virtual:
+ bb.debug(2, "Skipping SOURCE_DATE_EPOCH: virtual '%s' != finalize '%s'" % (virtual, onlyfinalize))
+ return 0
+
cached = d.getVar('__CACHED_SOURCE_DATE_EPOCH')
if cached:
return cached
--
2.17.1
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes
2020-04-02 18:08 [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes Joshua Watt
@ 2020-04-02 20:15 ` Richard Purdie
2020-04-02 23:04 ` Joshua Watt
2020-04-02 20:42 ` Richard Purdie
1 sibling, 1 reply; 6+ messages in thread
From: Richard Purdie @ 2020-04-02 20:15 UTC (permalink / raw)
To: Joshua Watt, openembedded-core
On Thu, 2020-04-02 at 13:08 -0500, Joshua Watt wrote:
> Don't capture the source date epoch for recipes that are not being
> finalized. This prevents a virtual recipe (e.g. multilib) from
> attempting to read the source date epoch file from the non-virtual
> recipes workdir.
>
> [YOCTO #13851]
>
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
> meta/classes/reproducible_build.bbclass | 14 ++++++++++++++
> 1 file changed, 14 insertions(+)
Thanks, its a start. Unfortunately it pokes or assumes far too much
knowledge of bitbake internals so we'll have to come up with something
neater.
I'll give this some further thought.
Cheers,
Richard
> diff --git a/meta/classes/reproducible_build.bbclass
> b/meta/classes/reproducible_build.bbclass
> index 8da40f656a..48ec9108a5 100644
> --- a/meta/classes/reproducible_build.bbclass
> +++ b/meta/classes/reproducible_build.bbclass
> @@ -172,6 +172,20 @@ python create_source_date_epoch_stamp() {
> }
>
> def get_source_date_epoch_value(d):
> + onlyfinalize = d.getVar('__ONLYFINALISE')
> + virtual = "%s:%s" % (d.getVar("BBEXTENDCURR") or "",
> d.getVar("BBEXTENDVARIANT") or "")
> +
> + if onlyfinalize == "default":
> + onlyfinalize = ":"
> +
> + # Only capture the source date epoch if the current virtual
> recipe is the
> + # one being finalized. This prevents looking for the SDE file in
> another
> + # recipes ${WORKDIR}, where it may or may not exist
> + if onlyfinalize:
> + if onlyfinalize != virtual:
> + bb.debug(2, "Skipping SOURCE_DATE_EPOCH: virtual '%s' !=
> finalize '%s'" % (virtual, onlyfinalize))
> + return 0
> +
> cached = d.getVar('__CACHED_SOURCE_DATE_EPOCH')
> if cached:
> return cached
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes
2020-04-02 18:08 [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes Joshua Watt
2020-04-02 20:15 ` Richard Purdie
@ 2020-04-02 20:42 ` Richard Purdie
2020-04-02 23:09 ` Joshua Watt
2020-04-03 14:34 ` Joshua Watt
1 sibling, 2 replies; 6+ messages in thread
From: Richard Purdie @ 2020-04-02 20:42 UTC (permalink / raw)
To: Joshua Watt, openembedded-core
On Thu, 2020-04-02 at 13:08 -0500, Joshua Watt wrote:
> Don't capture the source date epoch for recipes that are not being
> finalized. This prevents a virtual recipe (e.g. multilib) from
> attempting to read the source date epoch file from the non-virtual
> recipes workdir.
>
> [YOCTO #13851]
>
> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> ---
> meta/classes/reproducible_build.bbclass | 14 ++++++++++++++
> 1 file changed, 14 insertions(+)
What puzzles me with this is that we're telling bitbake to ignore the
variable for hashing purposes, yet it still goes ahead and executes it.
If we decide that is a bug, we can come up with the patch that follows
below, which I think should also fix this issue and perhaps some
others?
data/siggen: Don't expand ignored variables
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
bitbake/lib/bb/data.py | 4 ++--
bitbake/lib/bb/siggen.py | 2 +-
bitbake/lib/bb/tests/data.py | 2 +-
3 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
index 6dc02172cb..b0683c5180 100644
--- a/bitbake/lib/bb/data.py
+++ b/bitbake/lib/bb/data.py
@@ -365,7 +365,7 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d):
#bb.note("Variable %s references %s and calls %s" % (key, str(deps), str(execs)))
#d.setVarFlag(key, "vardeps", deps)
-def generate_dependencies(d):
+def generate_dependencies(d, whitelist):
keys = set(key for key in d if not key.startswith("__"))
shelldeps = set(key for key in d.getVar("__exportlist", False) if d.getVarFlag(key, "export", False) and not d.getVarFlag(key, "unexport", False))
@@ -380,7 +380,7 @@ def generate_dependencies(d):
newdeps = deps[task]
seen = set()
while newdeps:
- nextdeps = newdeps
+ nextdeps = newdeps - whitelist
seen |= nextdeps
newdeps = set()
for dep in nextdeps:
diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
index 8bfc45235d..4c8d81c5da 100644
--- a/bitbake/lib/bb/siggen.py
+++ b/bitbake/lib/bb/siggen.py
@@ -146,7 +146,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
def _build_data(self, fn, d):
ignore_mismatch = ((d.getVar("BB_HASH_IGNORE_MISMATCH") or '') == '1')
- tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d)
+ tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d, self.basewhitelist)
taskdeps, basehash = bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache, self.basewhitelist, fn)
diff --git a/bitbake/lib/bb/tests/data.py b/bitbake/lib/bb/tests/data.py
index 2b137706dd..5f195047de 100644
--- a/bitbake/lib/bb/tests/data.py
+++ b/bitbake/lib/bb/tests/data.py
@@ -476,7 +476,7 @@ class Contains(unittest.TestCase):
class TaskHash(unittest.TestCase):
def test_taskhashes(self):
def gettask_bashhash(taskname, d):
- tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d)
+ tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d, set())
taskdeps, basehash = bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache, set(), "somefile")
bb.warn(str(lookupcache))
return basehash["somefile:" + taskname]
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes
2020-04-02 20:15 ` Richard Purdie
@ 2020-04-02 23:04 ` Joshua Watt
0 siblings, 0 replies; 6+ messages in thread
From: Joshua Watt @ 2020-04-02 23:04 UTC (permalink / raw)
To: Richard Purdie; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 2118 bytes --]
On Thu, Apr 2, 2020, 3:15 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> On Thu, 2020-04-02 at 13:08 -0500, Joshua Watt wrote:
> > Don't capture the source date epoch for recipes that are not being
> > finalized. This prevents a virtual recipe (e.g. multilib) from
> > attempting to read the source date epoch file from the non-virtual
> > recipes workdir.
> >
> > [YOCTO #13851]
> >
> > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > ---
> > meta/classes/reproducible_build.bbclass | 14 ++++++++++++++
> > 1 file changed, 14 insertions(+)
>
> Thanks, its a start. Unfortunately it pokes or assumes far too much
> knowledge of bitbake internals so we'll have to come up with something
> neater.
>
Ya I was in a hurry, but I was imagining some sort of
bb.utils.is_finalizing(d) to wrap the logic
> I'll give this some further thought.
>
> Cheers,
>
> Richard
>
> > diff --git a/meta/classes/reproducible_build.bbclass
> > b/meta/classes/reproducible_build.bbclass
> > index 8da40f656a..48ec9108a5 100644
> > --- a/meta/classes/reproducible_build.bbclass
> > +++ b/meta/classes/reproducible_build.bbclass
> > @@ -172,6 +172,20 @@ python create_source_date_epoch_stamp() {
> > }
> >
> > def get_source_date_epoch_value(d):
> > + onlyfinalize = d.getVar('__ONLYFINALISE')
> > + virtual = "%s:%s" % (d.getVar("BBEXTENDCURR") or "",
> > d.getVar("BBEXTENDVARIANT") or "")
> > +
> > + if onlyfinalize == "default":
> > + onlyfinalize = ":"
> > +
> > + # Only capture the source date epoch if the current virtual
> > recipe is the
> > + # one being finalized. This prevents looking for the SDE file in
> > another
> > + # recipes ${WORKDIR}, where it may or may not exist
> > + if onlyfinalize:
> > + if onlyfinalize != virtual:
> > + bb.debug(2, "Skipping SOURCE_DATE_EPOCH: virtual '%s' !=
> > finalize '%s'" % (virtual, onlyfinalize))
> > + return 0
> > +
> > cached = d.getVar('__CACHED_SOURCE_DATE_EPOCH')
> > if cached:
> > return cached
> >
>
>
[-- Attachment #2: Type: text/html, Size: 3138 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes
2020-04-02 20:42 ` Richard Purdie
@ 2020-04-02 23:09 ` Joshua Watt
2020-04-03 14:34 ` Joshua Watt
1 sibling, 0 replies; 6+ messages in thread
From: Joshua Watt @ 2020-04-02 23:09 UTC (permalink / raw)
To: Richard Purdie; +Cc: OE-core
[-- Attachment #1: Type: text/plain, Size: 4065 bytes --]
On Thu, Apr 2, 2020, 3:42 PM Richard Purdie <
richard.purdie@linuxfoundation.org> wrote:
> On Thu, 2020-04-02 at 13:08 -0500, Joshua Watt wrote:
> > Don't capture the source date epoch for recipes that are not being
> > finalized. This prevents a virtual recipe (e.g. multilib) from
> > attempting to read the source date epoch file from the non-virtual
> > recipes workdir.
> >
> > [YOCTO #13851]
> >
> > Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
> > ---
> > meta/classes/reproducible_build.bbclass | 14 ++++++++++++++
> > 1 file changed, 14 insertions(+)
>
> What puzzles me with this is that we're telling bitbake to ignore the
> variable for hashing purposes, yet it still goes ahead and executes it.
>
> If we decide that is a bug, we can come up with the patch that follows
> below, which I think should also fix this issue and perhaps some
> others?
>
It definitely seems like a good thing to do, question is can we make this
change so close to the release? The very rare SDE race is the only thing I
know of that is fixed by this, which seems like very little pay off for a
lot of risk... We could do both changes; mine now for the release, then let
this one soak on master for a while and backport if all is well.
> data/siggen: Don't expand ignored variables
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> bitbake/lib/bb/data.py | 4 ++--
> bitbake/lib/bb/siggen.py | 2 +-
> bitbake/lib/bb/tests/data.py | 2 +-
> 3 files changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
> index 6dc02172cb..b0683c5180 100644
> --- a/bitbake/lib/bb/data.py
> +++ b/bitbake/lib/bb/data.py
> @@ -365,7 +365,7 @@ def build_dependencies(key, keys, shelldeps,
> varflagsexcl, d):
> #bb.note("Variable %s references %s and calls %s" % (key, str(deps),
> str(execs)))
> #d.setVarFlag(key, "vardeps", deps)
>
> -def generate_dependencies(d):
> +def generate_dependencies(d, whitelist):
>
> keys = set(key for key in d if not key.startswith("__"))
> shelldeps = set(key for key in d.getVar("__exportlist", False) if
> d.getVarFlag(key, "export", False) and not d.getVarFlag(key, "unexport",
> False))
> @@ -380,7 +380,7 @@ def generate_dependencies(d):
> newdeps = deps[task]
> seen = set()
> while newdeps:
> - nextdeps = newdeps
> + nextdeps = newdeps - whitelist
> seen |= nextdeps
> newdeps = set()
> for dep in nextdeps:
> diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
> index 8bfc45235d..4c8d81c5da 100644
> --- a/bitbake/lib/bb/siggen.py
> +++ b/bitbake/lib/bb/siggen.py
> @@ -146,7 +146,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> def _build_data(self, fn, d):
>
> ignore_mismatch = ((d.getVar("BB_HASH_IGNORE_MISMATCH") or '') ==
> '1')
> - tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d)
> + tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d,
> self.basewhitelist)
>
> taskdeps, basehash = bb.data.generate_dependency_hash(tasklist,
> gendeps, lookupcache, self.basewhitelist, fn)
>
> diff --git a/bitbake/lib/bb/tests/data.py b/bitbake/lib/bb/tests/data.py
> index 2b137706dd..5f195047de 100644
> --- a/bitbake/lib/bb/tests/data.py
> +++ b/bitbake/lib/bb/tests/data.py
> @@ -476,7 +476,7 @@ class Contains(unittest.TestCase):
> class TaskHash(unittest.TestCase):
> def test_taskhashes(self):
> def gettask_bashhash(taskname, d):
> - tasklist, gendeps, lookupcache =
> bb.data.generate_dependencies(d)
> + tasklist, gendeps, lookupcache =
> bb.data.generate_dependencies(d, set())
> taskdeps, basehash =
> bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache, set(),
> "somefile")
> bb.warn(str(lookupcache))
> return basehash["somefile:" + taskname]
>
>
[-- Attachment #2: Type: text/html, Size: 5204 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes
2020-04-02 20:42 ` Richard Purdie
2020-04-02 23:09 ` Joshua Watt
@ 2020-04-03 14:34 ` Joshua Watt
1 sibling, 0 replies; 6+ messages in thread
From: Joshua Watt @ 2020-04-03 14:34 UTC (permalink / raw)
To: Richard Purdie, openembedded-core
On 4/2/20 3:42 PM, Richard Purdie wrote:
> On Thu, 2020-04-02 at 13:08 -0500, Joshua Watt wrote:
>> Don't capture the source date epoch for recipes that are not being
>> finalized. This prevents a virtual recipe (e.g. multilib) from
>> attempting to read the source date epoch file from the non-virtual
>> recipes workdir.
>>
>> [YOCTO #13851]
>>
>> Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
>> ---
>> meta/classes/reproducible_build.bbclass | 14 ++++++++++++++
>> 1 file changed, 14 insertions(+)
> What puzzles me with this is that we're telling bitbake to ignore the
> variable for hashing purposes, yet it still goes ahead and executes it.
>
> If we decide that is a bug, we can come up with the patch that follows
> below, which I think should also fix this issue and perhaps some
> others?
>
> data/siggen: Don't expand ignored variables
>
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> bitbake/lib/bb/data.py | 4 ++--
> bitbake/lib/bb/siggen.py | 2 +-
> bitbake/lib/bb/tests/data.py | 2 +-
> 3 files changed, 4 insertions(+), 4 deletions(-)
>
> diff --git a/bitbake/lib/bb/data.py b/bitbake/lib/bb/data.py
> index 6dc02172cb..b0683c5180 100644
> --- a/bitbake/lib/bb/data.py
> +++ b/bitbake/lib/bb/data.py
> @@ -365,7 +365,7 @@ def build_dependencies(key, keys, shelldeps, varflagsexcl, d):
> #bb.note("Variable %s references %s and calls %s" % (key, str(deps), str(execs)))
> #d.setVarFlag(key, "vardeps", deps)
>
> -def generate_dependencies(d):
> +def generate_dependencies(d, whitelist):
>
> keys = set(key for key in d if not key.startswith("__"))
> shelldeps = set(key for key in d.getVar("__exportlist", False) if d.getVarFlag(key, "export", False) and not d.getVarFlag(key, "unexport", False))
> @@ -380,7 +380,7 @@ def generate_dependencies(d):
> newdeps = deps[task]
> seen = set()
> while newdeps:
> - nextdeps = newdeps
> + nextdeps = newdeps - whitelist
> seen |= nextdeps
> newdeps = set()
> for dep in nextdeps:
> diff --git a/bitbake/lib/bb/siggen.py b/bitbake/lib/bb/siggen.py
> index 8bfc45235d..4c8d81c5da 100644
> --- a/bitbake/lib/bb/siggen.py
> +++ b/bitbake/lib/bb/siggen.py
> @@ -146,7 +146,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
> def _build_data(self, fn, d):
>
> ignore_mismatch = ((d.getVar("BB_HASH_IGNORE_MISMATCH") or '') == '1')
> - tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d)
> + tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d, self.basewhitelist)
>
> taskdeps, basehash = bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache, self.basewhitelist, fn)
>
> diff --git a/bitbake/lib/bb/tests/data.py b/bitbake/lib/bb/tests/data.py
> index 2b137706dd..5f195047de 100644
> --- a/bitbake/lib/bb/tests/data.py
> +++ b/bitbake/lib/bb/tests/data.py
> @@ -476,7 +476,7 @@ class Contains(unittest.TestCase):
> class TaskHash(unittest.TestCase):
> def test_taskhashes(self):
> def gettask_bashhash(taskname, d):
> - tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d)
> + tasklist, gendeps, lookupcache = bb.data.generate_dependencies(d, set())
> taskdeps, basehash = bb.data.generate_dependency_hash(tasklist, gendeps, lookupcache, set(), "somefile")
> bb.warn(str(lookupcache))
> return basehash["somefile:" + taskname]
>
LGTM
Review-by: Joshua Watt <JPEWhacker@gmail.com>
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-04-03 14:34 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-04-02 18:08 [OE-core][PATCH] classes/reproducible_build: Skip SDE for non-finalizing virtual recipes Joshua Watt
2020-04-02 20:15 ` Richard Purdie
2020-04-02 23:04 ` Joshua Watt
2020-04-02 20:42 ` Richard Purdie
2020-04-02 23:09 ` Joshua Watt
2020-04-03 14:34 ` Joshua Watt
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox