* [OE-core][dunfell 1/4] rpm: Deterministically set vendor macro entry
2021-10-25 23:31 [OE-core][dunfell 0/4] Patch review Steve Sakoman
@ 2021-10-25 23:31 ` Steve Sakoman
2021-10-25 23:31 ` [OE-core][dunfell 2/4] reproducible_build: Work around caching issues Steve Sakoman
` (2 subsequent siblings)
3 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2021-10-25 23:31 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
On an aarch64 build host, vendor is found to be "unknown", on x86 systems
it is "pc". This filters through to the PLATFORM tag in target rpms.
We saw reproducibility test failures where the PLATFORM tags in noarch
rpms were changing depending upon which host built them. Forcing the
vendor value to a consistent one makes things deterministic.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit f6434075b2bdfc23c683d22281b674b1e6abde77)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/recipes-devtools/rpm/rpm_4.14.2.1.bb | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/meta/recipes-devtools/rpm/rpm_4.14.2.1.bb b/meta/recipes-devtools/rpm/rpm_4.14.2.1.bb
index c93654aa8f..ab9f0e8e29 100644
--- a/meta/recipes-devtools/rpm/rpm_4.14.2.1.bb
+++ b/meta/recipes-devtools/rpm/rpm_4.14.2.1.bb
@@ -62,7 +62,8 @@ export PYTHON_ABI
# OE-core patches autoreconf to additionally run gnu-configize, which fails with this recipe
EXTRA_AUTORECONF_append = " --exclude=gnu-configize"
-EXTRA_OECONF_append = " --without-lua --enable-python --with-crypto=openssl"
+# Vendor is detected differently on x86 and aarch64 hosts and can feed into target packages
+EXTRA_OECONF_append = " --without-lua --enable-python --with-crypto=openssl --with-vendor=pc"
EXTRA_OECONF_append_libc-musl = " --disable-nls"
# --sysconfdir prevents rpm from attempting to access machine-specific configuration in sysroot/etc; we need to have it in rootfs
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [OE-core][dunfell 2/4] reproducible_build: Work around caching issues
2021-10-25 23:31 [OE-core][dunfell 0/4] Patch review Steve Sakoman
2021-10-25 23:31 ` [OE-core][dunfell 1/4] rpm: Deterministically set vendor macro entry Steve Sakoman
@ 2021-10-25 23:31 ` Steve Sakoman
2021-10-25 23:31 ` [OE-core][dunfell 3/4] classes/reproducible_build: Use atomic rename for SDE file Steve Sakoman
2021-10-25 23:31 ` [OE-core][dunfell 4/4] selftest/reproducible: adjust exclusion list for dunfell Steve Sakoman
3 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2021-10-25 23:31 UTC (permalink / raw)
To: openembedded-core
From: Richard Purdie <richard.purdie@linuxfoundation.org>
SOURCE_DATE_EPOCH can be expanded early in the parsing process before
the class extensions are applied. This can mean the directory pointed
to for the SDE can be incorrect until later in parsing. Cache the file
name in the cached value and allow it to dynamically update.
This isn't ideal but avoding expansion of the variable likely isn't
possible and I'm not sure how else to handle this. This works around
the issue until a better solution can be found.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 44dc97cd1223e4d2b635669627ec5f796838d42d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/classes/reproducible_build.bbclass | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index f06e00d70d..43cf9dc894 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -91,11 +91,14 @@ python create_source_date_epoch_stamp() {
}
def get_source_date_epoch_value(d):
- cached = d.getVar('__CACHED_SOURCE_DATE_EPOCH')
- if cached:
+ epochfile = d.getVar('SDE_FILE')
+ cached, efile = d.getVar('__CACHED_SOURCE_DATE_EPOCH') or (None, None)
+ if cached and efile == epochfile:
return cached
- epochfile = d.getVar('SDE_FILE')
+ if cached and epochfile != efile:
+ bb.debug(1, "Epoch file changed from %s to %s" % (efile, epochfile))
+
source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK'))
if os.path.isfile(epochfile):
with open(epochfile, 'r') as f:
@@ -113,7 +116,7 @@ def get_source_date_epoch_value(d):
else:
bb.debug(1, "Cannot find %s. SOURCE_DATE_EPOCH will default to %d" % (epochfile, source_date_epoch))
- d.setVar('__CACHED_SOURCE_DATE_EPOCH', str(source_date_epoch))
+ d.setVar('__CACHED_SOURCE_DATE_EPOCH', (str(source_date_epoch), epochfile))
return str(source_date_epoch)
export SOURCE_DATE_EPOCH ?= "${@get_source_date_epoch_value(d)}"
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [OE-core][dunfell 3/4] classes/reproducible_build: Use atomic rename for SDE file
2021-10-25 23:31 [OE-core][dunfell 0/4] Patch review Steve Sakoman
2021-10-25 23:31 ` [OE-core][dunfell 1/4] rpm: Deterministically set vendor macro entry Steve Sakoman
2021-10-25 23:31 ` [OE-core][dunfell 2/4] reproducible_build: Work around caching issues Steve Sakoman
@ 2021-10-25 23:31 ` Steve Sakoman
2021-10-25 23:31 ` [OE-core][dunfell 4/4] selftest/reproducible: adjust exclusion list for dunfell Steve Sakoman
3 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2021-10-25 23:31 UTC (permalink / raw)
To: openembedded-core
From: Joshua Watt <JPEWhacker@gmail.com>
If an existing source date epoch file was found during do_unpack, it was
deleted and a new one would be written in its place. This causes a race
with check-before-use code in get_source_date_epoch_value. Resolve the
problem by making do_unpack write the new source date epoch to a
temporary file, then do an atomic rename to ensure it's always present,
and change the check-before-use code to use a EAFP exception instead of
checking for file existence.
[YOCTO #14384]
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0b5e3b33187bf78a2d62cc886463e4b27d6bd228)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
meta/classes/reproducible_build.bbclass | 13 ++++++-------
1 file changed, 6 insertions(+), 7 deletions(-)
diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index 43cf9dc894..62655c2a5b 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -77,17 +77,16 @@ python create_source_date_epoch_stamp() {
import oe.reproducible
epochfile = d.getVar('SDE_FILE')
- # If it exists we need to regenerate as the sources may have changed
- if os.path.isfile(epochfile):
- bb.debug(1, "Deleting existing SOURCE_DATE_EPOCH from: %s" % epochfile)
- os.remove(epochfile)
+ tmp_file = "%s.new" % epochfile
source_date_epoch = oe.reproducible.get_source_date_epoch(d, d.getVar('S'))
bb.debug(1, "SOURCE_DATE_EPOCH: %d" % source_date_epoch)
bb.utils.mkdirhier(d.getVar('SDE_DIR'))
- with open(epochfile, 'w') as f:
+ with open(tmp_file, 'w') as f:
f.write(str(source_date_epoch))
+
+ os.rename(tmp_file, epochfile)
}
def get_source_date_epoch_value(d):
@@ -100,7 +99,7 @@ def get_source_date_epoch_value(d):
bb.debug(1, "Epoch file changed from %s to %s" % (efile, epochfile))
source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK'))
- if os.path.isfile(epochfile):
+ try:
with open(epochfile, 'r') as f:
s = f.read()
try:
@@ -113,7 +112,7 @@ def get_source_date_epoch_value(d):
bb.warn("SOURCE_DATE_EPOCH value '%s' is invalid. Reverting to SOURCE_DATE_EPOCH_FALLBACK" % s)
source_date_epoch = int(d.getVar('SOURCE_DATE_EPOCH_FALLBACK'))
bb.debug(1, "SOURCE_DATE_EPOCH: %d" % source_date_epoch)
- else:
+ except FileNotFoundError:
bb.debug(1, "Cannot find %s. SOURCE_DATE_EPOCH will default to %d" % (epochfile, source_date_epoch))
d.setVar('__CACHED_SOURCE_DATE_EPOCH', (str(source_date_epoch), epochfile))
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread* [OE-core][dunfell 4/4] selftest/reproducible: adjust exclusion list for dunfell
2021-10-25 23:31 [OE-core][dunfell 0/4] Patch review Steve Sakoman
` (2 preceding siblings ...)
2021-10-25 23:31 ` [OE-core][dunfell 3/4] classes/reproducible_build: Use atomic rename for SDE file Steve Sakoman
@ 2021-10-25 23:31 ` Steve Sakoman
3 siblings, 0 replies; 8+ messages in thread
From: Steve Sakoman @ 2021-10-25 23:31 UTC (permalink / raw)
To: openembedded-core
Signed-off-be: Steve Sakoman <steve@sakoman.com>
---
meta/lib/oeqa/selftest/cases/reproducible.py | 5 -----
1 file changed, 5 deletions(-)
diff --git a/meta/lib/oeqa/selftest/cases/reproducible.py b/meta/lib/oeqa/selftest/cases/reproducible.py
index 0e44ce4dbf..c8604a2054 100644
--- a/meta/lib/oeqa/selftest/cases/reproducible.py
+++ b/meta/lib/oeqa/selftest/cases/reproducible.py
@@ -31,7 +31,6 @@ exclude_packages = [
'bootchart2-doc',
'epiphany',
'gcr',
- 'git',
'glide',
'go-dep',
'go-helloworld',
@@ -44,7 +43,6 @@ exclude_packages = [
'libcap-ng',
'libjson',
'libproxy',
- 'lsb-release',
'lttng-tools-dbg',
'lttng-tools-ptest',
'ltp',
@@ -55,15 +53,12 @@ exclude_packages = [
'pybootchartgui',
'qemu',
'quilt-ptest',
- "rpm",
'rsync',
'ruby',
'stress-ng',
'systemd-bootchart',
'systemtap',
'valgrind-ptest',
- 'vim',
- 'webkitgtk',
]
def is_excluded(package):
--
2.25.1
^ permalink raw reply related [flat|nested] 8+ messages in thread