Openembedded Core Discussions
 help / color / mirror / Atom feed
From: "André Draszik" <git@andred.net>
To: openembedded-core@lists.openembedded.org
Subject: [PATCH v2 4/4] reproducible: try to ensure reproducible xz archives
Date: Tue,  3 Mar 2020 16:05:13 +0000	[thread overview]
Message-ID: <20200303160513.17645-4-git@andred.net> (raw)
In-Reply-To: <20200303160513.17645-1-git@andred.net>

xz suffers from a reproducibility problem when not using multi-
threaded mode:
a) archives are created differently in single- vs multi-threaded
   modes
b) xz will scale down the compression level so as to be able to
   work within any memory limit given to it when being launched
  in single-threaded mode.

Thus, for reproducible xz archives we need to launch xz with
at least two threads.

Add a little sanity test, and error out otherwise, so as to
guarantee no difference due this fact.

Assumptions:
* The thread limit should be set via XZ_THREADS, not via
  modifying XZ_DEFAULTS or XZ_OPTS, or any other way
* The thread limit should not be set to xz's magic value
  zero (0), as that will lead to single-threaded mode on
  single-core systems

This patch here doesn't prevent people from shooting themselves
into the foot by changing XZ_DEFAULTS to change the number
of threads directly, but it's can serve as a hint at least.

Signed-off-by: André Draszik <git@andred.net>
---
 meta/classes/reproducible_build.bbclass | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
index 750eb950f2..e07bef87d8 100644
--- a/meta/classes/reproducible_build.bbclass
+++ b/meta/classes/reproducible_build.bbclass
@@ -35,6 +35,7 @@
 # SOURCE_DATE_EPOCH is set for all tasks that might use it (do_configure, do_compile, do_package, ...)
 
 BUILD_REPRODUCIBLE_BINARIES ??= '1'
+BUILD_REPRODUCIBLE_XZ_ARCHIVES ??= '1'
 inherit ${@oe.utils.ifelse(d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1', 'reproducible_build_simple', '')}
 
 SDE_DIR ="${WORKDIR}/source-date-epoch"
@@ -198,4 +199,8 @@ BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH"
 python () {
     if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
         d.appendVarFlag("do_unpack", "postfuncs", " do_create_source_date_epoch_stamp")
+
+        if d.getVar('BUILD_REPRODUCIBLE_XZ_ARCHIVES') == '1':
+            if int(d.getVar('XZ_THREADS')) < 2:
+                bb.fatal("Can not build reproducible XZ archives without threading")
 }
-- 
2.23.0.rc1



  parent reply	other threads:[~2020-03-03 16:05 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-03-03 16:05 [PATCH v2 1/4] lib/oe/utils: allow to set a lower bound on returned cpu_count() André Draszik
2020-03-03 16:05 ` [PATCH v2 2/4] bitbake.conf: more deterministic xz compression (threads) André Draszik
2020-03-03 16:05 ` [PATCH v2 3/4] bitbake.conf: omit XZ threads and RAM from sstate signatures André Draszik
2020-03-03 16:05 ` André Draszik [this message]
2020-03-03 16:08   ` [PATCH v2 4/4] reproducible: try to ensure reproducible xz archives André Draszik
2020-03-04  0:31     ` Otavio Salvador
2020-03-04  6:03       ` Richard Purdie

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200303160513.17645-4-git@andred.net \
    --to=git@andred.net \
    --cc=openembedded-core@lists.openembedded.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox