* [PATCH v3 01/11] reproducible_build.bbclass: initial support for binary reproducibility
2017-08-09 17:48 [PATCH v3 00/11] Reproducible binaries Juro Bystricky
@ 2017-08-09 17:48 ` Juro Bystricky
2017-08-19 8:25 ` Martin Jansa
2017-08-09 17:48 ` [PATCH v3 02/11] image-prelink.bbclass: support " Juro Bystricky
` (10 subsequent siblings)
11 siblings, 1 reply; 28+ messages in thread
From: Juro Bystricky @ 2017-08-09 17:48 UTC (permalink / raw)
To: openembedded-core; +Cc: jurobystricky
Conditionally set some environment variables in order to achieve
improved binary reproducibility. Providing BUILD_REPRODUCIBLE_BINARIES is
set to "1", we set the following environment variables:
export PYTHONHASHSEED=0
export PERL_HASH_SEED=0
export TZ="UTC"
Additionally, we export and set SOURCE_DATE_EPOCH. This is the most crucial step to
achieve binary reproducibility. The value for this variable (timestamp) is
obtained after source code for a recipe has been unpacked, but before it is patched.
If the code sources come from a GIT repo, we get the timestamp from the top
commit. (GIT repo does not preserve file mktime timestamps). It is not safe to assume
folders named "git" contain git repositories, so we check for presence of .git folder in ${S}.
Otherwise, if GIT repo is not present, we get mtime from known files suche as NEWS,
ChangeLog, etc. If this fails, we go through all files and get the timestamp
from the youngest one. We create an individual timestamp for each recipe.
The timestamp is stored in the file 'src_date_epoch.txt'. Later on, each task
reads this file and sets SOURCE_DATE_EPOCH based on the value found in the file.
The file src_date_epoch.txt file is re-used if found. This can be the file
we previously created ourselves, or it can be a file provided by a user via
a recipe.
[YOCTO#11178]
[YOCTO#11179]
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
meta/classes/base.bbclass | 4 ++
meta/classes/reproducible_build.bbclass | 108 ++++++++++++++++++++++++++++++++
2 files changed, 112 insertions(+)
create mode 100644 meta/classes/reproducible_build.bbclass
diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 8c86977..bf79eb9 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -9,6 +9,7 @@ inherit utils
inherit utility-tasks
inherit metadata_scm
inherit logging
+inherit reproducible_build
OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license"
OE_IMPORTS[type] = "list"
@@ -166,6 +167,7 @@ python base_do_unpack() {
try:
fetcher = bb.fetch2.Fetch(src_uri, d)
fetcher.unpack(d.getVar('WORKDIR'))
+ create_src_date_epoch_stamp(d)
except bb.fetch2.BBFetchException as e:
bb.fatal(str(e))
}
@@ -386,6 +388,7 @@ def set_packagetriplet(d):
settriplet(d, "PKGMLTRIPLETS", archs, tos, tvs)
+
python () {
import string, re
@@ -685,6 +688,7 @@ python () {
bb.warn("Recipe %s is marked as only being architecture specific but seems to have machine specific packages?! The recipe may as well mark itself as machine specific directly." % d.getVar("PN"))
}
+
addtask cleansstate after do_clean
python do_cleansstate() {
sstate_clean_cachefiles(d)
diff --git a/meta/classes/reproducible_build.bbclass b/meta/classes/reproducible_build.bbclass
new file mode 100644
index 0000000..af8db95
--- /dev/null
+++ b/meta/classes/reproducible_build.bbclass
@@ -0,0 +1,108 @@
+
+BUILD_REPRODUCIBLE_BINARIES ??= "0"
+BUILD_REPRODUCIBLE_BINARIES[export] = "1"
+
+# Unix timestamp
+REPRODUCIBLE_TIMESTAMP_ROOTFS ??= ""
+
+def get_src_date_epoch_quick(d, path):
+ import subprocess
+ src_date_epoch = 0
+ saved_cwd = os.getcwd()
+ os.chdir(path)
+ if os.path.isdir(".git"):
+ try:
+ src_date_epoch = int(subprocess.check_output(['git','log','-1','--pretty=%ct']))
+ except subprocess.CalledProcessError as grepexc:
+ bb.warn("Not a git repository in .git folder? error:%d" % (grepexc.returncode))
+ else:
+ known_files = set(["NEWS", "ChangeLog", "Changelog", "CHANGES"])
+
+ for file in known_files:
+ if os.path.isfile(file):
+ mtime = int(os.path.getmtime(file))
+
+ # There may be more than one "known_file" present.
+ # If so, use the youngest one
+ if mtime > src_date_epoch:
+ src_date_epoch = mtime
+
+ os.chdir(saved_cwd)
+ return src_date_epoch
+
+
+def create_src_date_epoch_stamp(d):
+ if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
+ path = d.getVar('S')
+
+ epochfile = os.path.join(path,'src_date_epoch.txt')
+ if os.path.isfile(epochfile):
+ bb.debug(1, " path: %s reusing src_date_epoch.txt" % epochfile)
+ return
+
+ filename_dbg = None
+ src_date_epoch = get_src_date_epoch_quick(d, path)
+
+ if src_date_epoch == 0:
+ exclude = set(["temp", "licenses", "patches", "recipe-sysroot-native", "recipe-sysroot", "pseudo"])
+ for root, dirs, files in os.walk(path, topdown=True):
+ files = [f for f in files if not f[0] == '.']
+ dirs[:] = [d for d in dirs if d not in exclude]
+
+ for fname in files:
+ filename = os.path.join(root, fname)
+ try:
+ mtime = int(os.path.getmtime(filename))
+ except:
+ mtime = 0
+ if mtime > src_date_epoch:
+ src_date_epoch = mtime
+ filename_dbg = filename
+
+ # Most likely an empty folder
+ if src_date_epoch == 0:
+ bb.warn("Unable to determine src_date_epoch! path:%s" % path)
+
+ f = open(os.path.join(path,'src_date_epoch.txt'), 'w')
+ f.write(str(src_date_epoch))
+ f.close()
+
+ if filename_dbg != None:
+ bb.debug(1," src_date_epoch %d derived from: %s" % (src_date_epoch, filename_dbg))
+ #bb.warn(" src_date_epoch %d derived from: %s" % (src_date_epoch, filename_dbg))
+
+export PYTHONHASHSEED
+export PERL_HASH_SEED
+export SOURCE_DATE_EPOCH
+
+BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH PYTHONHASHSEED PERL_HASH_SEED "
+
+python () {
+ import string, re
+
+ # Create reproducible_environment
+
+ if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
+ import subprocess
+ d.setVar('PYTHONHASHSEED', '0')
+ d.setVar('PERL_HASH_SEED', '0')
+ d.setVar('TZ', 'UTC')
+ path = d.getVar('S')
+ epochfile = os.path.join(path,'src_date_epoch.txt')
+ if os.path.isfile(epochfile):
+ f = open(epochfile, 'r')
+ src_date_epoch = f.read()
+ f.close()
+ bb.debug(1, "src_date_epoch stamp found ---> stamp %s" % src_date_epoch)
+ d.setVar('SOURCE_DATE_EPOCH', src_date_epoch)
+ else:
+ bb.debug(1, "src_date_epoch stamp not found.")
+ d.setVar('SOURCE_DATE_EPOCH', '0')
+ else:
+ if 'PYTHONHASHSEED' in os.environ:
+ del os.environ['PYTHONHASHSEED']
+ if 'PERL_HASH_SEED' in os.environ:
+ del os.environ['PERL_HASH_SEED']
+ if 'SOURCE_DATE_EPOCH' in os.environ:
+ del os.environ['SOURCE_DATE_EPOCH']
+}
--
2.7.4
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [PATCH v3 01/11] reproducible_build.bbclass: initial support for binary reproducibility
2017-08-09 17:48 ` [PATCH v3 01/11] reproducible_build.bbclass: initial support for binary reproducibility Juro Bystricky
@ 2017-08-19 8:25 ` Martin Jansa
2017-08-19 16:08 ` Bystricky, Juro
0 siblings, 1 reply; 28+ messages in thread
From: Martin Jansa @ 2017-08-19 8:25 UTC (permalink / raw)
To: Juro Bystricky
Cc: Juro Bystricky, Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 8180 bytes --]
Would it make sense to create src_date_epoch.txt outside ${S} directory?
In some recipes (which weren't picky what they install from ${S} to ${D})
I've noticed that src_date_epoch.txt is now included in their package (I've
noticed, because 2 different packages were trying to install this file on
the same path in rootfs).
Can we use temp or ${WORKDIR} for this file? ${S} might be also be
completely archived with archiver.bbclass, not sure if you want
src_date_epoch.txt part of that or not.
Regards,
On Wed, Aug 9, 2017 at 7:48 PM, Juro Bystricky <juro.bystricky@intel.com>
wrote:
> Conditionally set some environment variables in order to achieve
> improved binary reproducibility. Providing BUILD_REPRODUCIBLE_BINARIES is
> set to "1", we set the following environment variables:
>
> export PYTHONHASHSEED=0
> export PERL_HASH_SEED=0
> export TZ="UTC"
>
> Additionally, we export and set SOURCE_DATE_EPOCH. This is the most
> crucial step to
> achieve binary reproducibility. The value for this variable (timestamp) is
> obtained after source code for a recipe has been unpacked, but before it
> is patched.
> If the code sources come from a GIT repo, we get the timestamp from the top
> commit. (GIT repo does not preserve file mktime timestamps). It is not
> safe to assume
> folders named "git" contain git repositories, so we check for presence of
> .git folder in ${S}.
> Otherwise, if GIT repo is not present, we get mtime from known files suche
> as NEWS,
> ChangeLog, etc. If this fails, we go through all files and get the
> timestamp
> from the youngest one. We create an individual timestamp for each recipe.
> The timestamp is stored in the file 'src_date_epoch.txt'. Later on, each
> task
> reads this file and sets SOURCE_DATE_EPOCH based on the value found in the
> file.
>
> The file src_date_epoch.txt file is re-used if found. This can be the file
> we previously created ourselves, or it can be a file provided by a user via
> a recipe.
>
> [YOCTO#11178]
> [YOCTO#11179]
>
> Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
> ---
> meta/classes/base.bbclass | 4 ++
> meta/classes/reproducible_build.bbclass | 108
> ++++++++++++++++++++++++++++++++
> 2 files changed, 112 insertions(+)
> create mode 100644 meta/classes/reproducible_build.bbclass
>
> diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
> index 8c86977..bf79eb9 100644
> --- a/meta/classes/base.bbclass
> +++ b/meta/classes/base.bbclass
> @@ -9,6 +9,7 @@ inherit utils
> inherit utility-tasks
> inherit metadata_scm
> inherit logging
> +inherit reproducible_build
>
> OE_IMPORTS += "os sys time oe.path oe.utils oe.types oe.package
> oe.packagegroup oe.sstatesig oe.lsb oe.cachedpath oe.license"
> OE_IMPORTS[type] = "list"
> @@ -166,6 +167,7 @@ python base_do_unpack() {
> try:
> fetcher = bb.fetch2.Fetch(src_uri, d)
> fetcher.unpack(d.getVar('WORKDIR'))
> + create_src_date_epoch_stamp(d)
> except bb.fetch2.BBFetchException as e:
> bb.fatal(str(e))
> }
> @@ -386,6 +388,7 @@ def set_packagetriplet(d):
>
> settriplet(d, "PKGMLTRIPLETS", archs, tos, tvs)
>
> +
> python () {
> import string, re
>
> @@ -685,6 +688,7 @@ python () {
> bb.warn("Recipe %s is marked as only being architecture
> specific but seems to have machine specific packages?! The recipe may as
> well mark itself as machine specific directly." % d.getVar("PN"))
> }
>
> +
> addtask cleansstate after do_clean
> python do_cleansstate() {
> sstate_clean_cachefiles(d)
> diff --git a/meta/classes/reproducible_build.bbclass
> b/meta/classes/reproducible_build.bbclass
> new file mode 100644
> index 0000000..af8db95
> --- /dev/null
> +++ b/meta/classes/reproducible_build.bbclass
> @@ -0,0 +1,108 @@
> +
> +BUILD_REPRODUCIBLE_BINARIES ??= "0"
> +BUILD_REPRODUCIBLE_BINARIES[export] = "1"
> +
> +# Unix timestamp
> +REPRODUCIBLE_TIMESTAMP_ROOTFS ??= ""
> +
> +def get_src_date_epoch_quick(d, path):
> + import subprocess
> + src_date_epoch = 0
> + saved_cwd = os.getcwd()
> + os.chdir(path)
> + if os.path.isdir(".git"):
> + try:
> + src_date_epoch = int(subprocess.check_output(['
> git','log','-1','--pretty=%ct']))
> + except subprocess.CalledProcessError as grepexc:
> + bb.warn("Not a git repository in .git folder? error:%d" %
> (grepexc.returncode))
> + else:
> + known_files = set(["NEWS", "ChangeLog", "Changelog", "CHANGES"])
> +
> + for file in known_files:
> + if os.path.isfile(file):
> + mtime = int(os.path.getmtime(file))
> +
> + # There may be more than one "known_file" present.
> + # If so, use the youngest one
> + if mtime > src_date_epoch:
> + src_date_epoch = mtime
> +
> + os.chdir(saved_cwd)
> + return src_date_epoch
> +
> +
> +def create_src_date_epoch_stamp(d):
> + if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
> + path = d.getVar('S')
> +
> + epochfile = os.path.join(path,'src_date_epoch.txt')
> + if os.path.isfile(epochfile):
> + bb.debug(1, " path: %s reusing src_date_epoch.txt" %
> epochfile)
> + return
> +
> + filename_dbg = None
> + src_date_epoch = get_src_date_epoch_quick(d, path)
> +
> + if src_date_epoch == 0:
> + exclude = set(["temp", "licenses", "patches",
> "recipe-sysroot-native", "recipe-sysroot", "pseudo"])
> + for root, dirs, files in os.walk(path, topdown=True):
> + files = [f for f in files if not f[0] == '.']
> + dirs[:] = [d for d in dirs if d not in exclude]
> +
> + for fname in files:
> + filename = os.path.join(root, fname)
> + try:
> + mtime = int(os.path.getmtime(filename))
> + except:
> + mtime = 0
> + if mtime > src_date_epoch:
> + src_date_epoch = mtime
> + filename_dbg = filename
> +
> + # Most likely an empty folder
> + if src_date_epoch == 0:
> + bb.warn("Unable to determine src_date_epoch! path:%s" % path)
> +
> + f = open(os.path.join(path,'src_date_epoch.txt'), 'w')
> + f.write(str(src_date_epoch))
> + f.close()
> +
> + if filename_dbg != None:
> + bb.debug(1," src_date_epoch %d derived from: %s" %
> (src_date_epoch, filename_dbg))
> + #bb.warn(" src_date_epoch %d derived from: %s" %
> (src_date_epoch, filename_dbg))
> +
> +export PYTHONHASHSEED
> +export PERL_HASH_SEED
> +export SOURCE_DATE_EPOCH
> +
> +BB_HASHBASE_WHITELIST += "SOURCE_DATE_EPOCH PYTHONHASHSEED PERL_HASH_SEED
> "
> +
> +python () {
> + import string, re
> +
> + # Create reproducible_environment
> +
> + if d.getVar('BUILD_REPRODUCIBLE_BINARIES') == '1':
> + import subprocess
> + d.setVar('PYTHONHASHSEED', '0')
> + d.setVar('PERL_HASH_SEED', '0')
> + d.setVar('TZ', 'UTC')
> + path = d.getVar('S')
> + epochfile = os.path.join(path,'src_date_epoch.txt')
> + if os.path.isfile(epochfile):
> + f = open(epochfile, 'r')
> + src_date_epoch = f.read()
> + f.close()
> + bb.debug(1, "src_date_epoch stamp found ---> stamp %s" %
> src_date_epoch)
> + d.setVar('SOURCE_DATE_EPOCH', src_date_epoch)
> + else:
> + bb.debug(1, "src_date_epoch stamp not found.")
> + d.setVar('SOURCE_DATE_EPOCH', '0')
> + else:
> + if 'PYTHONHASHSEED' in os.environ:
> + del os.environ['PYTHONHASHSEED']
> + if 'PERL_HASH_SEED' in os.environ:
> + del os.environ['PERL_HASH_SEED']
> + if 'SOURCE_DATE_EPOCH' in os.environ:
> + del os.environ['SOURCE_DATE_EPOCH']
> +}
> --
> 2.7.4
>
>
[-- Attachment #2: Type: text/html, Size: 10786 bytes --]
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v3 01/11] reproducible_build.bbclass: initial support for binary reproducibility
2017-08-19 8:25 ` Martin Jansa
@ 2017-08-19 16:08 ` Bystricky, Juro
0 siblings, 0 replies; 28+ messages in thread
From: Bystricky, Juro @ 2017-08-19 16:08 UTC (permalink / raw)
To: Martin Jansa
Cc: Juro Bystricky, Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 1047 bytes --]
Yes, this should not be a problem, it is actually a good idea. I am currently running a test buid witth
the file created in ${WORKDIR}.
I will send a patch to ML soon.
________________________________
From: Martin Jansa [martin.jansa@gmail.com]
Sent: Saturday, August 19, 2017 1:25 AM
To: Bystricky, Juro
Cc: Patches and discussions about the oe-core layer; Richard Purdie; Joshua G Lock; Burton, Ross; Juro Bystricky
Subject: Re: [PATCH v3 01/11] reproducible_build.bbclass: initial support for binary reproducibility
Would it make sense to create src_date_epoch.txt outside ${S} directory?
In some recipes (which weren't picky what they install from ${S} to ${D}) I've noticed that src_date_epoch.txt is now included in their package (I've noticed, because 2 different packages were trying to install this file on the same path in rootfs).
Can we use temp or ${WORKDIR} for this file? ${S} might be also be completely archived with archiver.bbclass, not sure if you want src_date_epoch.txt part of that or not.
Regards,
[-- Attachment #2: Type: text/html, Size: 2482 bytes --]
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH v3 02/11] image-prelink.bbclass: support binary reproducibility
2017-08-09 17:48 [PATCH v3 00/11] Reproducible binaries Juro Bystricky
2017-08-09 17:48 ` [PATCH v3 01/11] reproducible_build.bbclass: initial support for binary reproducibility Juro Bystricky
@ 2017-08-09 17:48 ` Juro Bystricky
2017-08-17 19:54 ` Leonardo Sandoval
2017-08-09 17:48 ` [PATCH v3 03/11] rootfs-postcommands.bbclass: " Juro Bystricky
` (9 subsequent siblings)
11 siblings, 1 reply; 28+ messages in thread
From: Juro Bystricky @ 2017-08-09 17:48 UTC (permalink / raw)
To: openembedded-core; +Cc: jurobystricky
Conditionally support binary reproducibility in built images.
If BUILD_REPRODUCIBLE_BINARIES = 1 then:
1. Do not randomize library addresses
2. Set/export PRELINK_TIMESTAMP to a reproducible value.
If REPRODUCIBLE_TIMESTAMP_ROOTFS is specified, then the value will
be used. Otherwise the timestamp will be derived from the top git commit.
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
meta/classes/image-prelink.bbclass | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git a/meta/classes/image-prelink.bbclass b/meta/classes/image-prelink.bbclass
index 4157df0..e833d47 100644
--- a/meta/classes/image-prelink.bbclass
+++ b/meta/classes/image-prelink.bbclass
@@ -36,7 +36,17 @@ prelink_image () {
dynamic_loader=$(linuxloader)
# prelink!
- ${STAGING_SBINDIR_NATIVE}/prelink --root ${IMAGE_ROOTFS} -amR -N -c ${sysconfdir}/prelink.conf --dynamic-linker $dynamic_loader
+ if [ "$BUILD_REPRODUCIBLE_BINARIES" = "1" ]; then
+ bbnote " prelink: BUILD_REPRODUCIBLE_BINARIES..."
+ if [ "$REPRODUCIBLE_TIMESTAMP_ROOTFS" = "" ]; then
+ export PRELINK_TIMESTAMP=`git log -1 --pretty=%ct `
+ else
+ export PRELINK_TIMESTAMP=$REPRODUCIBLE_TIMESTAMP_ROOTFS
+ fi
+ ${STAGING_SBINDIR_NATIVE}/prelink --root ${IMAGE_ROOTFS} -am -N -c ${sysconfdir}/prelink.conf --dynamic-linker $dynamic_loader
+ else
+ ${STAGING_SBINDIR_NATIVE}/prelink --root ${IMAGE_ROOTFS} -amR -N -c ${sysconfdir}/prelink.conf --dynamic-linker $dynamic_loader
+ fi
# Remove the prelink.conf if we had to add it.
if [ "$dummy_prelink_conf" = "true" ]; then
--
2.7.4
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [PATCH v3 02/11] image-prelink.bbclass: support binary reproducibility
2017-08-09 17:48 ` [PATCH v3 02/11] image-prelink.bbclass: support " Juro Bystricky
@ 2017-08-17 19:54 ` Leonardo Sandoval
0 siblings, 0 replies; 28+ messages in thread
From: Leonardo Sandoval @ 2017-08-17 19:54 UTC (permalink / raw)
To: Juro Bystricky; +Cc: jurobystricky, openembedded-core
On Wed, 2017-08-09 at 10:48 -0700, Juro Bystricky wrote:
> Conditionally support binary reproducibility in built images.
> If BUILD_REPRODUCIBLE_BINARIES = 1 then:
>
> 1. Do not randomize library addresses
> 2. Set/export PRELINK_TIMESTAMP to a reproducible value.
> If REPRODUCIBLE_TIMESTAMP_ROOTFS is specified, then the value will
> be used. Otherwise the timestamp will be derived from the top git commit.
>
> Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
> ---
> meta/classes/image-prelink.bbclass | 12 +++++++++++-
> 1 file changed, 11 insertions(+), 1 deletion(-)
>
> diff --git a/meta/classes/image-prelink.bbclass b/meta/classes/image-prelink.bbclass
> index 4157df0..e833d47 100644
> --- a/meta/classes/image-prelink.bbclass
> +++ b/meta/classes/image-prelink.bbclass
> @@ -36,7 +36,17 @@ prelink_image () {
> dynamic_loader=$(linuxloader)
>
> # prelink!
> - ${STAGING_SBINDIR_NATIVE}/prelink --root ${IMAGE_ROOTFS} -amR -N -c ${sysconfdir}/prelink.conf --dynamic-linker $dynamic_loader
> + if [ "$BUILD_REPRODUCIBLE_BINARIES" = "1" ]; then
> + bbnote " prelink: BUILD_REPRODUCIBLE_BINARIES..."
> + if [ "$REPRODUCIBLE_TIMESTAMP_ROOTFS" = "" ]; then
> + export PRELINK_TIMESTAMP=`git log -1 --pretty=%ct `
as Chris suggested in other email, better to used $() instead of ``
> + else
> + export PRELINK_TIMESTAMP=$REPRODUCIBLE_TIMESTAMP_ROOTFS
> + fi
> + ${STAGING_SBINDIR_NATIVE}/prelink --root ${IMAGE_ROOTFS} -am -N -c ${sysconfdir}/prelink.conf --dynamic-linker $dynamic_loader
> + else
> + ${STAGING_SBINDIR_NATIVE}/prelink --root ${IMAGE_ROOTFS} -amR -N -c ${sysconfdir}/prelink.conf --dynamic-linker $dynamic_loader
> + fi
>
> # Remove the prelink.conf if we had to add it.
> if [ "$dummy_prelink_conf" = "true" ]; then
> --
> 2.7.4
>
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH v3 03/11] rootfs-postcommands.bbclass: support binary reproducibility
2017-08-09 17:48 [PATCH v3 00/11] Reproducible binaries Juro Bystricky
2017-08-09 17:48 ` [PATCH v3 01/11] reproducible_build.bbclass: initial support for binary reproducibility Juro Bystricky
2017-08-09 17:48 ` [PATCH v3 02/11] image-prelink.bbclass: support " Juro Bystricky
@ 2017-08-09 17:48 ` Juro Bystricky
2017-08-09 17:48 ` [PATCH v3 04/11] busybox.inc: improve reproducibility Juro Bystricky
` (8 subsequent siblings)
11 siblings, 0 replies; 28+ messages in thread
From: Juro Bystricky @ 2017-08-09 17:48 UTC (permalink / raw)
To: openembedded-core; +Cc: jurobystricky
Conditionally support binary reproducibility of rootfs images.
If BUILD_REPRODUCIBLE_BINARIES = 1 then:
1. set /etc/timestamp to a reproducible value
2. set /etc/version to a reproducible value
3. set /etc/gconf: set mtime in all %gconf.xml to reproducible values
The reproducible value is taken from the variable REPRODUCIBLE_TIMESTAMP_ROOTFS.
If the variable is not specified, the timestamp value is derived from
the top git commit.
[YOCTO#11176]
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
meta/classes/rootfs-postcommands.bbclass | 27 ++++++++++++++++++++++++---
1 file changed, 24 insertions(+), 3 deletions(-)
diff --git a/meta/classes/rootfs-postcommands.bbclass b/meta/classes/rootfs-postcommands.bbclass
index 78f7c55..46cb530 100644
--- a/meta/classes/rootfs-postcommands.bbclass
+++ b/meta/classes/rootfs-postcommands.bbclass
@@ -48,6 +48,7 @@ ROOTFS_POSTPROCESS_COMMAND_append_qemuall = "${SSH_DISABLE_DNS_LOOKUP}"
SORT_PASSWD_POSTPROCESS_COMMAND ??= " sort_passwd; "
python () {
d.appendVar('ROOTFS_POSTPROCESS_COMMAND', '${SORT_PASSWD_POSTPROCESS_COMMAND}')
+ d.appendVar('ROOTFS_POSTPROCESS_COMMAND', 'rootfs_reproducible;')
}
systemd_create_users () {
@@ -245,10 +246,12 @@ python write_image_manifest () {
os.symlink(os.path.basename(manifest_name), manifest_link)
}
-# Can be use to create /etc/timestamp during image construction to give a reasonably
+# Can be used to create /etc/timestamp during image construction to give a reasonably
# sane default time setting
rootfs_update_timestamp () {
- date -u +%4Y%2m%2d%2H%2M%2S >${IMAGE_ROOTFS}/etc/timestamp
+ if [ "$BUILD_REPRODUCIBLE_BINARIES" = "0" ]; then
+ date -u +%4Y%2m%2d%2H%2M%2S >${IMAGE_ROOTFS}/etc/timestamp
+ fi
}
# Prevent X from being started
@@ -288,7 +291,6 @@ rootfs_sysroot_relativelinks () {
sysroot-relativelinks.py ${SDK_OUTPUT}/${SDKTARGETSYSROOT}
}
-
# Generated test data json file
python write_image_test_data() {
from oe.data import export2json
@@ -304,3 +306,22 @@ python write_image_test_data() {
os.remove(testdata_link)
os.symlink(os.path.basename(testdata), testdata_link)
}
+
+# Perform any additional adjustments needed to make rootf binary reproducible
+rootfs_reproducible () {
+ if [ "$BUILD_REPRODUCIBLE_BINARIES" = "1" ]; then
+ if [ "$REPRODUCIBLE_TIMESTAMP_ROOTFS" = "" ]; then
+ REPRODUCIBLE_TIMESTAMP_ROOTFS=`git log -1 --pretty=%ct`
+ fi
+
+ # Convert UTC into %4Y%2m%2d%2H%2M%2S
+ sformatted=`date -u -d @$REPRODUCIBLE_TIMESTAMP_ROOTFS +%4Y%2m%2d%2H%2M%2S`
+ echo $sformatted > ${IMAGE_ROOTFS}/etc/version
+ bbnote "rootfs_reproducible: set /etc/version to $sformatted"
+ echo $sformatted > ${IMAGE_ROOTFS}/etc/timestamp
+ bbnote "rootfs_reproducible: set /etc/timestamp to $sformatted"
+
+ find ${IMAGE_ROOTFS}/etc/gconf -name '%gconf.xml' -print0 | xargs -0r \
+ sed -i -e 's@\bmtime="[0-9][0-9]*"@mtime="'$REPRODUCIBLE_TIMESTAMP_ROOTFS'"@g'
+ fi
+}
--
2.7.4
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH v3 04/11] busybox.inc: improve reproducibility
2017-08-09 17:48 [PATCH v3 00/11] Reproducible binaries Juro Bystricky
` (2 preceding siblings ...)
2017-08-09 17:48 ` [PATCH v3 03/11] rootfs-postcommands.bbclass: " Juro Bystricky
@ 2017-08-09 17:48 ` Juro Bystricky
2017-08-09 17:48 ` [PATCH v3 05/11] image.bbclass: support binary reproducibility Juro Bystricky
` (7 subsequent siblings)
11 siblings, 0 replies; 28+ messages in thread
From: Juro Bystricky @ 2017-08-09 17:48 UTC (permalink / raw)
To: openembedded-core; +Cc: jurobystricky
For reproducible builds do not generate build timestamp as part of
the version string.
Remove host tools references from .config file.
With this patch all eight busybox packages are built as
binary reproducible.
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
meta/recipes-core/busybox/busybox.inc | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/meta/recipes-core/busybox/busybox.inc b/meta/recipes-core/busybox/busybox.inc
index b8edd39..735b1f6 100644
--- a/meta/recipes-core/busybox/busybox.inc
+++ b/meta/recipes-core/busybox/busybox.inc
@@ -102,6 +102,9 @@ python () {
}
do_prepare_config () {
+ if [ "$BUILD_REPRODUCIBLE_BINARIES" = "1" ]; then
+ export KCONFIG_NOTIMESTAMP=1
+ fi
sed -e '/CONFIG_STATIC/d' \
< ${WORKDIR}/defconfig > ${S}/.config
echo "# CONFIG_STATIC is not set" >> .config
@@ -118,6 +121,7 @@ do_prepare_config () {
${S}/.config.oe-tmp > ${S}/.config
fi
sed -i 's/CONFIG_IFUPDOWN_UDHCPC_CMD_OPTIONS="-R -n"/CONFIG_IFUPDOWN_UDHCPC_CMD_OPTIONS="-R -b"/' ${S}/.config
+ sed -i 's|${DEBUG_PREFIX_MAP}||g' ${S}/.config
}
# returns all the elements from the src uri that are .cfg files
@@ -138,6 +142,9 @@ do_configure () {
do_compile() {
unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS
+ if [ "$BUILD_REPRODUCIBLE_BINARIES" = "1" ]; then
+ export KCONFIG_NOTIMESTAMP=1
+ fi
if [ "${BUSYBOX_SPLIT_SUID}" = "1" -a x`grep "CONFIG_FEATURE_INDIVIDUAL=y" .config` = x ]; then
# split the .config into two parts, and make two busybox binaries
if [ -e .config.orig ]; then
--
2.7.4
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH v3 05/11] image.bbclass: support binary reproducibility
2017-08-09 17:48 [PATCH v3 00/11] Reproducible binaries Juro Bystricky
` (3 preceding siblings ...)
2017-08-09 17:48 ` [PATCH v3 04/11] busybox.inc: improve reproducibility Juro Bystricky
@ 2017-08-09 17:48 ` Juro Bystricky
2017-08-09 17:48 ` [PATCH v3 06/11] cpio: provide cpio-replacement-native Juro Bystricky
` (6 subsequent siblings)
11 siblings, 0 replies; 28+ messages in thread
From: Juro Bystricky @ 2017-08-09 17:48 UTC (permalink / raw)
To: openembedded-core; +Cc: jurobystricky
Added a new task "reproducible_final_image_task".
If binary reproducibility is desired ($BUILD_REPRODUCIBLE_BINARIES" = "1"),
then recursivley modify mtimes of all files to a reproducible vale.
The value is obtained via REPRODUCIBLE_TIMESTAMP_ROOTFS.
This task is executed as the very last step in image creation, once all
the files in the image have been finalized.
[YOCTO#11176]
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
meta/classes/image.bbclass | 16 +++++++++++++++-
1 file changed, 15 insertions(+), 1 deletion(-)
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2c1dc81..92f4714 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -117,7 +117,7 @@ def rootfs_variables(d):
'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
- 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY']
+ 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED', 'IMGDEPLOYDIR', 'PACKAGE_EXCLUDE_COMPLEMENTARY', 'REPRODUCIBLE_TIMESTAMP_ROOTFS']
variables.extend(rootfs_command_variables(d))
variables.extend(variable_depends(d))
return " ".join(variables)
@@ -254,6 +254,7 @@ fakeroot python do_rootfs () {
progress_reporter.next_stage()
# generate rootfs
+ d.setVarFlag('REPRODUCIBLE_TIMESTAMP_ROOTFS', 'export', '1')
create_rootfs(d, progress_reporter=progress_reporter, logcatcher=logcatcher)
progress_reporter.finish()
@@ -266,6 +267,7 @@ addtask rootfs after do_prepare_recipe_sysroot
fakeroot python do_image () {
from oe.utils import execute_pre_post_process
+ d.setVarFlag('REPRODUCIBLE_TIMESTAMP_ROOTFS', 'export', '1')
pre_process_cmds = d.getVar("IMAGE_PREPROCESS_COMMAND")
execute_pre_post_process(d, pre_process_cmds)
@@ -650,3 +652,15 @@ create_merged_usr_symlinks_sdk() {
ROOTFS_PREPROCESS_COMMAND += "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', 'create_merged_usr_symlinks_rootfs; ', '',d)}"
POPULATE_SDK_PRE_TARGET_COMMAND += "${@bb.utils.contains('DISTRO_FEATURES', 'usrmerge', 'create_merged_usr_symlinks_sdk; ', '',d)}"
+
+reproducible_final_image_task () {
+ if [ "$BUILD_REPRODUCIBLE_BINARIES" = "1" ]; then
+ if [ "$REPRODUCIBLE_TIMESTAMP_ROOTFS" = "" ]; then
+ REPRODUCIBLE_TIMESTAMP_ROOTFS=`git log -1 --pretty=%ct`
+ fi
+ # Set mtime of all files to a reproducible value
+ bbnote "reproducible_final_image_task: mtime set to $REPRODUCIBLE_TIMESTAMP_ROOTFS"
+ find ${IMAGE_ROOTFS} -exec touch -h --date=@$REPRODUCIBLE_TIMESTAMP_ROOTFS {} \;
+ fi
+}
+IMAGE_PREPROCESS_COMMAND_append = " reproducible_final_image_task; "
--
2.7.4
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH v3 06/11] cpio: provide cpio-replacement-native
2017-08-09 17:48 [PATCH v3 00/11] Reproducible binaries Juro Bystricky
` (4 preceding siblings ...)
2017-08-09 17:48 ` [PATCH v3 05/11] image.bbclass: support binary reproducibility Juro Bystricky
@ 2017-08-09 17:48 ` Juro Bystricky
2017-08-09 17:48 ` [PATCH v3 07/11] image_types.bbclass: improve cpio image reproducibility Juro Bystricky
` (5 subsequent siblings)
11 siblings, 0 replies; 28+ messages in thread
From: Juro Bystricky @ 2017-08-09 17:48 UTC (permalink / raw)
To: openembedded-core; +Cc: jurobystricky
By default, bitbake uses host cpio, which can be quite old
and missing crucial newever features.
This patch allows to use the cpio-native instead, which is the
latest upstream version. To use the cpio-native instead of the
cpio from host, you need to specify:
IMAGE_DEPENDS_cpio_append = " cpio-replacement-native"
EXTRANATIVEPATH += "cpio-native"
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
meta/recipes-extended/cpio/cpio_v2.inc | 2 ++
1 file changed, 2 insertions(+)
diff --git a/meta/recipes-extended/cpio/cpio_v2.inc b/meta/recipes-extended/cpio/cpio_v2.inc
index 31adb71..b2a0d1b 100644
--- a/meta/recipes-extended/cpio/cpio_v2.inc
+++ b/meta/recipes-extended/cpio/cpio_v2.inc
@@ -6,6 +6,8 @@ SECTION = "base"
DEPENDS = "texinfo-native"
+PROVIDES_append_class-native = " cpio-replacement-native"
+
SRC_URI = "${GNU_MIRROR}/cpio/cpio-${PV}.tar.gz \
"
--
2.7.4
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH v3 07/11] image_types.bbclass: improve cpio image reproducibility
2017-08-09 17:48 [PATCH v3 00/11] Reproducible binaries Juro Bystricky
` (5 preceding siblings ...)
2017-08-09 17:48 ` [PATCH v3 06/11] cpio: provide cpio-replacement-native Juro Bystricky
@ 2017-08-09 17:48 ` Juro Bystricky
2017-08-09 18:06 ` Andre McCurdy
2017-08-09 17:48 ` [PATCH v3 08/11] python2.7: improve reproducibility Juro Bystricky
` (4 subsequent siblings)
11 siblings, 1 reply; 28+ messages in thread
From: Juro Bystricky @ 2017-08-09 17:48 UTC (permalink / raw)
To: openembedded-core; +Cc: jurobystricky
This patch helps to build cpio images that are binary reproducible.
The changes are as follows:
1. By default, cpio from the host is used, which can be quite old.
Hence we need to implement a way to use/call cpio-native, which supports
new features needed for binary reproducibility, notably the arguments
such as "--reproducible" and "--ignore-devno".
This can be achieved by specifying the following (in local.conf):
do_image_cpio[depends] += "cpio-replacement-native:do_populate_sysroot"
EXTRANATIVEPATH += "cpio-native"
2. Provide a way to pass custom arguments to cpio. This is done via
a new variable IMAGE_CMD_CPIO. For binary reproducible cpio archives
one needs to set (in local.conf)
IMAGE_CMD_CPIO = "cpio --ignore-devno --reproducible "
3. A symlink is created as part of the image. Here we make sure it gets the
timestamps (mtime) based on the timestamp of the symlink target, rather than
the timestamp corresponding to the build time.
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
meta/classes/image_types.bbclass | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
index e0368c7..50d0c07 100644
--- a/meta/classes/image_types.bbclass
+++ b/meta/classes/image_types.bbclass
@@ -122,9 +122,17 @@ IMAGE_CMD_TAR ?= "tar"
# ignore return code 1 "file changed as we read it" as other tasks(e.g. do_image_wic) may be hardlinking rootfs
IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]"
+# By default, cpio from the host is used, which can be quite old. If
+# you need special parameters (like --ignore-devno --reproducible) which are only
+# supported by GNU cpio upstream >= 2.12, then override that default:
+# IMAGE_CMD_CPIO = "cpio --ignore-devno"
+# do_image_cpio[depends] += "cpio-replacement-native:do_populate_sysroot"
+# EXTRANATIVEPATH += "cpio-native"
+
+IMAGE_CMD_CPIO ?= "cpio"
do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
IMAGE_CMD_cpio () {
- (cd ${IMAGE_ROOTFS} && find . | cpio -o -H newc >${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
+ (cd ${IMAGE_ROOTFS} && find . | ${IMAGE_CMD_CPIO} -o -H newc -v >${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
# We only need the /init symlink if we're building the real
# image. The -dbg image doesn't need it! By being clever
# about this we also avoid 'touch' below failing, as it
@@ -134,10 +142,12 @@ IMAGE_CMD_cpio () {
if [ ! -L ${IMAGE_ROOTFS}/init ] && [ ! -e ${IMAGE_ROOTFS}/init ]; then
if [ -L ${IMAGE_ROOTFS}/sbin/init ] || [ -e ${IMAGE_ROOTFS}/sbin/init ]; then
ln -sf /sbin/init ${WORKDIR}/cpio_append/init
+ # improve reproducibility: set the link mtime to be the same as the target
+ touch -h -r ${IMAGE_ROOTFS}/sbin/init ${WORKDIR}/cpio_append/init
else
touch ${WORKDIR}/cpio_append/init
fi
- (cd ${WORKDIR}/cpio_append && echo ./init | cpio -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
+ (cd ${WORKDIR}/cpio_append && echo ./init | ${IMAGE_CMD_CPIO} -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
fi
fi
}
--
2.7.4
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [PATCH v3 07/11] image_types.bbclass: improve cpio image reproducibility
2017-08-09 17:48 ` [PATCH v3 07/11] image_types.bbclass: improve cpio image reproducibility Juro Bystricky
@ 2017-08-09 18:06 ` Andre McCurdy
2017-08-10 19:35 ` Bystricky, Juro
0 siblings, 1 reply; 28+ messages in thread
From: Andre McCurdy @ 2017-08-09 18:06 UTC (permalink / raw)
To: Juro Bystricky; +Cc: Juro Bystricky, OE Core mailing list
On Wed, Aug 9, 2017 at 10:48 AM, Juro Bystricky
<juro.bystricky@intel.com> wrote:
> This patch helps to build cpio images that are binary reproducible.
> The changes are as follows:
>
> 1. By default, cpio from the host is used, which can be quite old.
> Hence we need to implement a way to use/call cpio-native, which supports
> new features needed for binary reproducibility, notably the arguments
> such as "--reproducible" and "--ignore-devno".
> This can be achieved by specifying the following (in local.conf):
Is it useful to keep the option using cpio from the host? Why not
always use cpio-native?
> do_image_cpio[depends] += "cpio-replacement-native:do_populate_sysroot"
> EXTRANATIVEPATH += "cpio-native"
>
> 2. Provide a way to pass custom arguments to cpio. This is done via
> a new variable IMAGE_CMD_CPIO. For binary reproducible cpio archives
> one needs to set (in local.conf)
>
> IMAGE_CMD_CPIO = "cpio --ignore-devno --reproducible "
>
> 3. A symlink is created as part of the image. Here we make sure it gets the
> timestamps (mtime) based on the timestamp of the symlink target, rather than
> the timestamp corresponding to the build time.
>
> Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
> ---
> meta/classes/image_types.bbclass | 14 ++++++++++++--
> 1 file changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/image_types.bbclass b/meta/classes/image_types.bbclass
> index e0368c7..50d0c07 100644
> --- a/meta/classes/image_types.bbclass
> +++ b/meta/classes/image_types.bbclass
> @@ -122,9 +122,17 @@ IMAGE_CMD_TAR ?= "tar"
> # ignore return code 1 "file changed as we read it" as other tasks(e.g. do_image_wic) may be hardlinking rootfs
> IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cf ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} . || [ $? -eq 1 ]"
>
> +# By default, cpio from the host is used, which can be quite old. If
> +# you need special parameters (like --ignore-devno --reproducible) which are only
> +# supported by GNU cpio upstream >= 2.12, then override that default:
> +# IMAGE_CMD_CPIO = "cpio --ignore-devno"
> +# do_image_cpio[depends] += "cpio-replacement-native:do_populate_sysroot"
> +# EXTRANATIVEPATH += "cpio-native"
> +
> +IMAGE_CMD_CPIO ?= "cpio"
> do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
> IMAGE_CMD_cpio () {
> - (cd ${IMAGE_ROOTFS} && find . | cpio -o -H newc >${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> + (cd ${IMAGE_ROOTFS} && find . | ${IMAGE_CMD_CPIO} -o -H newc -v >${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> # We only need the /init symlink if we're building the real
> # image. The -dbg image doesn't need it! By being clever
> # about this we also avoid 'touch' below failing, as it
> @@ -134,10 +142,12 @@ IMAGE_CMD_cpio () {
> if [ ! -L ${IMAGE_ROOTFS}/init ] && [ ! -e ${IMAGE_ROOTFS}/init ]; then
> if [ -L ${IMAGE_ROOTFS}/sbin/init ] || [ -e ${IMAGE_ROOTFS}/sbin/init ]; then
> ln -sf /sbin/init ${WORKDIR}/cpio_append/init
> + # improve reproducibility: set the link mtime to be the same as the target
> + touch -h -r ${IMAGE_ROOTFS}/sbin/init ${WORKDIR}/cpio_append/init
> else
> touch ${WORKDIR}/cpio_append/init
> fi
> - (cd ${WORKDIR}/cpio_append && echo ./init | cpio -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> + (cd ${WORKDIR}/cpio_append && echo ./init | ${IMAGE_CMD_CPIO} -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> fi
> fi
> }
> --
> 2.7.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v3 07/11] image_types.bbclass: improve cpio image reproducibility
2017-08-09 18:06 ` Andre McCurdy
@ 2017-08-10 19:35 ` Bystricky, Juro
2017-08-10 20:06 ` Andre McCurdy
0 siblings, 1 reply; 28+ messages in thread
From: Bystricky, Juro @ 2017-08-10 19:35 UTC (permalink / raw)
To: Andre McCurdy; +Cc: Juro Bystricky, OE Core mailing list
>
> Is it useful to keep the option using cpio from the host? Why not
> always use cpio-native?
>
I guess it is a judgement call which tools to build and which tools to use from the host. Not building tools which are guaranteed to be on the host saves build time. Also, the need for cpio-native with the reproducible options it is going to be fairly rare.
> > do_image_cpio[depends] += "cpio-replacement-
> native:do_populate_sysroot"
> > EXTRANATIVEPATH += "cpio-native"
> >
> > 2. Provide a way to pass custom arguments to cpio. This is done via
> > a new variable IMAGE_CMD_CPIO. For binary reproducible cpio archives
> > one needs to set (in local.conf)
> >
> > IMAGE_CMD_CPIO = "cpio --ignore-devno --reproducible "
> >
> > 3. A symlink is created as part of the image. Here we make sure it gets
> the
> > timestamps (mtime) based on the timestamp of the symlink target,
> rather than
> > the timestamp corresponding to the build time.
> >
> > Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
> > ---
> > meta/classes/image_types.bbclass | 14 ++++++++++++--
> > 1 file changed, 12 insertions(+), 2 deletions(-)
> >
> > diff --git a/meta/classes/image_types.bbclass
> b/meta/classes/image_types.bbclass
> > index e0368c7..50d0c07 100644
> > --- a/meta/classes/image_types.bbclass
> > +++ b/meta/classes/image_types.bbclass
> > @@ -122,9 +122,17 @@ IMAGE_CMD_TAR ?= "tar"
> > # ignore return code 1 "file changed as we read it" as other tasks(e.g.
> do_image_wic) may be hardlinking rootfs
> > IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cf
> ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} .
> || [ $? -eq 1 ]"
> >
> > +# By default, cpio from the host is used, which can be quite old. If
> > +# you need special parameters (like --ignore-devno --reproducible) which
> are only
> > +# supported by GNU cpio upstream >= 2.12, then override that default:
> > +# IMAGE_CMD_CPIO = "cpio --ignore-devno"
> > +# do_image_cpio[depends] += "cpio-replacement-
> native:do_populate_sysroot"
> > +# EXTRANATIVEPATH += "cpio-native"
> > +
> > +IMAGE_CMD_CPIO ?= "cpio"
> > do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
> > IMAGE_CMD_cpio () {
> > - (cd ${IMAGE_ROOTFS} && find . | cpio -o -H newc
> >${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> > + (cd ${IMAGE_ROOTFS} && find . | ${IMAGE_CMD_CPIO} -o -H newc -v
> >${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> > # We only need the /init symlink if we're building the real
> > # image. The -dbg image doesn't need it! By being clever
> > # about this we also avoid 'touch' below failing, as it
> > @@ -134,10 +142,12 @@ IMAGE_CMD_cpio () {
> > if [ ! -L ${IMAGE_ROOTFS}/init ] && [ ! -e
> ${IMAGE_ROOTFS}/init ]; then
> > if [ -L ${IMAGE_ROOTFS}/sbin/init ] || [ -e
> ${IMAGE_ROOTFS}/sbin/init ]; then
> > ln -sf /sbin/init
> ${WORKDIR}/cpio_append/init
> > + # improve reproducibility: set the link
> mtime to be the same as the target
> > + touch -h -r ${IMAGE_ROOTFS}/sbin/init
> ${WORKDIR}/cpio_append/init
> > else
> > touch ${WORKDIR}/cpio_append/init
> > fi
> > - (cd ${WORKDIR}/cpio_append && echo ./init | cpio
> -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> > + (cd ${WORKDIR}/cpio_append && echo ./init |
> ${IMAGE_CMD_CPIO} -oA -H newc -F
> ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> > fi
> > fi
> > }
> > --
> > 2.7.4
> >
> > --
> > _______________________________________________
> > Openembedded-core mailing list
> > Openembedded-core@lists.openembedded.org
> > http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v3 07/11] image_types.bbclass: improve cpio image reproducibility
2017-08-10 19:35 ` Bystricky, Juro
@ 2017-08-10 20:06 ` Andre McCurdy
2017-08-10 20:36 ` Bystricky, Juro
0 siblings, 1 reply; 28+ messages in thread
From: Andre McCurdy @ 2017-08-10 20:06 UTC (permalink / raw)
To: Bystricky, Juro; +Cc: Juro Bystricky, OE Core mailing list
On Thu, Aug 10, 2017 at 12:35 PM, Bystricky, Juro
<juro.bystricky@intel.com> wrote:
>>
>> Is it useful to keep the option using cpio from the host? Why not
>> always use cpio-native?
>
> I guess it is a judgement call which tools to build and which tools to use from the host. Not building tools which are guaranteed to be on the host saves build time.
I guess build time for cpio-native will be somewhere in the noise for
a typical build. Avoiding the extra complexity (code + test and QA) of
maintaining support for both cpio from the host and cpio-native would
seem to be far more of an issue. There's no guarantee that cpio is
available on the host (it currently needs to be installed manually if
building within the ubuntu:16.04 docker image, for example).
> Also, the need for cpio-native with the reproducible options it is going to be fairly rare.
I'm not sure I agree that wanting reproducible builds is a rare
requirement but if it is then isn't eliminating a rarely used code
path an argument *in favour of* switching to a single solution which
supports both cases?
>
>> > do_image_cpio[depends] += "cpio-replacement-
>> native:do_populate_sysroot"
>> > EXTRANATIVEPATH += "cpio-native"
>> >
>> > 2. Provide a way to pass custom arguments to cpio. This is done via
>> > a new variable IMAGE_CMD_CPIO. For binary reproducible cpio archives
>> > one needs to set (in local.conf)
>> >
>> > IMAGE_CMD_CPIO = "cpio --ignore-devno --reproducible "
>> >
>> > 3. A symlink is created as part of the image. Here we make sure it gets
>> the
>> > timestamps (mtime) based on the timestamp of the symlink target,
>> rather than
>> > the timestamp corresponding to the build time.
>> >
>> > Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
>> > ---
>> > meta/classes/image_types.bbclass | 14 ++++++++++++--
>> > 1 file changed, 12 insertions(+), 2 deletions(-)
>> >
>> > diff --git a/meta/classes/image_types.bbclass
>> b/meta/classes/image_types.bbclass
>> > index e0368c7..50d0c07 100644
>> > --- a/meta/classes/image_types.bbclass
>> > +++ b/meta/classes/image_types.bbclass
>> > @@ -122,9 +122,17 @@ IMAGE_CMD_TAR ?= "tar"
>> > # ignore return code 1 "file changed as we read it" as other tasks(e.g.
>> do_image_wic) may be hardlinking rootfs
>> > IMAGE_CMD_tar = "${IMAGE_CMD_TAR} -cf
>> ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.tar -C ${IMAGE_ROOTFS} .
>> || [ $? -eq 1 ]"
>> >
>> > +# By default, cpio from the host is used, which can be quite old. If
>> > +# you need special parameters (like --ignore-devno --reproducible) which
>> are only
>> > +# supported by GNU cpio upstream >= 2.12, then override that default:
>> > +# IMAGE_CMD_CPIO = "cpio --ignore-devno"
>> > +# do_image_cpio[depends] += "cpio-replacement-
>> native:do_populate_sysroot"
>> > +# EXTRANATIVEPATH += "cpio-native"
>> > +
>> > +IMAGE_CMD_CPIO ?= "cpio"
>> > do_image_cpio[cleandirs] += "${WORKDIR}/cpio_append"
>> > IMAGE_CMD_cpio () {
>> > - (cd ${IMAGE_ROOTFS} && find . | cpio -o -H newc
>> >${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
>> > + (cd ${IMAGE_ROOTFS} && find . | ${IMAGE_CMD_CPIO} -o -H newc -v
>> >${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
>> > # We only need the /init symlink if we're building the real
>> > # image. The -dbg image doesn't need it! By being clever
>> > # about this we also avoid 'touch' below failing, as it
>> > @@ -134,10 +142,12 @@ IMAGE_CMD_cpio () {
>> > if [ ! -L ${IMAGE_ROOTFS}/init ] && [ ! -e
>> ${IMAGE_ROOTFS}/init ]; then
>> > if [ -L ${IMAGE_ROOTFS}/sbin/init ] || [ -e
>> ${IMAGE_ROOTFS}/sbin/init ]; then
>> > ln -sf /sbin/init
>> ${WORKDIR}/cpio_append/init
>> > + # improve reproducibility: set the link
>> mtime to be the same as the target
>> > + touch -h -r ${IMAGE_ROOTFS}/sbin/init
>> ${WORKDIR}/cpio_append/init
>> > else
>> > touch ${WORKDIR}/cpio_append/init
>> > fi
>> > - (cd ${WORKDIR}/cpio_append && echo ./init | cpio
>> -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
>> > + (cd ${WORKDIR}/cpio_append && echo ./init |
>> ${IMAGE_CMD_CPIO} -oA -H newc -F
>> ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
>> > fi
>> > fi
>> > }
>> > --
>> > 2.7.4
>> >
>> > --
>> > _______________________________________________
>> > Openembedded-core mailing list
>> > Openembedded-core@lists.openembedded.org
>> > http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v3 07/11] image_types.bbclass: improve cpio image reproducibility
2017-08-10 20:06 ` Andre McCurdy
@ 2017-08-10 20:36 ` Bystricky, Juro
0 siblings, 0 replies; 28+ messages in thread
From: Bystricky, Juro @ 2017-08-10 20:36 UTC (permalink / raw)
To: Andre McCurdy; +Cc: Juro Bystricky, OE Core mailing list
> >> Is it useful to keep the option using cpio from the host? Why not
> >> always use cpio-native?
> >
> > I guess it is a judgement call which tools to build and which tools to
> use from the host. Not building tools which are guaranteed to be on the
> host saves build time.
>
> I guess build time for cpio-native will be somewhere in the noise for
> a typical build. Avoiding the extra complexity (code + test and QA) of
> maintaining support for both cpio from the host and cpio-native would
> seem to be far more of an issue. There's no guarantee that cpio is
> available on the host (it currently needs to be installed manually if
> building within the ubuntu:16.04 docker image, for example).
>
Looking at the HOSTTOOLS in poky/meta/conf/bitbake.conf I see "cpio" is listed among required tools. So at least for "poky" a missing cpio would/should generate an error.
> > Also, the need for cpio-native with the reproducible options it is going
> to be fairly rare.
>
> I'm not sure I agree that wanting reproducible builds is a rare
> requirement but if it is then isn't eliminating a rarely used code
> path an argument *in favour of* switching to a single solution which
> supports both cases?
Agreed. I don't have a problem with defaulting to cpio-native,
it would solve a few problems as well, but I think some kind of consensus is needed first.
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH v3 08/11] python2.7: improve reproducibility
2017-08-09 17:48 [PATCH v3 00/11] Reproducible binaries Juro Bystricky
` (6 preceding siblings ...)
2017-08-09 17:48 ` [PATCH v3 07/11] image_types.bbclass: improve cpio image reproducibility Juro Bystricky
@ 2017-08-09 17:48 ` Juro Bystricky
2017-08-17 19:39 ` Leonardo Sandoval
2017-08-09 17:48 ` [PATCH v3 09/11] python3: " Juro Bystricky
` (3 subsequent siblings)
11 siblings, 1 reply; 28+ messages in thread
From: Juro Bystricky @ 2017-08-09 17:48 UTC (permalink / raw)
To: openembedded-core; +Cc: jurobystricky
The compiled .pyc files contain time stamp corresponding to the compile time.
This prevents binary reproducibility. This patch allows to achieve binary
reproducibility by overriding the build time stamp by the value
exported via SOURCE_DATE_EPOCH.
Patch by Bernhard M. Wiedemann, backported from https://github.com/python/cpython/pull/296
[YOCTO#11241]
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
.../python/python-native_2.7.13.bb | 1 +
.../python/python/reproducible.patch | 34 ++++++++++++++++++++++
meta/recipes-devtools/python/python_2.7.13.bb | 1 +
3 files changed, 36 insertions(+)
create mode 100644 meta/recipes-devtools/python/python/reproducible.patch
diff --git a/meta/recipes-devtools/python/python-native_2.7.13.bb b/meta/recipes-devtools/python/python-native_2.7.13.bb
index 7edf153..a82b7bb 100644
--- a/meta/recipes-devtools/python/python-native_2.7.13.bb
+++ b/meta/recipes-devtools/python/python-native_2.7.13.bb
@@ -17,6 +17,7 @@ SRC_URI += "\
file://builddir.patch \
file://parallel-makeinst-create-bindir.patch \
file://revert_use_of_sysconfigdata.patch \
+ file://reproducible.patch \
"
S = "${WORKDIR}/Python-${PV}"
diff --git a/meta/recipes-devtools/python/python/reproducible.patch b/meta/recipes-devtools/python/python/reproducible.patch
new file mode 100644
index 0000000..1265179
--- /dev/null
+++ b/meta/recipes-devtools/python/python/reproducible.patch
@@ -0,0 +1,34 @@
+The compiled .pyc files contain time stamp corresponding to the compile time.
+This prevents binary reproducibility. This patch allows to achieve binary
+reproducibility by overriding the build time stamp by the value
+exported via SOURCE_DATE_EPOCH.
+
+Patch by Bernhard M. Wiedemann
+
+Upstream-Status: Backport
+
+Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
+
+Fri Feb 24 17:08:25 UTC 2017 - bwiedemann@suse.com
+
+- Add reproducible.patch to allow reproducible builds of various
+ python packages like python-amqp
+ Upstream: https://github.com/python/cpython/pull/296
+
+
+@@ -0,0 +1,15 @@
+Index: Python-2.7.13/Lib/py_compile.py
+===================================================================
+--- Python-2.7.13.orig/Lib/py_compile.py
++++ Python-2.7.13/Lib/py_compile.py
+@@ -108,6 +108,10 @@ def compile(file, cfile=None, dfile=None
+ timestamp = long(os.fstat(f.fileno()).st_mtime)
+ except AttributeError:
+ timestamp = long(os.stat(file).st_mtime)
++ sde = os.environ.get('SOURCE_DATE_EPOCH')
++ if sde and timestamp > int(sde):
++ timestamp = int(sde)
++ os.utime(file, (timestamp, timestamp))
+ codestring = f.read()
+ try:
+ codeobject = __builtin__.compile(codestring, dfile or file,'exec')
diff --git a/meta/recipes-devtools/python/python_2.7.13.bb b/meta/recipes-devtools/python/python_2.7.13.bb
index 4ef9952..96c3ab2 100644
--- a/meta/recipes-devtools/python/python_2.7.13.bb
+++ b/meta/recipes-devtools/python/python_2.7.13.bb
@@ -27,6 +27,7 @@ SRC_URI += "\
file://use_sysroot_ncurses_instead_of_host.patch \
file://add-CROSSPYTHONPATH-for-PYTHON_FOR_BUILD.patch \
file://Don-t-use-getentropy-on-Linux.patch \
+ file://reproducible.patch \
"
S = "${WORKDIR}/Python-${PV}"
--
2.7.4
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [PATCH v3 08/11] python2.7: improve reproducibility
2017-08-09 17:48 ` [PATCH v3 08/11] python2.7: improve reproducibility Juro Bystricky
@ 2017-08-17 19:39 ` Leonardo Sandoval
2017-08-19 16:39 ` Bystricky, Juro
0 siblings, 1 reply; 28+ messages in thread
From: Leonardo Sandoval @ 2017-08-17 19:39 UTC (permalink / raw)
To: Juro Bystricky; +Cc: jurobystricky, openembedded-core
On Wed, 2017-08-09 at 10:48 -0700, Juro Bystricky wrote:
> The compiled .pyc files contain time stamp corresponding to the compile time.
> This prevents binary reproducibility. This patch allows to achieve binary
> reproducibility by overriding the build time stamp by the value
> exported via SOURCE_DATE_EPOCH.
>
> Patch by Bernhard M. Wiedemann, backported from https://github.com/python/cpython/pull/296
>
> [YOCTO#11241]
>
> Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
> ---
> .../python/python-native_2.7.13.bb | 1 +
> .../python/python/reproducible.patch | 34 ++++++++++++++++++++++
> meta/recipes-devtools/python/python_2.7.13.bb | 1 +
> 3 files changed, 36 insertions(+)
> create mode 100644 meta/recipes-devtools/python/python/reproducible.patch
>
> diff --git a/meta/recipes-devtools/python/python-native_2.7.13.bb b/meta/recipes-devtools/python/python-native_2.7.13.bb
> index 7edf153..a82b7bb 100644
> --- a/meta/recipes-devtools/python/python-native_2.7.13.bb
> +++ b/meta/recipes-devtools/python/python-native_2.7.13.bb
> @@ -17,6 +17,7 @@ SRC_URI += "\
> file://builddir.patch \
> file://parallel-makeinst-create-bindir.patch \
> file://revert_use_of_sysconfigdata.patch \
> + file://reproducible.patch \
Not really important but it would be better to rename the patch to
something more specific
('repoducible-build-time-override-by-source-date-epoch.path'), allowing
future patches in this area. Besides a better name, it helps to quickly
identify the patch purpose without looking at the code itself.
> "
>
> S = "${WORKDIR}/Python-${PV}"
> diff --git a/meta/recipes-devtools/python/python/reproducible.patch b/meta/recipes-devtools/python/python/reproducible.patch
> new file mode 100644
> index 0000000..1265179
> --- /dev/null
> +++ b/meta/recipes-devtools/python/python/reproducible.patch
> @@ -0,0 +1,34 @@
> +The compiled .pyc files contain time stamp corresponding to the compile time.
> +This prevents binary reproducibility. This patch allows to achieve binary
> +reproducibility by overriding the build time stamp by the value
> +exported via SOURCE_DATE_EPOCH.
> +
> +Patch by Bernhard M. Wiedemann
> +
> +Upstream-Status: Backport
> +
> +Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
> +
> +Fri Feb 24 17:08:25 UTC 2017 - bwiedemann@suse.com
> +
> +- Add reproducible.patch to allow reproducible builds of various
> + python packages like python-amqp
> + Upstream: https://github.com/python/cpython/pull/296
> +
> +
> +@@ -0,0 +1,15 @@
> +Index: Python-2.7.13/Lib/py_compile.py
> +===================================================================
> +--- Python-2.7.13.orig/Lib/py_compile.py
> ++++ Python-2.7.13/Lib/py_compile.py
> +@@ -108,6 +108,10 @@ def compile(file, cfile=None, dfile=None
> + timestamp = long(os.fstat(f.fileno()).st_mtime)
> + except AttributeError:
> + timestamp = long(os.stat(file).st_mtime)
> ++ sde = os.environ.get('SOURCE_DATE_EPOCH')
> ++ if sde and timestamp > int(sde):
> ++ timestamp = int(sde)
> ++ os.utime(file, (timestamp, timestamp))
> + codestring = f.read()
> + try:
> + codeobject = __builtin__.compile(codestring, dfile or file,'exec')
> diff --git a/meta/recipes-devtools/python/python_2.7.13.bb b/meta/recipes-devtools/python/python_2.7.13.bb
> index 4ef9952..96c3ab2 100644
> --- a/meta/recipes-devtools/python/python_2.7.13.bb
> +++ b/meta/recipes-devtools/python/python_2.7.13.bb
> @@ -27,6 +27,7 @@ SRC_URI += "\
> file://use_sysroot_ncurses_instead_of_host.patch \
> file://add-CROSSPYTHONPATH-for-PYTHON_FOR_BUILD.patch \
> file://Don-t-use-getentropy-on-Linux.patch \
> + file://reproducible.patch \
> "
>
> S = "${WORKDIR}/Python-${PV}"
> --
> 2.7.4
>
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH v3 09/11] python3: improve reproducibility
2017-08-09 17:48 [PATCH v3 00/11] Reproducible binaries Juro Bystricky
` (7 preceding siblings ...)
2017-08-09 17:48 ` [PATCH v3 08/11] python2.7: improve reproducibility Juro Bystricky
@ 2017-08-09 17:48 ` Juro Bystricky
2017-08-09 17:48 ` [PATCH v3 10/11] kernel.bbclass: " Juro Bystricky
` (2 subsequent siblings)
11 siblings, 0 replies; 28+ messages in thread
From: Juro Bystricky @ 2017-08-09 17:48 UTC (permalink / raw)
To: openembedded-core; +Cc: jurobystricky
The compiled .pyc files contain time stamp corresponding to the compile time.
This prevents binary reproducibility. This patch allows to achieve binary
reproducibility by overriding the build time stamp by the value
exported via SOURCE_DATE_EPOCH.
Patch by Bernhard M. Wiedemann.
[YOCTO#11241]
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
.../python/python3-native_3.5.3.bb | 1 +
.../support_SOURCE_DATE_EPOCH_in_py_compile.patch | 97 ++++++++++++++++++++++
meta/recipes-devtools/python/python3_3.5.3.bb | 1 +
3 files changed, 99 insertions(+)
create mode 100644 meta/recipes-devtools/python/python3/support_SOURCE_DATE_EPOCH_in_py_compile.patch
diff --git a/meta/recipes-devtools/python/python3-native_3.5.3.bb b/meta/recipes-devtools/python/python3-native_3.5.3.bb
index 250697f..3467d29 100644
--- a/meta/recipes-devtools/python/python3-native_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3-native_3.5.3.bb
@@ -24,6 +24,7 @@ ${DISTRO_SRC_URI} \
file://sysconfig.py-add-_PYTHON_PROJECT_SRC.patch \
file://setup.py-check-cross_compiling-when-get-FLAGS.patch \
file://0001-Do-not-use-the-shell-version-of-python-config-that-w.patch \
+file://support_SOURCE_DATE_EPOCH_in_py_compile.patch \
"
SRC_URI[md5sum] = "57d1f8bfbabf4f2500273fb0706e6f21"
diff --git a/meta/recipes-devtools/python/python3/support_SOURCE_DATE_EPOCH_in_py_compile.patch b/meta/recipes-devtools/python/python3/support_SOURCE_DATE_EPOCH_in_py_compile.patch
new file mode 100644
index 0000000..32ecab9
--- /dev/null
+++ b/meta/recipes-devtools/python/python3/support_SOURCE_DATE_EPOCH_in_py_compile.patch
@@ -0,0 +1,97 @@
+The compiled .pyc files contain time stamp corresponding to the compile time.
+This prevents binary reproducibility. This patch allows to achieve binary
+reproducibility by overriding the build time stamp by the value
+exported via SOURCE_DATE_EPOCH.
+
+Upstream-Status: Backport
+
+Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
+
+
+From aeab488630fdb1b56a8d0b0c13fa88706b2afe9b Mon Sep 17 00:00:00 2001
+From: "Bernhard M. Wiedemann" <bwiedemann@suse.de>
+Date: Sat, 25 Feb 2017 06:42:28 +0100
+Subject: [PATCH] bpo-29708: support SOURCE_DATE_EPOCH env var in py_compile
+
+to allow for reproducible builds of python packages
+
+See https://reproducible-builds.org/ for why this is good
+and https://reproducible-builds.org/specs/source-date-epoch/
+for the definition of this variable.
+
+Background:
+In some distributions like openSUSE, binary rpms contain precompiled .pyc files.
+
+And packages like amqp or twisted dynamically generate .py files at build time
+so those have the current time and that timestamp gets embedded
+into the .pyc file header.
+When we then adapt file timestamps in rpms to be constant,
+the timestamp in the .pyc header will no more match
+the .py timestamp in the filesystem.
+The software will still work, but it will not use the .pyc file as it should.
+---
+ Doc/library/py_compile.rst | 4 ++++
+ Lib/py_compile.py | 4 ++++
+ Lib/test/test_py_compile.py | 19 +++++++++++++++++++
+ 3 files changed, 27 insertions(+)
+
+diff --git a/Doc/library/py_compile.rst b/Doc/library/py_compile.rst
+index 0af8fb1..841f3e8 100644
+--- a/Doc/library/py_compile.rst
++++ b/Doc/library/py_compile.rst
+@@ -53,6 +53,10 @@ byte-code cache files in the directory containing the source code.
+ :func:`compile` function. The default of ``-1`` selects the optimization
+ level of the current interpreter.
+
++ If the SOURCE_DATE_EPOCH environment variable is set, the .py file mtime
++ and timestamp entry in .pyc file header, will be limited to this value.
++ See https://reproducible-builds.org/specs/source-date-epoch/ for more info.
++
+ .. versionchanged:: 3.2
+ Changed default value of *cfile* to be :PEP:`3147`-compliant. Previous
+ default was *file* + ``'c'`` (``'o'`` if optimization was enabled).
+diff --git a/Lib/py_compile.py b/Lib/py_compile.py
+index 11c5b50..62dcdc7 100644
+--- a/Lib/py_compile.py
++++ b/Lib/py_compile.py
+@@ -137,6 +137,10 @@ def compile(file, cfile=None, dfile=None, doraise=False, optimize=-1):
+ except FileExistsError:
+ pass
+ source_stats = loader.path_stats(file)
++ sde = os.environ.get('SOURCE_DATE_EPOCH')
++ if sde and source_stats['mtime'] > int(sde):
++ source_stats['mtime'] = int(sde)
++ os.utime(file, (source_stats['mtime'], source_stats['mtime']))
+ bytecode = importlib._bootstrap_external._code_to_bytecode(
+ code, source_stats['mtime'], source_stats['size'])
+ mode = importlib._bootstrap_external._calc_mode(file)
+diff --git a/Lib/test/test_py_compile.py b/Lib/test/test_py_compile.py
+index 4a6caa5..3d09963 100644
+--- a/Lib/test/test_py_compile.py
++++ b/Lib/test/test_py_compile.py
+@@ -98,6 +98,25 @@ def test_bad_coding(self):
+ self.assertFalse(os.path.exists(
+ importlib.util.cache_from_source(bad_coding)))
+
++ def test_source_date_epoch(self):
++ testtime = 123456789
++ orig_sde = os.getenv("SOURCE_DATE_EPOCH")
++ os.environ["SOURCE_DATE_EPOCH"] = str(testtime)
++ py_compile.compile(self.source_path, self.pyc_path)
++ if orig_sde:
++ os.environ["SOURCE_DATE_EPOCH"] = orig_sde
++ else:
++ del os.environ["SOURCE_DATE_EPOCH"]
++ self.assertTrue(os.path.exists(self.pyc_path))
++ self.assertFalse(os.path.exists(self.cache_path))
++ statinfo = os.stat(self.source_path)
++ self.assertEqual(statinfo.st_mtime, testtime)
++ f = open(self.pyc_path, "rb")
++ f.read(4)
++ timebytes = f.read(4) # read timestamp from pyc header
++ f.close()
++ self.assertEqual(timebytes, (testtime).to_bytes(4, 'little'))
++
+ @unittest.skipIf(sys.flags.optimize > 0, 'test does not work with -O')
+ def test_double_dot_no_clobber(self):
+ # http://bugs.python.org/issue22966
diff --git a/meta/recipes-devtools/python/python3_3.5.3.bb b/meta/recipes-devtools/python/python3_3.5.3.bb
index 7419c71..0c32b85 100644
--- a/meta/recipes-devtools/python/python3_3.5.3.bb
+++ b/meta/recipes-devtools/python/python3_3.5.3.bb
@@ -16,6 +16,7 @@ file://130-readline-setup.patch \
file://150-fix-setupterm.patch \
file://0001-h2py-Fix-issue-13032-where-it-fails-with-UnicodeDeco.patch \
file://tweak-MULTIARCH-for-powerpc-linux-gnuspe.patch \
+file://support_SOURCE_DATE_EPOCH_in_py_compile.patch \
${DISTRO_SRC_URI} \
"
--
2.7.4
^ permalink raw reply related [flat|nested] 28+ messages in thread* [PATCH v3 10/11] kernel.bbclass: improve reproducibility
2017-08-09 17:48 [PATCH v3 00/11] Reproducible binaries Juro Bystricky
` (8 preceding siblings ...)
2017-08-09 17:48 ` [PATCH v3 09/11] python3: " Juro Bystricky
@ 2017-08-09 17:48 ` Juro Bystricky
2017-08-17 16:14 ` Christopher Larson
2017-08-09 17:48 ` [PATCH v3 11/11] poky-reproducible.conf: Initial version Juro Bystricky
2017-08-18 18:11 ` [PATCH v3 00/11] Reproducible binaries Martin Jansa
11 siblings, 1 reply; 28+ messages in thread
From: Juro Bystricky @ 2017-08-09 17:48 UTC (permalink / raw)
To: openembedded-core; +Cc: jurobystricky
Several tweaks to improve reproducibility:
1. If BUILD_REPRODUCIBLE_BINARIES == 1, set KBUILD_BUILD_TIMESTAMP
to a reproducible value. This is either a non-zero SOURCE_DATE_EPOCH, or the
value obtained from top entry of GIT repo, or (if there is no GIT repo)
fallback to REPRODUCIBLE_TIMESTAMP_ROOTFS as the last resort.
Also export KCONFIG_NOTIMESTAMP=1.
2. When compressing vmlinux.gz, use gzip "-n" option
3. Kernel and kernel modules contain hard coded paths referencing the host
build system. This is usually because the source code contains __FILE__
at some place. This prevents binary reproducibility. However, some compilers
allow remapping of the __FILE__ value. If we detect the compiler is capable
of doing this, we replace the source path $(S) part of __FILE__ by a string "/kernel-source".
For example:
/data/master/build/tmp/work-shared/qemux86/kernel-source/drivers/media/v4l2-core/videobuf2-core.c
will be replaced by a reproducible value:
/kernel-source/drivers/media/v4l2-core/videobuf2-core.c.
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
meta/classes/kernel.bbclass | 39 ++++++++++++++++++++++++++++++++++++---
1 file changed, 36 insertions(+), 3 deletions(-)
diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
index ce2cab6..2a76554 100644
--- a/meta/classes/kernel.bbclass
+++ b/meta/classes/kernel.bbclass
@@ -255,8 +255,39 @@ python do_devshell_prepend () {
addtask bundle_initramfs after do_install before do_deploy
+get_cc_option () {
+ # Check if KERNEL_CC supports the option "file-prefix-map".
+ # This option allows us to build images with __FILE__ values that do not
+ # contain the host build path.
+ cc_option_supported=`${KERNEL_CC} -Q --help=joined | grep ffile-prefix-map`
+ cc_extra=""
+ if [ $cc_option_supported = "-ffile-prefix-map=<old=new>" ]; then
+ cc_extra=-ffile-prefix-map=${S}=/kernel-source/
+ fi
+ echo $cc_extra
+}
+
kernel_do_compile() {
unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE
+ if [ "$BUILD_REPRODUCIBLE_BINARIES" = "1" ]; then
+ # kernel sources do not use do_unpack, so SOURCE_DATE_EPOCH may not
+ # be set....
+ if [ "$SOURCE_DATE_EPOCH" = "0" ]; then
+ olddir=`pwd`
+ cd ${S}
+ SOURCE_DATE_EPOCH=`git log -1 --pretty=%ct`
+ # git repo not guaranteed, so fall back to REPRODUCIBLE_TIMESTAMP_ROOTFS
+ if [ $? -ne 0 ]; then
+ SOURCE_DATE_EPOCH=${REPRODUCIBLE_TIMESTAMP_ROOTFS}
+ fi
+ cd $olddir
+ fi
+
+ ts=`LC_ALL=C date -d @$SOURCE_DATE_EPOCH`
+ export KBUILD_BUILD_TIMESTAMP="$ts"
+ export KCONFIG_NOTIMESTAMP=1
+ bbnote "KBUILD_BUILD_TIMESTAMP: $ts"
+ fi
# The $use_alternate_initrd is only set from
# do_bundle_initramfs() This variable is specifically for the
# case where we are making a second pass at the kernel
@@ -270,20 +301,22 @@ kernel_do_compile() {
copy_initramfs
use_alternate_initrd=CONFIG_INITRAMFS_SOURCE=${B}/usr/${INITRAMFS_IMAGE_NAME}.cpio
fi
+ cc_extra=$(get_cc_option)
for typeformake in ${KERNEL_IMAGETYPE_FOR_MAKE} ; do
- oe_runmake ${typeformake} CC="${KERNEL_CC}" LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS} $use_alternate_initrd
+ oe_runmake ${typeformake} CC="${KERNEL_CC} $cc_extra " LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS} $use_alternate_initrd
done
# vmlinux.gz is not built by kernel
if (echo "${KERNEL_IMAGETYPES}" | grep -wq "vmlinux\.gz"); then
mkdir -p "${KERNEL_OUTPUT_DIR}"
- gzip -9c < ${B}/vmlinux > "${KERNEL_OUTPUT_DIR}/vmlinux.gz"
+ gzip -9cn < ${B}/vmlinux > "${KERNEL_OUTPUT_DIR}/vmlinux.gz"
fi
}
do_compile_kernelmodules() {
unset CFLAGS CPPFLAGS CXXFLAGS LDFLAGS MACHINE
if (grep -q -i -e '^CONFIG_MODULES=y$' ${B}/.config); then
- oe_runmake -C ${B} ${PARALLEL_MAKE} modules CC="${KERNEL_CC}" LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS}
+ cc_extra=$(get_cc_option)
+ oe_runmake -C ${B} ${PARALLEL_MAKE} modules CC="${KERNEL_CC} $cc_extra " LD="${KERNEL_LD}" ${KERNEL_EXTRA_ARGS}
# Module.symvers gets updated during the
# building of the kernel modules. We need to
--
2.7.4
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [PATCH v3 10/11] kernel.bbclass: improve reproducibility
2017-08-09 17:48 ` [PATCH v3 10/11] kernel.bbclass: " Juro Bystricky
@ 2017-08-17 16:14 ` Christopher Larson
2017-08-17 18:18 ` Bystricky, Juro
0 siblings, 1 reply; 28+ messages in thread
From: Christopher Larson @ 2017-08-17 16:14 UTC (permalink / raw)
To: Juro Bystricky
Cc: Juro Bystricky, Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 2860 bytes --]
On Wed, Aug 9, 2017 at 10:48 AM, Juro Bystricky <juro.bystricky@intel.com>
wrote:
> Several tweaks to improve reproducibility:
>
> 1. If BUILD_REPRODUCIBLE_BINARIES == 1, set KBUILD_BUILD_TIMESTAMP
> to a reproducible value. This is either a non-zero SOURCE_DATE_EPOCH, or
> the
> value obtained from top entry of GIT repo, or (if there is no GIT repo)
> fallback to REPRODUCIBLE_TIMESTAMP_ROOTFS as the last resort.
> Also export KCONFIG_NOTIMESTAMP=1.
>
> 2. When compressing vmlinux.gz, use gzip "-n" option
>
> 3. Kernel and kernel modules contain hard coded paths referencing the host
> build system. This is usually because the source code contains __FILE__
> at some place. This prevents binary reproducibility. However, some
> compilers
> allow remapping of the __FILE__ value. If we detect the compiler is capable
> of doing this, we replace the source path $(S) part of __FILE__ by a
> string "/kernel-source".
> For example:
>
> /data/master/build/tmp/work-shared/qemux86/kernel-source/
> drivers/media/v4l2-core/videobuf2-core.c
>
> will be replaced by a reproducible value:
>
> /kernel-source/drivers/media/v4l2-core/videobuf2-core.c.
>
> Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
> ---
> meta/classes/kernel.bbclass | 39 ++++++++++++++++++++++++++++++++++++---
> 1 file changed, 36 insertions(+), 3 deletions(-)
>
> diff --git a/meta/classes/kernel.bbclass b/meta/classes/kernel.bbclass
> index ce2cab6..2a76554 100644
> --- a/meta/classes/kernel.bbclass
> +++ b/meta/classes/kernel.bbclass
> @@ -255,8 +255,39 @@ python do_devshell_prepend () {
>
> addtask bundle_initramfs after do_install before do_deploy
>
> +get_cc_option () {
> + # Check if KERNEL_CC supports the option "file-prefix-map".
> + # This option allows us to build images with __FILE__
> values that do not
> + # contain the host build path.
> + cc_option_supported=`${KERNEL_CC} -Q --help=joined | grep
> ffile-prefix-map`
>
This fails the do_compile task if ffile-prefix-map isn’t found in the help
output, resulting in build failures with an external toolchain, for
example. This function doesn’t work the way you intended at all. It also
fails to quote things properly, uses the old `` syntax rather than $(), and
relies on grep output rather than grep exit codes.
At a minimum we need a “|| true” or “|| :” on the end of the
cc_option_supported= line. I don’t see why this doesn’t just do this,
instead, though:
if ${KERNEL_CC} -Q —help=joined | grep -q
“\-ffile-prefix-map=<old=new>”; then
echo “\-ffile-prefix-map=${S}=/kernel-source/“
fi
--
Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Senior Software Engineer, Mentor Graphics
[-- Attachment #2: Type: text/html, Size: 3626 bytes --]
^ permalink raw reply [flat|nested] 28+ messages in thread
* [PATCH v3 11/11] poky-reproducible.conf: Initial version
2017-08-09 17:48 [PATCH v3 00/11] Reproducible binaries Juro Bystricky
` (9 preceding siblings ...)
2017-08-09 17:48 ` [PATCH v3 10/11] kernel.bbclass: " Juro Bystricky
@ 2017-08-09 17:48 ` Juro Bystricky
2017-08-09 19:09 ` Andre McCurdy
2017-08-18 18:11 ` [PATCH v3 00/11] Reproducible binaries Martin Jansa
11 siblings, 1 reply; 28+ messages in thread
From: Juro Bystricky @ 2017-08-09 17:48 UTC (permalink / raw)
To: openembedded-core; +Cc: jurobystricky
Simplify building reproducible images by using
DISTRO="poky-reproducible"
Sets some variables to reasonable values so users do not
have to set them in local.conf.
Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
---
meta-poky/conf/distro/include/reproducible-group | 50 +++++++++++++++++++++++
meta-poky/conf/distro/include/reproducible-passwd | 25 ++++++++++++
meta-poky/conf/distro/poky-reproducible.conf | 38 +++++++++++++++++
3 files changed, 113 insertions(+)
create mode 100644 meta-poky/conf/distro/include/reproducible-group
create mode 100644 meta-poky/conf/distro/include/reproducible-passwd
create mode 100644 meta-poky/conf/distro/poky-reproducible.conf
diff --git a/meta-poky/conf/distro/include/reproducible-group b/meta-poky/conf/distro/include/reproducible-group
new file mode 100644
index 0000000..4213d4e
--- /dev/null
+++ b/meta-poky/conf/distro/include/reproducible-group
@@ -0,0 +1,50 @@
+root:x:0:
+daemon:x:1:
+bin:x:2:
+sys:x:3:
+adm:x:4:
+tty:x:5:
+disk:x:6:
+lp:x:7:
+mail:x:8:
+news:x:9:
+uucp:x:10:
+man:x:12:
+proxy:x:13:
+kmem:x:15:
+input:x:19:
+dialout:x:20:
+fax:x:21:
+voice:x:22:
+cdrom:x:24:
+floppy:x:25:
+tape:x:26:
+sudo:x:27:
+audio:x:29:pulse
+dip:x:30:
+www-data:x:33:
+backup:x:34:
+operator:x:37:
+list:x:38:
+irc:x:39:
+src:x:40:
+gnats:x:41:
+shadow:x:42:
+utmp:x:43:
+video:x:44:
+sasl:x:45:
+plugdev:x:46:
+staff:x:50:
+games:x:60:
+shutdown:x:70:
+users:x:100:
+crontab:x:993:
+sshd:x:994:
+avahi:x:995:
+rpcuser:x:996:
+rpc:x:997:
+messagebus:x:998:
+netdev:x:999:
+tracing:x:1000:
+pulse:x:1001:pulse
+nogroup:x:65534:
diff --git a/meta-poky/conf/distro/include/reproducible-passwd b/meta-poky/conf/distro/include/reproducible-passwd
new file mode 100644
index 0000000..876195e
--- /dev/null
+++ b/meta-poky/conf/distro/include/reproducible-passwd
@@ -0,0 +1,25 @@
+root:x:0:0:root:/home/root:/bin/sh
+daemon:x:1:1:daemon:/usr/sbin:/bin/sh
+bin:x:2:2:bin:/bin:/bin/sh
+sys:x:3:3:sys:/dev:/bin/sh
+sync:x:4:65534:sync:/bin:/bin/sync
+games:x:5:60:games:/usr/games:/bin/sh
+man:x:6:12:man:/var/cache/man:/bin/sh
+lp:x:7:7:lp:/var/spool/lpd:/bin/sh
+mail:x:8:8:mail:/var/mail:/bin/sh
+news:x:9:9:news:/var/spool/news:/bin/sh
+uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh
+proxy:x:13:13:proxy:/bin:/bin/sh
+www-data:x:33:33:www-data:/var/www:/bin/sh
+backup:x:34:34:backup:/var/backups:/bin/sh
+list:x:38:38:Mailing List Manager:/var/list:/bin/sh
+irc:x:39:39:ircd:/var/run/ircd:/bin/sh
+gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/bin/sh
+pulse:x:993:1001::/var/run/pulse:/bin/false
+distcc:x:994:65534::/dev/null:/bin/sh
+sshd:x:995:994::/var/run/sshd:/bin/false
+avahi:x:996:995::/var/run/avahi-daemon:/bin/false
+rpcuser:x:997:996::/var/lib/nfs:/bin/false
+rpc:x:998:997::/:/bin/false
+messagebus:x:999:998::/var/lib/dbus:/bin/false
+nobody:x:65534:65534:nobody:/nonexistent:/bin/sh
diff --git a/meta-poky/conf/distro/poky-reproducible.conf b/meta-poky/conf/distro/poky-reproducible.conf
new file mode 100644
index 0000000..c94f673
--- /dev/null
+++ b/meta-poky/conf/distro/poky-reproducible.conf
@@ -0,0 +1,38 @@
+require conf/distro/poky.conf
+DISTRO = "poky-reproducible"
+
+BUILD_REPRODUCIBLE_BINARIES = "1"
+REPRODUCIBLE_TIMESTAMP_ROOTFS ?= "1483228802"
+LDCONFIGDEPEND = ""
+do_image_cpio[depends] += "cpio-replacement-native:do_populate_sysroot"
+EXTRANATIVEPATH += "cpio-native"
+IMAGE_CMD_CPIO = "cpio --ignore-devno --reproducible "
+
+IMAGE_CMD_TAR = "tar -v --sort=name "
+
+PACKAGE_CLASSES ="package_deb"
+
+# For reproducibility, we need to consistently assign the UID/GID values.
+# Use the static uid and gid mechanism from OE-core for that:
+# http://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-classes-useradd
+#
+# Dynamically assigned IDs are detected and lead to an error during
+# the build.
+#
+# Developers who need to add new entries should add their own mapping
+# file to USERADD_UID_TABLES and/or USERADD_GID_TABLES, either in a
+# derived distro config or in their local.conf.
+#
+# It is also possible to disable the mechanism by modifying USERADD_ERROR_DYNAMIC:
+# "warn" merely prints a warning, empty value silently allows dynamic
+# ID allocation.
+#
+# The actual files for UID/GID values come from core-image-minimal-sdk
+# /etc/group
+# /etc/passwd
+
+USERADDEXTENSION = "useradd-staticids"
+USERADD_ERROR_DYNAMIC ??= "error"
+USERADD_UID_TABLES += "conf/distro/include/reproducible-passwd"
+USERADD_GID_TABLES += "conf/distro/include/reproducible-group"
+
--
2.7.4
^ permalink raw reply related [flat|nested] 28+ messages in thread* Re: [PATCH v3 11/11] poky-reproducible.conf: Initial version
2017-08-09 17:48 ` [PATCH v3 11/11] poky-reproducible.conf: Initial version Juro Bystricky
@ 2017-08-09 19:09 ` Andre McCurdy
0 siblings, 0 replies; 28+ messages in thread
From: Andre McCurdy @ 2017-08-09 19:09 UTC (permalink / raw)
To: Juro Bystricky; +Cc: Juro Bystricky, OE Core mailing list
On Wed, Aug 9, 2017 at 10:48 AM, Juro Bystricky
<juro.bystricky@intel.com> wrote:
> Simplify building reproducible images by using
>
> DISTRO="poky-reproducible"
>
> Sets some variables to reasonable values so users do not
> have to set them in local.conf.
>
> Signed-off-by: Juro Bystricky <juro.bystricky@intel.com>
> ---
> meta-poky/conf/distro/include/reproducible-group | 50 +++++++++++++++++++++++
> meta-poky/conf/distro/include/reproducible-passwd | 25 ++++++++++++
> meta-poky/conf/distro/poky-reproducible.conf | 38 +++++++++++++++++
> 3 files changed, 113 insertions(+)
> create mode 100644 meta-poky/conf/distro/include/reproducible-group
> create mode 100644 meta-poky/conf/distro/include/reproducible-passwd
> create mode 100644 meta-poky/conf/distro/poky-reproducible.conf
>
> diff --git a/meta-poky/conf/distro/include/reproducible-group b/meta-poky/conf/distro/include/reproducible-group
> new file mode 100644
> index 0000000..4213d4e
> --- /dev/null
> +++ b/meta-poky/conf/distro/include/reproducible-group
> @@ -0,0 +1,50 @@
> +root:x:0:
> +daemon:x:1:
> +bin:x:2:
> +sys:x:3:
> +adm:x:4:
> +tty:x:5:
> +disk:x:6:
> +lp:x:7:
> +mail:x:8:
> +news:x:9:
> +uucp:x:10:
> +man:x:12:
> +proxy:x:13:
> +kmem:x:15:
> +input:x:19:
> +dialout:x:20:
> +fax:x:21:
> +voice:x:22:
> +cdrom:x:24:
> +floppy:x:25:
> +tape:x:26:
> +sudo:x:27:
> +audio:x:29:pulse
> +dip:x:30:
> +www-data:x:33:
> +backup:x:34:
> +operator:x:37:
> +list:x:38:
> +irc:x:39:
> +src:x:40:
> +gnats:x:41:
> +shadow:x:42:
> +utmp:x:43:
> +video:x:44:
> +sasl:x:45:
> +plugdev:x:46:
> +staff:x:50:
> +games:x:60:
> +shutdown:x:70:
> +users:x:100:
> +crontab:x:993:
> +sshd:x:994:
> +avahi:x:995:
> +rpcuser:x:996:
> +rpc:x:997:
> +messagebus:x:998:
> +netdev:x:999:
> +tracing:x:1000:
> +pulse:x:1001:pulse
> +nogroup:x:65534:
> diff --git a/meta-poky/conf/distro/include/reproducible-passwd b/meta-poky/conf/distro/include/reproducible-passwd
> new file mode 100644
> index 0000000..876195e
> --- /dev/null
> +++ b/meta-poky/conf/distro/include/reproducible-passwd
> @@ -0,0 +1,25 @@
> +root:x:0:0:root:/home/root:/bin/sh
> +daemon:x:1:1:daemon:/usr/sbin:/bin/sh
> +bin:x:2:2:bin:/bin:/bin/sh
> +sys:x:3:3:sys:/dev:/bin/sh
> +sync:x:4:65534:sync:/bin:/bin/sync
> +games:x:5:60:games:/usr/games:/bin/sh
> +man:x:6:12:man:/var/cache/man:/bin/sh
> +lp:x:7:7:lp:/var/spool/lpd:/bin/sh
> +mail:x:8:8:mail:/var/mail:/bin/sh
> +news:x:9:9:news:/var/spool/news:/bin/sh
> +uucp:x:10:10:uucp:/var/spool/uucp:/bin/sh
> +proxy:x:13:13:proxy:/bin:/bin/sh
> +www-data:x:33:33:www-data:/var/www:/bin/sh
> +backup:x:34:34:backup:/var/backups:/bin/sh
> +list:x:38:38:Mailing List Manager:/var/list:/bin/sh
> +irc:x:39:39:ircd:/var/run/ircd:/bin/sh
> +gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/bin/sh
> +pulse:x:993:1001::/var/run/pulse:/bin/false
> +distcc:x:994:65534::/dev/null:/bin/sh
> +sshd:x:995:994::/var/run/sshd:/bin/false
> +avahi:x:996:995::/var/run/avahi-daemon:/bin/false
> +rpcuser:x:997:996::/var/lib/nfs:/bin/false
> +rpc:x:998:997::/:/bin/false
> +messagebus:x:999:998::/var/lib/dbus:/bin/false
> +nobody:x:65534:65534:nobody:/nonexistent:/bin/sh
> diff --git a/meta-poky/conf/distro/poky-reproducible.conf b/meta-poky/conf/distro/poky-reproducible.conf
> new file mode 100644
> index 0000000..c94f673
> --- /dev/null
> +++ b/meta-poky/conf/distro/poky-reproducible.conf
> @@ -0,0 +1,38 @@
> +require conf/distro/poky.conf
> +DISTRO = "poky-reproducible"
> +
> +BUILD_REPRODUCIBLE_BINARIES = "1"
> +REPRODUCIBLE_TIMESTAMP_ROOTFS ?= "1483228802"
> +LDCONFIGDEPEND = ""
Perhaps this should be disabling the ldconfig distro feature instead?
(Which implies the value of LDCONFIGDEPEND set in image.bbclass should
be updated to be conditional on the ldconfig distro feature too).
> +do_image_cpio[depends] += "cpio-replacement-native:do_populate_sysroot"
> +EXTRANATIVEPATH += "cpio-native"
> +IMAGE_CMD_CPIO = "cpio --ignore-devno --reproducible "
> +
> +IMAGE_CMD_TAR = "tar -v --sort=name "
> +
> +PACKAGE_CLASSES ="package_deb"
> +
> +# For reproducibility, we need to consistently assign the UID/GID values.
> +# Use the static uid and gid mechanism from OE-core for that:
> +# http://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-classes-useradd
> +#
> +# Dynamically assigned IDs are detected and lead to an error during
> +# the build.
> +#
> +# Developers who need to add new entries should add their own mapping
> +# file to USERADD_UID_TABLES and/or USERADD_GID_TABLES, either in a
> +# derived distro config or in their local.conf.
> +#
> +# It is also possible to disable the mechanism by modifying USERADD_ERROR_DYNAMIC:
> +# "warn" merely prints a warning, empty value silently allows dynamic
> +# ID allocation.
> +#
> +# The actual files for UID/GID values come from core-image-minimal-sdk
> +# /etc/group
> +# /etc/passwd
> +
> +USERADDEXTENSION = "useradd-staticids"
> +USERADD_ERROR_DYNAMIC ??= "error"
> +USERADD_UID_TABLES += "conf/distro/include/reproducible-passwd"
> +USERADD_GID_TABLES += "conf/distro/include/reproducible-group"
> +
> --
> 2.7.4
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v3 00/11] Reproducible binaries
2017-08-09 17:48 [PATCH v3 00/11] Reproducible binaries Juro Bystricky
` (10 preceding siblings ...)
2017-08-09 17:48 ` [PATCH v3 11/11] poky-reproducible.conf: Initial version Juro Bystricky
@ 2017-08-18 18:11 ` Martin Jansa
2017-08-18 18:27 ` Bystricky, Juro
11 siblings, 1 reply; 28+ messages in thread
From: Martin Jansa @ 2017-08-18 18:11 UTC (permalink / raw)
To: Juro Bystricky
Cc: Juro Bystricky, Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 10058 bytes --]
I've seen that some of these patches were already merged to master.
Can you please resend remaining patches for oe-core? I'm testing some
slightly older version of your patches and I see couple recipes failing
e.g. like this one:
http://errors.yoctoproject.org/Errors/Details/151939/
On Wed, Aug 9, 2017 at 7:48 PM, Juro Bystricky <juro.bystricky@intel.com>
wrote:
> This patch-set contains basic changes needed in order to support building
> of
> reproducible bianries. The set containes the following patches:
>
> 0001-reproducible_build.bbclass-initial-support-for-binar.patch
> 0002-image-prelink.bbclass-support-binary-reproducibility.patch
> 0003-rootfs-postcommands.bbclass-support-binary-reproduci.patch
> 0004-busybox.inc-improve-reproducibility.patch
> 0005-image.bbclass-support-binary-reproducibility.patch
> 0006-cpio-provide-cpio-replacement-native.patch
> 0007-image_types.bbclass-improve-cpio-image-reproducibili.patch
> 0008-python2.7-improve-reproducibility.patch
> 0009-python3-improve-reproducibility.patch
> 0010-kernel.bbclass-improve-reproducibility.patch
> 0011-poky-reproducible.conf-Initial-version.patch
>
> Using this patch set while building core-image minimal (two clean builds,
> same
> machine/OS, same date, two different folders, at two different times) I
> got the
> following results:
>
> Same:
>
> core-image-minimal-initramfs-qemux86
> bzImage-qemux86.bin
> vmlinux.gz-qemux86.bin
> (Some binaries i.e. ext4 differ, but the differnce is due to conversion to
> .ext4)
>
> Comparing Debian packages in tmp/deploy/deb:
>
> Same: 4005
> Different: 38
> Total: 4043
>
> (The remaining packages that still differ can be dealt with on an
> individual basis)
>
>
> Although the patches contain commit messages explaining the purpose and
> implementation,
> a somewhat more detailed description of selected patches seems prudent:
>
> 0001-reproducible_build.bbclass-initial-support-for-binar.patch
> ===============================================================
>
> This patch creates a new class "reproducible_build.bbclass",
> introducing two new variables:
>
> BUILD_REPRODUCIBLE_BINARIES: "0" (default) business as usual, "1" turn on
> various pieces of
> codes to improve reproducible builds
>
> REPRODUCIBLE_TIMESTAMP_ROOTFS: only used if BUILD_REPRODUCIBLE_BINARIES="
> 1".
> Catch-all timestamp for various rootfs files, pre-linker, etc. If needed,
> timestamps can
> be better granulated later on, right now we use a single value.
>
> Having a new variable BUILD_REPRODUCIBLE_BINARIES serves two purposes:
> 1. Lets user decide (there are minor trade-offs)
> 2. Setting to "0" will guarantee to cause zero regressions.
> 3. Setting to "1" will force the the environment to contain
> SOURCE_DATE_EPOCH
>
> BUILD_REPRODUCIBLE_BINARIES is globally exported, as this will initially
> force all kinds
> of rebuilds. I know no simple way around this, though. This variable is
> needed in numerous
> places: configuration, compilation, rootfs creation, packaging etc.
> REPRODUCIBLE_TIMESTAMP_ROOTFS does not need to be globally exported, it is
> exported locally
> based on the need.
> Once these variables are "official", various classes and recipes can be
> modified to conditionally
> support binary reproducibility.
>
> Setting SOURCE_DATE_EPOCH is essential for binary reproducibility.
> We need to set a recipe specific SOURCE_DATE_EPOCH in each recipe
> environment for various tasks.
> One way would be to modify all recipes one-by-one, but that is not
> realistic. So determining
> SOURCE_DATE_EPOCH is done in this class automatically: After sources are
> unpacked (but
> before they are patched), we try to determine the value for
> SOURCE_DATE_EPOCH.
>
> There are 4 ways to determine SOURCE_DATE_EPOCH:
> 1. Use value from src-data-epoch.txt file if this file exists. This file
> was most likely created
> in the previous build by one of the following methods 2,3,4.
> (But it could be actually provided by a recipe via SRC_URI)
>
> If the file does not exist:
> 2. Use .git last commit date timestamp (git does not allow checking out
> files and preserving their
> timestamps)
> 3. Use "known" files such as NEWS, CHANGLELOG, ...
> 4. Use the youngest file of the source tree.
>
> Once the value of SOURCE_DATE_EPOCH is determined, it is stored in the
> recipe source tree in
> a text file "src-date-epoch.txt'.
>
> If this file is found by other recipe task, the value is placed in the
> SOURCE_DATE_EPOCH var in
> the task environment. This is done in an anonymous python function, so
> SOURCE_DATE_EPOCH is
> guaranteed to exist for all tasks. (If the file is not found
> SOURCE_DATE_EPOCH is set to 0)
> This can optimized in the future, as some tasks (all tasks before fetch,
> tasks such as package QA,
> rm_work, ...) do not need SOURCE_DATE_EPOCH in the environment.
>
>
> 0008-python2.7-improve-reproducibility.patch
> 0009-python3-improve-reproducibility.patch
> ============================================
> These are back ports of existing patches. They ensure the compiled .pyc
> files
> contain timestamp based on SOURCE_DATE_EPOCH (if defined in the
> environment).
> (May not be needed in the future, my understanding is support for
> SOURCE_DATE_EPOCH is already
> upstreamed in master)
>
>
> 0010-kernel.bbclass-improve-reproducibility.patch
> =================================================
>
> This patch contains several changes, was created by squashing several
> commits.
> Several tweaks to improve reproducibility:
>
> We want to set KBUILD_BUILD_TIMESTAMP to some reproducible value. Normally,
> we would use the value for SOURCE_DATE_EPOCH. However, to accommodate
> local kernel sources,
> these are not obtained the usual way via do_unpack and hHence we end up
> with
> SOURCE_DATE_EPOCH set to 0. In this case we obtain the timestamp from top
> entry of GIT repo,
> or (if there is no GIT repo) fallback to REPRODUCIBLE_TIMESTAMP_ROOTFS as
> the last resort.
>
> Kernel and kernel modules contain hard coded paths referencing the host
> build system. This is usually because the source code contains __FILE__
> at some place. This prevents binary reproducibility. However, some
> compilers
> allow remapping of the __FILE__ value. If we detect the compiler is capable
> of doing this, we replace the source path $(S) part of __FILE__ by a
> string "/kernel-source".
> This works very well for oe-embedded cross-compilers, but it is not
> guaranteed to work for
> external toolchains. Hence, the check for the option being supported. Note
> that this
> is done regardless of the value od BUILD_REPRODUCIBLE_BINARIES.
>
> When compressing vmlinux.gz, use gzip "-n" option as recommended in all
> guidelines to achieve
> binary reproducibility.
>
>
> 0011-poky-reproducible.conf-Initial-version.patch
> =================================================
> Support building of reproducible images by setting
> DISTRO="poky-reproducible"
>
> This is mostly for convenience so the user does not have to modify
> local.conf.
>
> Please note setting LDCONFIGDEPEND = ""
> This prevents building of ldconfig cache, which (when built) breaks binary
> reproducibility.
>
> Also, it should avoid reproducibility issue with etc/passwd, where for
> example
> two different builds can lead to two different values i.e:
>
> build 1:
> distcc:x:993:65534::/dev/null:/bin/sh
> pulse:x:994:1001::/var/run/pulse:/bin/false
>
> build 2:
> pulse:x:993:1001::/var/run/pulse:/bin/false
> distcc:x:994:65534::/dev/null:/bin/sh
>
>
>
> Juro Bystricky (11):
> reproducible_build.bbclass: initial support for binary reproducibility
> image-prelink.bbclass: support binary reproducibility
> rootfs-postcommands.bbclass: support binary reproducibility
> busybox.inc: improve reproducibility
> image.bbclass: support binary reproducibility
> cpio: provide cpio-replacement-native
> image_types.bbclass: improve cpio image reproducibility
> python2.7: improve reproducibility
> python3: improve reproducibility
> kernel.bbclass: improve reproducibility
> poky-reproducible.conf: Initial version
>
> meta-poky/conf/distro/include/reproducible-group | 50 ++++++++++
> meta-poky/conf/distro/include/reproducible-passwd | 25 +++++
> meta-poky/conf/distro/poky-reproducible.conf | 38 ++++++++
> meta/classes/base.bbclass | 4 +
> meta/classes/image-prelink.bbclass | 12 ++-
> meta/classes/image.bbclass | 16 ++-
> meta/classes/image_types.bbclass | 14 ++-
> meta/classes/kernel.bbclass | 39 +++++++-
> meta/classes/reproducible_build.bbclass | 108
> +++++++++++++++++++++
> meta/classes/rootfs-postcommands.bbclass | 27 +++++-
> meta/recipes-core/busybox/busybox.inc | 7 ++
> .../python/python-native_2.7.13.bb | 1 +
> .../python/python/reproducible.patch | 34 +++++++
> .../python/python3-native_3.5.3.bb | 1 +
> .../support_SOURCE_DATE_EPOCH_in_py_compile.patch | 97
> ++++++++++++++++++
> meta/recipes-devtools/python/python3_3.5.3.bb | 1 +
> meta/recipes-devtools/python/python_2.7.13.bb | 1 +
> meta/recipes-extended/cpio/cpio_v2.inc | 2 +
> 18 files changed, 467 insertions(+), 10 deletions(-)
> create mode 100644 meta-poky/conf/distro/include/reproducible-group
> create mode 100644 meta-poky/conf/distro/include/reproducible-passwd
> create mode 100644 meta-poky/conf/distro/poky-reproducible.conf
> create mode 100644 meta/classes/reproducible_build.bbclass
> create mode 100644 meta/recipes-devtools/python/python/reproducible.patch
> create mode 100644 meta/recipes-devtools/python/
> python3/support_SOURCE_DATE_EPOCH_in_py_compile.patch
>
> --
> 2.7.4
>
>
[-- Attachment #2: Type: text/html, Size: 11699 bytes --]
^ permalink raw reply [flat|nested] 28+ messages in thread* Re: [PATCH v3 00/11] Reproducible binaries
2017-08-18 18:11 ` [PATCH v3 00/11] Reproducible binaries Martin Jansa
@ 2017-08-18 18:27 ` Bystricky, Juro
2017-08-18 20:32 ` Martin Jansa
0 siblings, 1 reply; 28+ messages in thread
From: Bystricky, Juro @ 2017-08-18 18:27 UTC (permalink / raw)
To: Martin Jansa
Cc: Juro Bystricky, Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 847 bytes --]
I will resend the remaining patches, today & tomorrow. Obviously there are also some other changes, but the error you mention should be fixed. I will rebase them on the current master as of today
(0bd2dd08e3daf284a6bb7757651af8d40393aec2)
________________________________
From: Martin Jansa [martin.jansa@gmail.com]
Sent: Friday, August 18, 2017 11:11 AM
To: Bystricky, Juro
Cc: Patches and discussions about the oe-core layer; Richard Purdie; Joshua G Lock; Burton, Ross; Juro Bystricky
Subject: Re: [PATCH v3 00/11] Reproducible binaries
I've seen that some of these patches were already merged to master.
Can you please resend remaining patches for oe-core? I'm testing some slightly older version of your patches and I see couple recipes failing e.g. like this one:
http://errors.yoctoproject.org/Errors/Details/151939/
[-- Attachment #2: Type: text/html, Size: 1649 bytes --]
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v3 00/11] Reproducible binaries
2017-08-18 18:27 ` Bystricky, Juro
@ 2017-08-18 20:32 ` Martin Jansa
2017-08-18 20:42 ` Bystricky, Juro
0 siblings, 1 reply; 28+ messages in thread
From: Martin Jansa @ 2017-08-18 20:32 UTC (permalink / raw)
To: Bystricky, Juro
Cc: Juro Bystricky, Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]
Thanks!
FWIW: I've refreshed the patches from ML and still see:
WARNING: linux-yocto-4.10.17+gitAUTOINC+e92bd55409_6648a34e00-r0 do_unpack:
Unable to determine src_date_epoch!
path:/OE/build/oe-core/tmp-glibc/work-shared/qemux86/kernel-source
On Fri, Aug 18, 2017 at 8:27 PM, Bystricky, Juro <juro.bystricky@intel.com>
wrote:
> I will resend the remaining patches, today & tomorrow. Obviously there are
> also some other changes, but the error you mention should be fixed. I will
> rebase them on the current master as of today
> (0bd2dd08e3daf284a6bb7757651af8d40393aec2)
>
>
> ------------------------------
> *From:* Martin Jansa [martin.jansa@gmail.com]
> *Sent:* Friday, August 18, 2017 11:11 AM
> *To:* Bystricky, Juro
> *Cc:* Patches and discussions about the oe-core layer; Richard Purdie;
> Joshua G Lock; Burton, Ross; Juro Bystricky
> *Subject:* Re: [PATCH v3 00/11] Reproducible binaries
>
> I've seen that some of these patches were already merged to master.
>
> Can you please resend remaining patches for oe-core? I'm testing some
> slightly older version of your patches and I see couple recipes failing
> e.g. like this one:
> http://errors.yoctoproject.org/Errors/Details/151939/
>
>
>
[-- Attachment #2: Type: text/html, Size: 2244 bytes --]
^ permalink raw reply [flat|nested] 28+ messages in thread
* Re: [PATCH v3 00/11] Reproducible binaries
2017-08-18 20:32 ` Martin Jansa
@ 2017-08-18 20:42 ` Bystricky, Juro
0 siblings, 0 replies; 28+ messages in thread
From: Bystricky, Juro @ 2017-08-18 20:42 UTC (permalink / raw)
To: Martin Jansa
Cc: Juro Bystricky, Patches and discussions about the oe-core layer
[-- Attachment #1: Type: text/plain, Size: 783 bytes --]
Yes, the warning is legit. because the kernel sources are not obtained the usual way via do_unpack (that's where SOURCE_DATE_EPOCH is being determined). This has been addressed in kernel.bbclass, kernel_do_compile() .
________________________________
From: Martin Jansa [martin.jansa@gmail.com]
Sent: Friday, August 18, 2017 1:32 PM
To: Bystricky, Juro
Cc: Patches and discussions about the oe-core layer; Richard Purdie; Joshua G Lock; Burton, Ross; Juro Bystricky
Subject: Re: [PATCH v3 00/11] Reproducible binaries
Thanks!
FWIW: I've refreshed the patches from ML and still see:
WARNING: linux-yocto-4.10.17+gitAUTOINC+e92bd55409_6648a34e00-r0 do_unpack: Unable to determine src_date_epoch! path:/OE/build/oe-core/tmp-glibc/work-shared/qemux86/kernel-source
[-- Attachment #2: Type: text/html, Size: 1542 bytes --]
^ permalink raw reply [flat|nested] 28+ messages in thread