[Buildroot] [PATCH v4 00/18] Reproducible builds

[Buildroot] [PATCH v4 00/18] Reproducible builds

[Jérôme Pouiller]

This series try to continue work initiated by Gilles Chanteperdrix:
  http://lists.busybox.net/pipermail/buildroot/2016-April/thread.html#160064
  http://lists.busybox.net/pipermail/buildroot/2016-June/thread.html#163905

I dropped some patchs from original series because either:
  - I handled things differently (timestamps in images, support SOURCE_DATE_EPOCH
    in gcc, ...)
  - I didn't had time to test them them (sysroot, cpio, cdrkit, iso9660,...)
  - They doesn't seems necessary anymore (libtool, libgcrypt, libgpg-error, ...)

This version focuses on timestamps. It provide good enough results as soon as
OUTDIR and TOPDIR are the same. Indeed build path appear in plenty of files.
Only patch called "remove full path from .pyc" try to solve this issue. Another
big step could be done by removing rpaths from ELF generated with libtool.

Other thing known to break reproducibility:
  - use of lzop (it unconditionally include timestamps in result)
  - /!\ since we build our own toolchain and toolchain include BR2_FULL_VERSION,
    ccache is incompatible with reproducible

Since this feature is experimental I did not (yet) reported these
incompatibilities in menuconfig.

Gilles Chanteperdrix (3):
  reproducibility: generate SOURCE_DATE_EPOCH
  reproducibility/linux: override build timestamp
  reproducibility/busybox: disable build timestamps

J?r?me Pouiller (15):
  reproducible: fix DATE/TIME macros in toolchain-wrapper
  reproducible: add '-n' to gzip invocations
  fs/tar: make results reproducible
  reproducibility/linux: inhibit build-id
  reproducible: lock modification times in $TARGET_DIR
  fakedate: new package
  core: do not reset DEPENDENCIES_HOST_PREREQ in dependencies.mk
  reproducible: enable fakedate
  pycompile: allow to force compilation
  python2: generate reproducible .pyc
  python3: generate reproducible .pyc
  python2: remove full path from .pyc
  python3: remove full path from .pyc
  reproducible: improve help text
  reproducible: fix coding style

 Config.in                            |  6 +++
 Makefile                             | 12 ++++--
 fs/common.mk                         |  3 ++
 fs/tar/tar.mk                        |  3 +-
 linux/linux.mk                       | 15 ++++++++
 package/busybox/busybox.mk           |  6 +++
 package/fakedate/fakedate            | 59 ++++++++++++++++++++++++++++
 package/fakedate/fakedate.mk         | 15 ++++++++
 package/python/python.mk             | 15 ++++++--
 package/python3/python3.mk           | 15 ++++++--
 support/dependencies/dependencies.mk |  2 -
 support/scripts/pycompile.py         | 11 +++++-
 toolchain/toolchain-wrapper.c        | 74 +++++++++++++++++++++++++++++++++++-
 13 files changed, 222 insertions(+), 14 deletions(-)
 create mode 100755 package/fakedate/fakedate
 create mode 100644 package/fakedate/fakedate.mk

-- 
1.9.1

[Buildroot] [PATCH v4 01/18] reproducibility: generate SOURCE_DATE_EPOCH

[Jérôme Pouiller]

From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

When reproducibility is requested, generate a global SOURCE_DATE_EPOCH
environment variable which contains either the date of Buildroot last
commit if running from a git repository, or the latest release date.

This means that all packages embedding build dates will appear to
have the same build date, so in case of new commit or release, all
packages will appear to have been changed, even though some of them
may not have changed in fact.

The meaning of SOURCE_DATE_EPOCH is specified by the following
specification:
  https://reproducible-builds.org/specs/source-date-epoch/

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v4:
     - s/SOURCE_DATE_GIT/BR2_VERSION_GIT_EPOCH/
    v3:
     - Fix typo in commit message (Arnout)
     - Introduce BR2_VERSION_EPOCH (Thomas and Arnout)
     - Set and export SOURCE_DATE_EPOCH together (Thomas)

 Makefile | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/Makefile b/Makefile
index eff814b..8acbe58 100644
--- a/Makefile
+++ b/Makefile
@@ -87,6 +87,8 @@ all:
 
 # Set and export the version string
 export BR2_VERSION := 2016.11-rc1
+# Actual time the release is cut (for reproducible builds)
+BR2_VERSION_EPOCH = 1478206447
 
 # Save running make version since it's clobbered by the make package
 RUNNING_MAKE_VERSION := $(MAKE_VERSION)
@@ -249,6 +251,8 @@ ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ=UTC
 export LANG=C
 export LC_ALL=C
+BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
+export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))
 endif
 
 # To put more focus on warnings, be less verbose as default
-- 
1.9.1

[Buildroot] [PATCH v4 02/18] reproducible: fix DATE/TIME macros in toolchain-wrapper

[Jérôme Pouiller]

The use __DATE__ and __TIME__ are one of most common sources of
non-reproducible binaries. In order to fix that, gcc begin to support
SOURCE_DATE_EPOCH variable. This patch take advantage of toolchain-wrapper
to provide support of SOURCE_DATE_EPOCH to older gcc versions.

Function get_source_date_epoch() come directly from gcc git.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - Handle $SOURCE_DATE_EPOCH at runtime (Thomas)
    v2:
      - Overload __TIME__ and __DATE__ instead of patching gcc (Thomas)

 toolchain/toolchain-wrapper.c | 74 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 73 insertions(+), 1 deletion(-)

diff --git a/toolchain/toolchain-wrapper.c b/toolchain/toolchain-wrapper.c
index 925d013..26d01b6 100644
--- a/toolchain/toolchain-wrapper.c
+++ b/toolchain/toolchain-wrapper.c
@@ -22,12 +22,17 @@
 #include <unistd.h>
 #include <stdlib.h>
 #include <errno.h>
+#include <time.h>
 
 #ifdef BR_CCACHE
 static char ccache_path[PATH_MAX];
 #endif
 static char path[PATH_MAX];
 static char sysroot[PATH_MAX];
+// strlen("-D__TIME__=\"HH:MM:SS\"") + 1 = 22
+static char source_time[22];
+// strlen("-D__DATE__=\"MMM DD YYYY\"") + 1 = 25
+static char source_date[25];
 
 /**
  * GCC errors out with certain combinations of arguments (examples are
@@ -39,8 +44,11 @@ static char sysroot[PATH_MAX];
  * 	-mfloat-abi=
  * 	-march=
  * 	-mcpu=
+ * 	-D__TIME__=
+ * 	-D__DATE__=
+ * 	-Wno-builtin-macro-redefined
  */
-#define EXCLUSIVE_ARGS	3
+#define EXCLUSIVE_ARGS	6
 
 static char *predef_args[] = {
 #ifdef BR_CCACHE
@@ -136,6 +144,47 @@ static void check_unsafe_path(const char *arg,
 	}
 }
 
+/* Read SOURCE_DATE_EPOCH from environment to have a deterministic
+ * timestamp to replace embedded current dates to get reproducible
+ * results.  Returns -1 if SOURCE_DATE_EPOCH is not defined.
+ */
+time_t get_source_date_epoch()
+{
+	char *source_date_epoch;
+	long long epoch;
+	char *endptr;
+
+	source_date_epoch = getenv("SOURCE_DATE_EPOCH");
+	if (!source_date_epoch)
+		return (time_t) -1;
+
+	errno = 0;
+	epoch = strtoll (source_date_epoch, &endptr, 10);
+	if ((errno == ERANGE && (epoch == LLONG_MAX || epoch == LLONG_MIN))
+			|| (errno != 0 && epoch == 0)) {
+		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
+				"strtoll: %s\n", strerror(errno));
+		exit(2);
+	}
+	if (endptr == source_date_epoch) {
+		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
+				"no digits were found: %s\n", endptr);
+		exit(2);
+	}
+	if (*endptr != '\0') {
+		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
+				"trailing garbage: %s\n", endptr);
+		exit(2);
+	}
+	if (epoch < 0) {
+		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
+				"value must be nonnegative: %lld \n", epoch);
+		exit(2);
+	}
+
+	return (time_t) epoch;
+}
+
 int main(int argc, char **argv)
 {
 	char **args, **cur, **exec_args;
@@ -146,6 +195,7 @@ int main(int argc, char **argv)
 	char *paranoid_wrapper;
 	int paranoid;
 	int ret, i, count = 0, debug;
+	time_t source_date_epoch;
 
 	/* Calculate the relative paths */
 	basename = strrchr(progpath, '/');
@@ -251,6 +301,28 @@ int main(int argc, char **argv)
 	}
 #endif /* ARCH || CPU */
 
+	source_date_epoch = get_source_date_epoch();
+	if (source_date_epoch != -1) {
+		struct tm *tm = localtime(&source_date_epoch);
+		if (!tm) {
+			perror("__FILE__: localtime");
+			return 3;
+		}
+		ret = strftime(source_time, sizeof(source_time), "-D__TIME__=\"%T\"", tm);
+		if (!ret) {
+			perror("__FILE__: overflow");
+			return 3;
+		}
+		*cur++ = source_time;
+		ret = strftime(source_date, sizeof(source_date), "-D__DATE__=\"%b %e %Y\"", tm);
+		if (!ret) {
+			perror("__FILE__: overflow");
+			return 3;
+		}
+		*cur++ = source_date;
+		*cur++ = "-Wno-builtin-macro-redefined";
+	}
+
 	paranoid_wrapper = getenv("BR_COMPILER_PARANOID_UNSAFE_PATH");
 	if (paranoid_wrapper && strlen(paranoid_wrapper) > 0)
 		paranoid = 1;
-- 
1.9.1

[Buildroot] [PATCH v4 03/18] reproducible: add '-n' to gzip invocations

[Jérôme Pouiller]

Default invocation to gzip include timestamp in output file. This feature is
incompatible with BR2_REPRODUCIBLE. It is possible to disable it with '-n'.

The environment variable GZIP can hold a set of default options for gzip. So
instead to find all gzip invocation in build process, we just export 'GZIP=-n'.

Notice bzip2, lzma and xz are not impacted by this problem. On the other hand, lzop
does include timestamp and does not provide any way to disable it.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---

Notes:
    v3:
      - Coding style (Arnout)
      - Fix typos in commit message (Arnout)

 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index 8acbe58..9e588da 100644
--- a/Makefile
+++ b/Makefile
@@ -251,6 +251,7 @@ ifeq ($(BR2_REPRODUCIBLE),y)
 export TZ=UTC
 export LANG=C
 export LC_ALL=C
+export GZIP = -n
 BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))
 endif
-- 
1.9.1

[Buildroot] [PATCH v4 04/18] fs/tar: make results reproducible

[Jérôme Pouiller]

In order to make tar images reproducible, force files order in tarball.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - Use `find | sort |' instead of --sort (Arnout)

 fs/tar/tar.mk | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/fs/tar/tar.mk b/fs/tar/tar.mk
index 11c69c5..70dc454 100644
--- a/fs/tar/tar.mk
+++ b/fs/tar/tar.mk
@@ -7,7 +7,8 @@
 TAR_OPTS := $(call qstrip,$(BR2_TARGET_ROOTFS_TAR_OPTIONS))
 
 define ROOTFS_TAR_CMD
-	tar $(TAR_OPTS) -cf $@ --numeric-owner -C $(TARGET_DIR) .
+	( cd $(TARGET_DIR); find -print0 | LC_ALL=C sort -z | \
+		tar $(TAR_OPTS) -cf $@ --null -T - --no-recursion --numeric-owner )
 endef
 
 $(eval $(call ROOTFS_TARGET,tar))
-- 
1.9.1

[Buildroot] [PATCH v4 05/18] reproducibility/linux: override build timestamp

[Jérôme Pouiller]

From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

Linux kernel include a few information about build environment in its binary.
This feature is incompatible with BR2_REPRODUCIBLE. This patch overload build
information when BR2_REPRODUCIBLE is enabled.

Note that usage of KBUILD_BUILD_TIMESTAMP is not mandatory since Buildroot
use `fakedate'. However, native solution is prefered when upstream
provide one.

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - Justify use of KBUILD_BUILD_TIMESTAMP (Arnout)

 linux/linux.mk | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/linux/linux.mk b/linux/linux.mk
index 988427c..7e826cc 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -94,6 +94,14 @@ LINUX_MAKE_ENV = \
 	$(TARGET_MAKE_ENV) \
 	BR_BINARIES_DIR=$(BINARIES_DIR)
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+LINUX_MAKE_ENV += \
+	KBUILD_BUILD_VERSION=1 \
+	KBUILD_BUILD_USER=buildroot \
+	KBUILD_BUILD_HOST=buildroot \
+	KBUILD_BUILD_TIMESTAMP="$(shell date -d @$(SOURCE_DATE_EPOCH))"
+endif
+
 # Get the real Linux version, which tells us where kernel modules are
 # going to be installed in the target filesystem.
 LINUX_VERSION_PROBED = `$(MAKE) $(LINUX_MAKE_FLAGS) -C $(LINUX_DIR) --no-print-directory -s kernelrelease 2>/dev/null`
-- 
1.9.1

[Buildroot] [PATCH v4 06/18] reproducibility/linux: inhibit build-id

[Jérôme Pouiller]

Linux kernel include build path in debug sections. These sections are stripped
and do not impact build reproducibility directly. However, 'build-id'
depends on content of all sections, including debug sections. So, it
add random bytes in section .notes of kernel image[1]:

  $ readelf -Wn .../vmlinux
  Displaying notes found at file offset 0x00008000 with length 0x00000024:
    Owner                 Data size       Description
    GNU                  0x00000014       NT_GNU_BUILD_ID (unique build ID bitstring)
      Build ID: ca689e2ed3944f49474715908e2ac1bb04907fb2

In order to not depend on build path, patch kernel Makefile to disable
'build-id'.

[1] https://kernelnewbies.org/BuildId

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
       - Better explain why disabling build-id is necessary (in fact, it not
         necessary if build paths are preserved between builds)

 linux/linux.mk | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/linux/linux.mk b/linux/linux.mk
index 7e826cc..a63d1f3 100644
--- a/linux/linux.mk
+++ b/linux/linux.mk
@@ -209,6 +209,13 @@ define LINUX_TRY_PATCH_TIMECONST
 endef
 LINUX_POST_PATCH_HOOKS += LINUX_TRY_PATCH_TIMECONST
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+define LINUX_REMOVE_BUILD_ID
+	sed -i -e s/--build-id/--build-id=none/ $(@D)/Makefile
+endef
+LINUX_POST_PATCH_HOOKS += LINUX_REMOVE_BUILD_ID
+endif
+
 ifeq ($(BR2_LINUX_KERNEL_USE_DEFCONFIG),y)
 LINUX_KCONFIG_DEFCONFIG = $(call qstrip,$(BR2_LINUX_KERNEL_DEFCONFIG))_defconfig
 else ifeq ($(BR2_LINUX_KERNEL_USE_ARCH_DEFAULT_CONFIG),y)
-- 
1.9.1

[Buildroot] [PATCH v4 07/18] reproducibility/busybox: disable build timestamps

[Jérôme Pouiller]

From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>

Busybox includes some information about the build environment in its
binary. For BR2_REPRODUCIBLE, remove that information.

Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
---

Notes:
    v3:
      - Reword commit message (Arnout)

 package/busybox/busybox.mk | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/package/busybox/busybox.mk b/package/busybox/busybox.mk
index fc23a90..f4a241d 100644
--- a/package/busybox/busybox.mk
+++ b/package/busybox/busybox.mk
@@ -36,6 +36,12 @@ BUSYBOX_MAKE_ENV = \
 	$(TARGET_MAKE_ENV) \
 	CFLAGS="$(BUSYBOX_CFLAGS)" \
 	CFLAGS_busybox="$(BUSYBOX_CFLAGS_busybox)"
+
+ifeq ($(BR2_REPRODUCIBLE),y)
+BUSYBOX_MAKE_ENV += \
+	KCONFIG_NOTIMESTAMP=1
+endif
+
 BUSYBOX_MAKE_OPTS = \
 	CC="$(TARGET_CC)" \
 	ARCH=$(KERNEL_ARCH) \
-- 
1.9.1

[Buildroot] [PATCH v4 08/18] reproducible: lock modification times in $TARGET_DIR

[Jérôme Pouiller]

Make sure all files in $TARGET_DIR have a defined modification time before to
generate filesystems.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---

Notes:
    v3:
      - Typo in commit log (Thomas)

 fs/common.mk | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/fs/common.mk b/fs/common.mk
index 2dbef4d..981dcb1 100644
--- a/fs/common.mk
+++ b/fs/common.mk
@@ -95,6 +95,9 @@ endif
 	$$(foreach s,$$(call qstrip,$$(BR2_ROOTFS_POST_FAKEROOT_SCRIPT)),\
 		echo "echo '$$(TERM_BOLD)>>>   Executing fakeroot script $$(s)$$(TERM_RESET)'" >> $$(FAKEROOT_SCRIPT); \
 		echo $$(s) $$(TARGET_DIR) $$(BR2_ROOTFS_POST_SCRIPT_ARGS) >> $$(FAKEROOT_SCRIPT)$$(sep))
+ifeq ($$(BR2_REPRODUCIBLE),y)
+	echo "find $$(TARGET_DIR) -print0 | xargs -0 -r touch -hd @$$(SOURCE_DATE_EPOCH)" >> $$(FAKEROOT_SCRIPT)
+endif
 	$$(call PRINTF,$$(ROOTFS_$(2)_CMD)) >> $$(FAKEROOT_SCRIPT)
 	chmod a+x $$(FAKEROOT_SCRIPT)
 	PATH=$$(BR_PATH) $$(HOST_DIR)/usr/bin/pseudo -- $$(FAKEROOT_SCRIPT)
-- 
1.9.1

[Buildroot] [PATCH v4 09/18] fakedate: new package

[Jérôme Pouiller]

`date' is widely used by packages to include build information in their
binaries. Unfortunately, this is incompatible with  BR2_REPRODUCIBLE.

Instead of having to identify all `date' invocations in the different
packages, this commit adds a small tool that allows to always return
the same date.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - Do not force $PATH. Find true `date' binary instead (Thomas)
      - Add explanations (Thomas)
      - Fix options detection (Arnout)
      - Rename INHIBIT in FORCE_EPOCH (Thomas)
      - Break after FORCE_EPOCH=0 (Arnout)
      - Add support for '--reference'
      - Add full copyright blurb (Arnout)

 package/fakedate/fakedate    | 59 ++++++++++++++++++++++++++++++++++++++++++++
 package/fakedate/fakedate.mk | 15 +++++++++++
 2 files changed, 74 insertions(+)
 create mode 100755 package/fakedate/fakedate
 create mode 100644 package/fakedate/fakedate.mk

diff --git a/package/fakedate/fakedate b/package/fakedate/fakedate
new file mode 100755
index 0000000..074c517
--- /dev/null
+++ b/package/fakedate/fakedate
@@ -0,0 +1,59 @@
+#!/bin/sh
+# vim: set sw=4 expandtab:
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+#
+# Copyright (C) 2016 J?r?me Pouiller <jezz@sysmic.org>
+#
+LOG=/dev/null
+
+# Sanity check
+if ! readlink -f "$0" | grep -q fakedate; then
+    echo "fakedate: Please name this script \`fakedate'"
+    exit 1
+fi
+
+DATE_BIN=false
+# Do not call `date'd directly since it will produce an infinite recursion.
+# Instead, find path of true `date' binary.
+for P in `echo $PATH | tr ':' ' '`; do
+    if [ -x "$P/date" ]; then
+        if readlink -f "$P/date" | grep -qv fakedate; then
+            DATE_BIN="$P/date"
+            break;
+        fi
+    fi
+done
+
+if [ -n "$SOURCE_DATE_EPOCH" ]; then
+    FORCE_EPOCH=1
+    for i in "$@"; do
+        # Use of --date, --file and --reference (and their short option counter
+        # parts) is incompatible with SOURCE_DATE_EPOCH.
+        # -u and -R are the only short options without argument. So they could
+        # appear between '-' and option we want to match.
+        if echo "$i" | grep -qE '^-([uR]*d|-date|[uR]*f|-file|[uR]*r|--reference)'; then
+            FORCE_EPOCH=0
+            break;
+        fi
+    done
+    if [ $FORCE_EPOCH -eq 1 ]; then
+        echo "date: Warning: using \$SOURCE_DATE_EPOCH instead of true time" >&2
+        echo "Catch call to date from `pwd` with parameters: '$@'" >> $LOG
+        exec $DATE_BIN -d "@$SOURCE_DATE_EPOCH" "$@"
+    fi
+fi
+
+exec $DATE_BIN "$@"
diff --git a/package/fakedate/fakedate.mk b/package/fakedate/fakedate.mk
new file mode 100644
index 0000000..61d4bd7
--- /dev/null
+++ b/package/fakedate/fakedate.mk
@@ -0,0 +1,15 @@
+################################################################################
+#
+# fakedate
+#
+################################################################################
+
+# source included in buildroot
+HOST_FAKEDATE_LICENSE = GPLv2+
+
+define HOST_FAKEDATE_INSTALL_CMDS
+	$(INSTALL) -D -m 755 package/fakedate/fakedate $(HOST_DIR)/usr/bin/fakedate
+	ln -sfn fakedate $(HOST_DIR)/usr/bin/date
+endef
+
+$(eval $(host-generic-package))
-- 
1.9.1

[Buildroot] [PATCH v4 10/18] core: do not reset DEPENDENCIES_HOST_PREREQ in dependencies.mk

[Jérôme Pouiller]

Usually, Buildroot does never initialize variables with empty content.
DEPENDENCIES_HOST_PREREQ was an unjustified exception.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - New patch

 support/dependencies/dependencies.mk | 2 --
 1 file changed, 2 deletions(-)

diff --git a/support/dependencies/dependencies.mk b/support/dependencies/dependencies.mk
index 4334dac..d4b0409 100644
--- a/support/dependencies/dependencies.mk
+++ b/support/dependencies/dependencies.mk
@@ -5,8 +5,6 @@
 #
 ################################################################################
 
-DEPENDENCIES_HOST_PREREQ :=
-
 # suitable-host-pkg: calls check-host-$(1).sh shell script. Parameter (2)
 # can be the candidate to be checked. If not present, the check-host-$(1).sh
 # script should use 'which' to find a candidate. The script should return
-- 
1.9.1

[Buildroot] [PATCH v4 11/18] reproducible: enable fakedate

[Jérôme Pouiller]

Enable fakedate for whole build process.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - Build host-fakedate before toolchain (Thomas)

 Makefile | 1 +
 1 file changed, 1 insertion(+)

diff --git a/Makefile b/Makefile
index 9e588da..819c8df 100644
--- a/Makefile
+++ b/Makefile
@@ -254,6 +254,7 @@ export LC_ALL=C
 export GZIP = -n
 BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))
+DEPENDENCIES_HOST_PREREQ += host-fakedate
 endif
 
 # To put more focus on warnings, be less verbose as default
-- 
1.9.1

[Buildroot] [PATCH v4 12/18] pycompile: allow to force compilation

[Jérôme Pouiller]

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - new patch

 support/scripts/pycompile.py | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

diff --git a/support/scripts/pycompile.py b/support/scripts/pycompile.py
index fde711a..5598f8a 100644
--- a/support/scripts/pycompile.py
+++ b/support/scripts/pycompile.py
@@ -10,6 +10,7 @@ from __future__ import print_function
 import sys
 import py_compile
 import compileall
+import argparse
 
 class ReportProblem:
     def __nonzero__(self):
@@ -21,4 +22,12 @@ class ReportProblem:
 
 report_problem = ReportProblem()
 
-compileall.compile_dir(sys.argv[1], quiet=report_problem)
+parser = argparse.ArgumentParser(description='Compile Python source files in a directory tree.')
+parser.add_argument("target", metavar='DIRECTORY',
+        help='Directory to scan')
+parser.add_argument("--force", action='store_true',
+        help="Force compilation even if alread compiled")
+
+args = parser.parse_args()
+
+compileall.compile_dir(args.target, force=args.force, quiet=report_problem)
-- 
1.9.1

[Buildroot] [PATCH v4 13/18] python2: generate reproducible .pyc

[Jérôme Pouiller]

.pyc files contain modification time of .py source. In order to make
build reproducible, we fix modification time of all .py before to
compile .pyc files.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - Force compilation instead of removing .pyc

 package/python/python.mk | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/package/python/python.mk b/package/python/python.mk
index cc65376..c17b267 100644
--- a/package/python/python.mk
+++ b/package/python/python.mk
@@ -226,10 +226,18 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+define PYTHON_FIX_TIME
+find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
+		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
+endef
+PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME
+endif
+
 define PYTHON_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON_PATH)" \
 	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
-		support/scripts/pycompile.py \
+		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
 		$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)
 endef
 
-- 
1.9.1

[Buildroot] [PATCH v4 14/18] python3: generate reproducible .pyc

[Jérôme Pouiller]

.pyc files contain modification time of .py source. In order to make
build reproducible, we fix modification time of all .py before to
compile .pyc files.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - Force compilation instead of removing .pyc

 package/python3/python3.mk | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/package/python3/python3.mk b/package/python3/python3.mk
index b3f31c0..a526c0f 100644
--- a/package/python3/python3.mk
+++ b/package/python3/python3.mk
@@ -219,10 +219,18 @@ PYTHON3_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)/sysconfigdat
 $(eval $(autotools-package))
 $(eval $(host-autotools-package))
 
+ifeq ($(BR2_REPRODUCIBLE),y)
+define PYTHON3_FIX_TIME
+	find $(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR) -name '*.py' -print0 | \
+		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
+endef
+PYTHON3_TARGET_FINALIZE_HOOKS += PYTHON3_FIX_TIME
+endif
+
 define PYTHON3_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON3_PATH)" \
 	$(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
-		support/scripts/pycompile.py \
+		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
 		$(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
 endef
 
-- 
1.9.1

[Buildroot] [PATCH v4 15/18] python2: remove full path from .pyc

[Jérôme Pouiller]

.pyc files include path to source .py file. This patch changes the way
`pycompile.py' is launched in order to only keep the part relative to
$TARGET_DIR.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---

Notes:
    v3:
     - Typo in commit log (Arnout)

 package/python/python.mk | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/package/python/python.mk b/package/python/python.mk
index c17b267..ffe4e14 100644
--- a/package/python/python.mk
+++ b/package/python/python.mk
@@ -236,9 +236,10 @@ endif
 
 define PYTHON_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON_PATH)" \
-	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
-		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
-		$(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)
+	cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
+		$(TOPDIR)/support/scripts/pycompile.py \
+		$(if $(BR2_REPRODUCIBLE),--force) \
+		usr/lib/python$(PYTHON_VERSION_MAJOR)
 endef
 
 ifeq ($(BR2_PACKAGE_PYTHON_PYC_ONLY)$(BR2_PACKAGE_PYTHON_PY_PYC),y)
-- 
1.9.1

[Buildroot] [PATCH v4 16/18] python3: remove full path from .pyc

[Jérôme Pouiller]

.pyc files include path to source .py file. This patch changes the way
`pycompile.py' is launched in order to only keep the part relative to
$TARGET_DIR.

This work was sponsored by `BA Robotic Systems'.

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
---

Notes:
    v3:
      - Typo in commit log (Arnout)

 package/python3/python3.mk | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/package/python3/python3.mk b/package/python3/python3.mk
index a526c0f..4f0ae5b 100644
--- a/package/python3/python3.mk
+++ b/package/python3/python3.mk
@@ -229,9 +229,10 @@ endif
 
 define PYTHON3_CREATE_PYC_FILES
 	PYTHONPATH="$(PYTHON3_PATH)" \
-	$(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
-		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
-		$(TARGET_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)
+	cd $(TARGET_DIR) && $(HOST_DIR)/usr/bin/python$(PYTHON3_VERSION_MAJOR) \
+		$(TOPDIR)/support/scripts/pycompile.py \
+		$(if $(BR2_REPRODUCIBLE),--force) \
+		usr/lib/python$(PYTHON3_VERSION_MAJOR)
 endef
 
 ifeq ($(BR2_PACKAGE_PYTHON3_PYC_ONLY)$(BR2_PACKAGE_PYTHON3_PY_PYC),y)
-- 
1.9.1

[Buildroot] [PATCH v4 17/18] reproducible: improve help text

[Jérôme Pouiller]

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---

Notes:
    v3:
      - Reword help message (Arnout)
    v2:
      - New patch (Thomas)

 Config.in | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/Config.in b/Config.in
index 5cf0c4d..e382cf6 100644
--- a/Config.in
+++ b/Config.in
@@ -707,6 +707,12 @@ config BR2_REPRODUCIBLE
 	  this allows to generate exactly identical binaries from one
 	  build to the other, including on different machines.
 
+	  The current implementation is restricted to builds with the same
+	  output directory. Many (absolute) paths are recorded in intermediary
+	  files, and it is very likely that some of these paths leak into the
+	  target rootfs. If you build with the same O=... path, however, the
+	  result is identical.
+
 	  This is labeled as an experimental feature, as not all
 	  packages behave properly to ensure reproducibility.
 
-- 
1.9.1

[Buildroot] [PATCH v4 18/18] reproducible: fix coding style

[Jérôme Pouiller]

Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
---
Notes:
   v3:
     - New patch (Arnout)

 Makefile | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/Makefile b/Makefile
index 819c8df..d965304 100644
--- a/Makefile
+++ b/Makefile
@@ -248,9 +248,9 @@ endif
 
 # timezone and locale may affect build output
 ifeq ($(BR2_REPRODUCIBLE),y)
-export TZ=UTC
-export LANG=C
-export LC_ALL=C
+export TZ = UTC
+export LANG = C
+export LC_ALL = C
 export GZIP = -n
 BR2_VERSION_GIT_EPOCH = $(shell GIT_DIR=$(TOPDIR)/.git $(GIT) log -1 --format=%at)
 export SOURCE_DATE_EPOCH = $(if $(wildcard $(TOPDIR)/.git),$(BR2_VERSION_GIT_EPOCH),$(BR2_VERSION_EPOCH))
-- 
1.9.1

[Buildroot] [PATCH v4 01/18] reproducibility: generate SOURCE_DATE_EPOCH

[Thomas Petazzoni]

Hello,

On Wed, 23 Nov 2016 13:58:40 +0100, J?r?me Pouiller wrote:
> From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> 
> When reproducibility is requested, generate a global SOURCE_DATE_EPOCH
> environment variable which contains either the date of Buildroot last
> commit if running from a git repository, or the latest release date.
> 
> This means that all packages embedding build dates will appear to
> have the same build date, so in case of new commit or release, all
> packages will appear to have been changed, even though some of them
> may not have changed in fact.
> 
> The meaning of SOURCE_DATE_EPOCH is specified by the following
> specification:
>   https://reproducible-builds.org/specs/source-date-epoch/
> 
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---

Applied to next, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v4 03/18] reproducible: add '-n' to gzip invocations

[Thomas Petazzoni]

Hello,

On Wed, 23 Nov 2016 13:58:42 +0100, J?r?me Pouiller wrote:
> Default invocation to gzip include timestamp in output file. This feature is
> incompatible with BR2_REPRODUCIBLE. It is possible to disable it with '-n'.
> 
> The environment variable GZIP can hold a set of default options for gzip. So
> instead to find all gzip invocation in build process, we just export 'GZIP=-n'.
> 
> Notice bzip2, lzma and xz are not impacted by this problem. On the other hand, lzop
> does include timestamp and does not provide any way to disable it.
> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---

Applied to next, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v4 04/18] fs/tar: make results reproducible

[Thomas Petazzoni]

Hello,

On Wed, 23 Nov 2016 13:58:43 +0100, J?r?me Pouiller wrote:

>  define ROOTFS_TAR_CMD
> -	tar $(TAR_OPTS) -cf $@ --numeric-owner -C $(TARGET_DIR) .
> +	( cd $(TARGET_DIR); find -print0 | LC_ALL=C sort -z | \
> +		tar $(TAR_OPTS) -cf $@ --null -T - --no-recursion --numeric-owner )

I've remove the space after ( and before ).

Applied with this changed. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v4 05/18] reproducibility/linux: override build timestamp

[Thomas Petazzoni]

Hello,

On Wed, 23 Nov 2016 13:58:44 +0100, J?r?me Pouiller wrote:
> From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> 
> Linux kernel include a few information about build environment in its binary.
> This feature is incompatible with BR2_REPRODUCIBLE. This patch overload build
> information when BR2_REPRODUCIBLE is enabled.
> 
> Note that usage of KBUILD_BUILD_TIMESTAMP is not mandatory since Buildroot
> use `fakedate'. However, native solution is prefered when upstream
> provide one.
> 
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---

Applied to next, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v4 07/18] reproducibility/busybox: disable build timestamps

[Thomas Petazzoni]

Hello,

On Wed, 23 Nov 2016 13:58:46 +0100, J?r?me Pouiller wrote:
> From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> 
> Busybox includes some information about the build environment in its
> binary. For BR2_REPRODUCIBLE, remove that information.
> 
> Signed-off-by: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> Reviewed-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> ---

Applied to next, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v4 08/18] reproducible: lock modification times in $TARGET_DIR

[Thomas Petazzoni]

Hello,

On Wed, 23 Nov 2016 13:58:47 +0100, J?r?me Pouiller wrote:
> Make sure all files in $TARGET_DIR have a defined modification time before to
> generate filesystems.
> 
> This work was sponsored by `BA Robotic Systems'.
> 
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> Reviewed-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Reviewed-by: Arnout Vandecappelle (Essensium/Mind) <arnout@mind.be>
> ---

Applied to next, thanks.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v4 12/18] pycompile: allow to force compilation

[Thomas Petazzoni]

Hello,

On Wed, 23 Nov 2016 13:58:51 +0100, J?r?me Pouiller wrote:
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>

Empty commit log. Why do we want to do this?

Remember: for any non-trivial patch, an empty commit log is always
wrong.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v4 13/18] python2: generate reproducible .pyc

[Thomas Petazzoni]

Hello,

On Wed, 23 Nov 2016 13:58:52 +0100, J?r?me Pouiller wrote:
> .pyc files contain modification time of .py source. In order to make
> build reproducible, we fix modification time of all .py before to
> compile .pyc files.

"before to compile" -> "before compiling".

> diff --git a/package/python/python.mk b/package/python/python.mk
> index cc65376..c17b267 100644
> --- a/package/python/python.mk
> +++ b/package/python/python.mk
> @@ -226,10 +226,18 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/
>  $(eval $(autotools-package))
>  $(eval $(host-autotools-package))
>  
> +ifeq ($(BR2_REPRODUCIBLE),y)
> +define PYTHON_FIX_TIME
> +find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
> +		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
> +endef
> +PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME
> +endif
> +
>  define PYTHON_CREATE_PYC_FILES
>  	PYTHONPATH="$(PYTHON_PATH)" \

It would make more sense to just do:

	$(PYTHON_FIX_TIME)

here, rather than registering it as a PYTHON_TARGET_FINALIZE_HOOKS.
Also, maybe it should be named PYTHON_SET_PY_FILES_TIME.

>  	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
> -		support/scripts/pycompile.py \
> +		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \

Why do we need to force?

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v4 14/18] python3: generate reproducible .pyc

[Thomas Petazzoni]

Hello,

On Wed, 23 Nov 2016 13:58:53 +0100, J?r?me Pouiller wrote:
> .pyc files contain modification time of .py source. In order to make
> build reproducible, we fix modification time of all .py before to
> compile .pyc files.

Is there a way of avoiding the modification time contained inside
the .pyc files? According to
http://wiki.baserock.org/projects/deterministic-builds/: "Debian solve
this by creating the .pyc and .pyo files at package install time",
which is essentially what we do. How do they handle this exactly?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH v4 17/18] reproducible: improve help text

[Thomas Petazzoni]

Hello,

On Wed, 23 Nov 2016 13:58:56 +0100, J?r?me Pouiller wrote:
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---

Applied to next, after tweaking the commit title. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v4 18/18] reproducible: fix coding style

[Thomas Petazzoni]

Hello,

On Wed, 23 Nov 2016 13:58:57 +0100, J?r?me Pouiller wrote:
> Signed-off-by: J?r?me Pouiller <jezz@sysmic.org>
> ---
> Notes:
>    v3:
>      - New patch (Arnout)

Applied to next, after tweaking the commit title and adding some
details in the commit log. Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[Buildroot] [PATCH v4 13/18] python2: generate reproducible .pyc

[Arnout Vandecappelle]

On 23-11-16 23:05, Thomas Petazzoni wrote:
> Hello,
> 
> On Wed, 23 Nov 2016 13:58:52 +0100, J?r?me Pouiller wrote:
>> .pyc files contain modification time of .py source. In order to make
>> build reproducible, we fix modification time of all .py before to
>> compile .pyc files.
> 
> "before to compile" -> "before compiling".
> 
>> diff --git a/package/python/python.mk b/package/python/python.mk
>> index cc65376..c17b267 100644
>> --- a/package/python/python.mk
>> +++ b/package/python/python.mk
>> @@ -226,10 +226,18 @@ PYTHON_PATH = $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR)/sysconfigdata/
>>  $(eval $(autotools-package))
>>  $(eval $(host-autotools-package))
>>  
>> +ifeq ($(BR2_REPRODUCIBLE),y)
>> +define PYTHON_FIX_TIME
>> +find $(TARGET_DIR)/usr/lib/python$(PYTHON_VERSION_MAJOR) -name '*.py' -print0 | \
>> +		xargs -0 --no-run-if-empty touch -d @$(SOURCE_DATE_EPOCH)
>> +endef
>> +PYTHON_TARGET_FINALIZE_HOOKS += PYTHON_FIX_TIME
>> +endif
>> +
>>  define PYTHON_CREATE_PYC_FILES
>>  	PYTHONPATH="$(PYTHON_PATH)" \
> 
> It would make more sense to just do:
> 
> 	$(PYTHON_FIX_TIME)
> 
> here, rather than registering it as a PYTHON_TARGET_FINALIZE_HOOKS.

 No. It iterates over all python files in $(TARGET_DIR), not just the ones
installed by this package, so it really should be a TARGET_FINALIZE_HOOK.


 Regards,
 Arnout



> Also, maybe it should be named PYTHON_SET_PY_FILES_TIME.
> 
>>  	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
>> -		support/scripts/pycompile.py \
>> +		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
> 
> Why do we need to force?
> 
> Thomas
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH v4 13/18] python2: generate reproducible .pyc

[Jérôme Pouiller]

Hello Thomas,

On Thursday 24 November 2016 20:06:55 Arnout Vandecappelle wrote:
[...]
> >>  	$(HOST_DIR)/usr/bin/python$(PYTHON_VERSION_MAJOR) \
> >> -		support/scripts/pycompile.py \
> >> +		support/scripts/pycompile.py $(if $(BR2_REPRODUCIBLE),--force) \
> > 
> > Why do we need to force?
Pycompile rely on .py modification times to know if a file need to be
recompiled. Since reproducible builds tweak modification time, it is
safer to force recompilation of all source files.

(I will add this explanation to commit log)

-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr

[Buildroot] [PATCH v4 14/18] python3: generate reproducible .pyc

[Jérôme Pouiller]

Hello Thomas,

On Wednesday 23 November 2016 23:09:39 Thomas Petazzoni wrote:
> Hello,
> 
> On Wed, 23 Nov 2016 13:58:53 +0100, J?r?me Pouiller wrote:
> > .pyc files contain modification time of .py source. In order to make
> > build reproducible, we fix modification time of all .py before to
> > compile .pyc files.
> 
> Is there a way of avoiding the modification time contained inside
> the .pyc files? According to
> http://wiki.baserock.org/projects/deterministic-builds/: "Debian solve
> this by creating the .pyc and .pyo files at package install time",
> which is essentially what we do. How do they handle this exactly?
Debian compile .pyc and .pyo when user install package. deb files does 
not contains compiled files. Therefore, .pyc reproducibility does not 
matter.

In Buildroot, .pyc are included in filesystem. .pyc reproducibility does 
matter.


-- 
J?r?me Pouiller, Sysmic
Embedded Linux specialist
http://www.sysmic.fr