[Buildroot] [PATCH 0/2 next] toolchain/wrapper: adhere to SOURCE_DATE_EPOCH

[Buildroot] [PATCH 0/2 next] toolchain/wrapper: adhere to SOURCE_DATE_EPOCH

[Yann E. MORIN]

Hello All!

This series, for the next branch, re-introduces conformance with
SOURCE_DATE_EPOCH, with a clean-room re-implementation.

The first patch is cherry-picked from master, to remove the previous
implementation, while the second patch actually provides the new
implementation.


Regards,
Yann E. MORIN.


The following changes since commit 9a7ac39605e1bf416db033b009e6c09f54f44fdf

  keepalived: new package (2017-02-20 22:44:08 +0100)


are available in the git repository at:

  git://git.buildroot.org/~ymorin/git/buildroot.git

for you to fetch changes up to 3c4084fc0c3cf518c6fe170bebd83124730eba00

  toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc (2017-02-21 21:31:44 +0100)


----------------------------------------------------------------
Yann E. MORIN (2):
      Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"
      toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc

 toolchain/toolchain-wrapper.c | 109 +++++++++++++++++++++---------------------
 1 file changed, 55 insertions(+), 54 deletions(-)

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/2 next] Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

[Yann E. MORIN]

This reverts commit 76838f63412a30a358210e457dda4b79f7730624.

The commit referenced above explicitly states that the function was
copied as-is from the gcc source code at the time. And indeed that is
exactly how the function appeared in gcc in commit
e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934.

However, our toolchain wrapper is "GPLv2 only", while the file this function
was copied from is "GPLv3 or later".  As such we can't include that function
and still comply to both licenses.

Furthermore, the code is far from optimum.

Since this feature is not release-critical, revert it until we re-implement
it.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: J?r?me Pouiller <jezz@sysmic.org>
Cc: Samuel Martin <s.martin49@gmail.com>
Cc: Peter Korsgaard <peter@korsgaard.com>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Signed-off-by: Peter Korsgaard <peter@korsgaard.com>
---
 toolchain/toolchain-wrapper.c | 72 +------------------------------------------
 1 file changed, 1 insertion(+), 71 deletions(-)

diff --git a/toolchain/toolchain-wrapper.c b/toolchain/toolchain-wrapper.c
index eb4ee8d..100aa18 100644
--- a/toolchain/toolchain-wrapper.c
+++ b/toolchain/toolchain-wrapper.c
@@ -22,15 +22,12 @@
 #include <unistd.h>
 #include <stdlib.h>
 #include <errno.h>
-#include <time.h>
 
 #ifdef BR_CCACHE
 static char ccache_path[PATH_MAX];
 #endif
 static char path[PATH_MAX];
 static char sysroot[PATH_MAX];
-static char source_time[sizeof("-D__TIME__=\"HH:MM:SS\"")];
-static char source_date[sizeof("-D__DATE__=\"MMM DD YYYY\"")];
 
 /**
  * GCC errors out with certain combinations of arguments (examples are
@@ -42,11 +39,8 @@ static char source_date[sizeof("-D__DATE__=\"MMM DD YYYY\"")];
  * 	-mfloat-abi=
  * 	-march=
  * 	-mcpu=
- * 	-D__TIME__=
- * 	-D__DATE__=
- * 	-Wno-builtin-macro-redefined
  */
-#define EXCLUSIVE_ARGS	6
+#define EXCLUSIVE_ARGS	3
 
 static char *predef_args[] = {
 #ifdef BR_CCACHE
@@ -163,47 +157,6 @@ static void check_unsafe_path(const char *arg,
 	}
 }
 
-/* Read SOURCE_DATE_EPOCH from environment to have a deterministic
- * timestamp to replace embedded current dates to get reproducible
- * results.  Returns -1 if SOURCE_DATE_EPOCH is not defined.
- */
-static time_t get_source_date_epoch()
-{
-	char *source_date_epoch;
-	long long epoch;
-	char *endptr;
-
-	source_date_epoch = getenv("SOURCE_DATE_EPOCH");
-	if (!source_date_epoch)
-		return (time_t) -1;
-
-	errno = 0;
-	epoch = strtoll(source_date_epoch, &endptr, 10);
-	if ((errno == ERANGE && (epoch == LLONG_MAX || epoch == LLONG_MIN))
-			|| (errno != 0 && epoch == 0)) {
-		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
-				"strtoll: %s\n", strerror(errno));
-		exit(2);
-	}
-	if (endptr == source_date_epoch) {
-		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
-				"no digits were found: %s\n", endptr);
-		exit(2);
-	}
-	if (*endptr != '\0') {
-		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
-				"trailing garbage: %s\n", endptr);
-		exit(2);
-	}
-	if (epoch < 0) {
-		fprintf(stderr, "environment variable $SOURCE_DATE_EPOCH: "
-				"value must be nonnegative: %lld \n", epoch);
-		exit(2);
-	}
-
-	return (time_t) epoch;
-}
-
 int main(int argc, char **argv)
 {
 	char **args, **cur, **exec_args;
@@ -214,7 +167,6 @@ int main(int argc, char **argv)
 	char *paranoid_wrapper;
 	int paranoid;
 	int ret, i, count = 0, debug;
-	time_t source_date_epoch;
 
 	/* Calculate the relative paths */
 	basename = strrchr(progpath, '/');
@@ -320,28 +272,6 @@ int main(int argc, char **argv)
 	}
 #endif /* ARCH || CPU */
 
-	source_date_epoch = get_source_date_epoch();
-	if (source_date_epoch != -1) {
-		struct tm *tm = localtime(&source_date_epoch);
-		if (!tm) {
-			perror("__FILE__: localtime");
-			return 3;
-		}
-		ret = strftime(source_time, sizeof(source_time), "-D__TIME__=\"%T\"", tm);
-		if (!ret) {
-			perror("__FILE__: overflow");
-			return 3;
-		}
-		*cur++ = source_time;
-		ret = strftime(source_date, sizeof(source_date), "-D__DATE__=\"%b %e %Y\"", tm);
-		if (!ret) {
-			perror("__FILE__: overflow");
-			return 3;
-		}
-		*cur++ = source_date;
-		*cur++ = "-Wno-builtin-macro-redefined";
-	}
-
 	paranoid_wrapper = getenv("BR_COMPILER_PARANOID_UNSAFE_PATH");
 	if (paranoid_wrapper && strlen(paranoid_wrapper) > 0)
 		paranoid = 1;
-- 
2.7.4

[Buildroot] [PATCH 1/2 next] Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

[Thomas Petazzoni]

Hello,

On Tue, 21 Feb 2017 21:32:35 +0100, Yann E. MORIN wrote:
> This reverts commit 76838f63412a30a358210e457dda4b79f7730624.
> 
> The commit referenced above explicitly states that the function was
> copied as-is from the gcc source code at the time. And indeed that is
> exactly how the function appeared in gcc in commit
> e3e8c48c4a494d9da741c1c8ea6c4c0b7c4ff934.
> 
> However, our toolchain wrapper is "GPLv2 only", while the file this function
> was copied from is "GPLv3 or later".  As such we can't include that function
> and still comply to both licenses.
> 
> Furthermore, the code is far from optimum.
> 
> Since this feature is not release-critical, revert it until we re-implement
> it.
> 
> Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
> Cc: J?r?me Pouiller <jezz@sysmic.org>
> Cc: Samuel Martin <s.martin49@gmail.com>
> Cc: Peter Korsgaard <peter@korsgaard.com>
> Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
> Cc: Arnout Vandecappelle <arnout@mind.be>
> Signed-off-by: Peter Korsgaard <peter@korsgaard.com>

This has already been applied in master:

  https://git.buildroot.org/buildroot/commit/?id=1c8c0d91a431fda2c052072af87e0de35bd3b953

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 1/2 next] Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

[Yann E. MORIN]

Thomas, All,

On 2017-02-21 21:47 +0100, Thomas Petazzoni spake thusly:
> On Tue, 21 Feb 2017 21:32:35 +0100, Yann E. MORIN wrote:
> > This reverts commit 76838f63412a30a358210e457dda4b79f7730624.
[--SNIP--]
> This has already been applied in master:
>   https://git.buildroot.org/buildroot/commit/?id=1c8c0d91a431fda2c052072af87e0de35bd3b953
I know.

And as the cover letter stated (and as the 'tag' in the subject tells),
this is intended for next. ;-)

It is needed to revert this patch in next before applying the following
patch.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] [PATCH 1/2 next] Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

[Thomas Petazzoni]

Hello,

On Tue, 21 Feb 2017 21:54:35 +0100, Yann E. MORIN wrote:

> It is needed to revert this patch in next before applying the following
> patch.

Since it has already been reverted in master, I believe we will wait
for the release, and next to be merged in master before applying.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 1/2 next] Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

[Arnout Vandecappelle]

On 21-02-17 22:01, Thomas Petazzoni wrote:
> Hello,
> 
> On Tue, 21 Feb 2017 21:54:35 +0100, Yann E. MORIN wrote:
> 
>> It is needed to revert this patch in next before applying the following
>> patch.
> 
> Since it has already been reverted in master, I believe we will wait
> for the release, and next to be merged in master before applying.

 Alternatively, merge master into next now and apply patch 2 to next.

 Is there any reason not to regularly merge master into next?

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 1/2 next] Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

[Peter Korsgaard]

>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:

 > On 21-02-17 22:01, Thomas Petazzoni wrote:
 >> Hello,
 >> 
 >> On Tue, 21 Feb 2017 21:54:35 +0100, Yann E. MORIN wrote:
 >> 
 >>> It is needed to revert this patch in next before applying the following
 >>> patch.
 >> 
 >> Since it has already been reverted in master, I believe we will wait
 >> for the release, and next to be merged in master before applying.

 >  Alternatively, merge master into next now and apply patch 2 to next.

 >  Is there any reason not to regularly merge master into next?

I have no problem doing that if it is useful, but I haven't personally
tried that kind of back/forward merges before. Don't we then end up with
a number of merge commits on next? What happens when next is then merged
back to master?

-- 
Bye, Peter Korsgaard

[Buildroot] [PATCH 1/2 next] Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

[Arnout Vandecappelle]

On 22-02-17 14:23, Peter Korsgaard wrote:
>>>>>> "Arnout" == Arnout Vandecappelle <arnout@mind.be> writes:
> 
>  > On 21-02-17 22:01, Thomas Petazzoni wrote:
>  >> Hello,
>  >> 
>  >> On Tue, 21 Feb 2017 21:54:35 +0100, Yann E. MORIN wrote:
>  >> 
>  >>> It is needed to revert this patch in next before applying the following
>  >>> patch.
>  >> 
>  >> Since it has already been reverted in master, I believe we will wait
>  >> for the release, and next to be merged in master before applying.
> 
>  >  Alternatively, merge master into next now and apply patch 2 to next.
> 
>  >  Is there any reason not to regularly merge master into next?
> 
> I have no problem doing that if it is useful, but I haven't personally
> tried that kind of back/forward merges before. Don't we then end up with
> a number of merge commits on next? 

 You indeed get merge commits on next, but is that a problem? With more merge
commits, it's IMHO easier to understand the history because each merge makes
things line up in 'git log --graph'.

> What happens when next is then merged
> back to master?

 It shouldn't make much of a difference. Except that you probably get less merge
conflicts.

 Regards,
 Arnout

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH 1/2 next] Revert "reproducible: fix DATE/TIME macros in toolchain-wrapper"

[Thomas Petazzoni]

Hello,

On Wed, 22 Feb 2017 12:50:36 +0100, Arnout Vandecappelle wrote:

>  Is there any reason not to regularly merge master into next?

Because there's usually not reason to do so, and this patch is also not
a very good reason to do so. There's still plenty of things to fix in
master, and master is really the focus. The release is only in 6 days,
so it's not a lot of time to wait to have next merged back into master
and the new development cycle opened.

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 2/2 next] toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc

[Yann E. MORIN]

Starting with verison 7, gcc will automatically recognise and enforce
the environment variable SOURCE_DATE_EPOCH, and will fake __DATE__ and
__TIME__ accordingly, to produce reproducible builds (at least in
regards to date and time).

However, older gcc versions (i.e. all so far) do not offer this feature.

So, we use our toolchain wrapper to force-feed __DATE__ and __TIME_ as
macros, that will take precedence over those that gcc may compute
itself. We compute them according to the specs:
    https://reproducible-builds.org/specs/source-date-epoch/
    https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html

Since we define macros otherwise internal to gcc, we have to tell it not
to warn about that. The -Wno-builtin-macro-redefined flag was introduced
in gcc-4.4.0, which is old enough now that we can consider that all
toolchains we care about will support that flag.

Note however: the behaviour of gcc-7 when __DATE__ and __TIME__ are
user-defined and SOURCE_DATE_EPOCH is also set is as yet unknown.
Whether the user-defined macros take precedence over the internal
computations in gcc, or whether gcc ignores the user-defined macros and
enforce them from the value of SOURCE_DATE_EPOCH is of no consequence:
that would yield the exact same end result, to use the date and time set
in SOURCE_DATE_EPOCH.

Signed-off-by: "Yann E. MORIN" <yann.morin.1998@free.fr>
Cc: J?r?me Pouiller <jezz@sysmic.org>
Cc: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: Arnout Vandecappelle <arnout@mind.be>
Cc: Peter Korsgaard <peter@korsgaard.com>
---
 toolchain/toolchain-wrapper.c | 75 +++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 73 insertions(+), 2 deletions(-)

diff --git a/toolchain/toolchain-wrapper.c b/toolchain/toolchain-wrapper.c
index 100aa18..a7f01e7 100644
--- a/toolchain/toolchain-wrapper.c
+++ b/toolchain/toolchain-wrapper.c
@@ -22,12 +22,19 @@
 #include <unistd.h>
 #include <stdlib.h>
 #include <errno.h>
+#include <time.h>
+#include <stdbool.h>
 
 #ifdef BR_CCACHE
 static char ccache_path[PATH_MAX];
 #endif
 static char path[PATH_MAX];
 static char sysroot[PATH_MAX];
+/* As would be defined by gcc:
+ *   https://gcc.gnu.org/onlinedocs/cpp/Standard-Predefined-Macros.html
+ * sizeof() on string literals includes the terminating \0. */
+static char _time_[sizeof("-D__TIME__=\"HH:MM:SS\"")];
+static char _date_[sizeof("-D__DATE__=\"MMM DD YYYY\"")];
 
 /**
  * GCC errors out with certain combinations of arguments (examples are
@@ -35,12 +42,15 @@ static char sysroot[PATH_MAX];
  * that we only pass the predefined one to the real compiler if the inverse
  * option isn't in the argument list.
  * This specifies the worst case number of extra arguments we might pass
- * Currently, we have:
+ * Currently, we may have:
  * 	-mfloat-abi=
  * 	-march=
  * 	-mcpu=
+ * 	-D__TIME__=
+ * 	-D__DATE__=
+ * 	-Wno-builtin-macro-redefined
  */
-#define EXCLUSIVE_ARGS	3
+#define EXCLUSIVE_ARGS	6
 
 static char *predef_args[] = {
 #ifdef BR_CCACHE
@@ -157,6 +167,60 @@ static void check_unsafe_path(const char *arg,
 	}
 }
 
+/* Returns false if SOURCE_DATE_EPOCH was not defined in the environment.
+ *
+ * Returns true if SOURCE_DATE_EPOCH is in the environment and represent
+ * a valid timestamp, in which case the timestamp is formatted into the
+ * global variables _source_ and _date_.
+ *
+ * Aborts if SOURCE_DATE_EPOCH was set in the environment but did not
+ * contain a valid timestamp.
+ *
+ * Valid values are defined in the spec:
+ *     https://reproducible-builds.org/specs/source-date-epoch/
+ * but we further restrict them to be positive or null.
+ */
+bool parse_source_date_epoch_from_env(void)
+{
+	char *epoch_env, *endptr;
+	time_t epoch;
+	struct tm epoch_tm;
+
+	if ((epoch_env = getenv("SOURCE_DATE_EPOCH")) == NULL)
+		return false;
+	errno = 0;
+	epoch = (time_t) strtoll(epoch_env, &endptr, 10);
+	/* We just need to test if it is incorrect, but we do not
+	 * care why it is incorrect.
+	 */
+	if ((errno != 0) || !*epoch_env || *endptr || (epoch < 0)) {
+		fprintf(stderr, "%s: invalid SOURCE_DATE_EPOCH='%s'\n",
+			program_invocation_short_name,
+			epoch_env);
+		exit(1);
+	}
+	tzset(); /* For localtime_r(), below. */
+	if (localtime_r(&epoch, &epoch_tm) == NULL) {
+		fprintf(stderr, "%s: cannot parse SOURCE_DATE_EPOCH=%s\n",
+				program_invocation_short_name,
+				getenv("SOURCE_DATE_EPOCH"));
+		exit(1);
+	}
+	if (!strftime(_time_, sizeof(_time_), "-D__TIME__=\"%T\"", &epoch_tm)) {
+		fprintf(stderr, "%s: cannot set time from SOURCE_DATE_EPOCH=%s\n",
+				program_invocation_short_name,
+				getenv("SOURCE_DATE_EPOCH"));
+		exit(1);
+	}
+	if (!strftime(_date_, sizeof(_date_), "-D__DATE__=\"%b %e %Y\"", &epoch_tm)) {
+		fprintf(stderr, "%s: cannot set date from SOURCE_DATE_EPOCH=%s\n",
+				program_invocation_short_name,
+				getenv("SOURCE_DATE_EPOCH"));
+		exit(1);
+	}
+	return true;
+}
+
 int main(int argc, char **argv)
 {
 	char **args, **cur, **exec_args;
@@ -272,6 +336,13 @@ int main(int argc, char **argv)
 	}
 #endif /* ARCH || CPU */
 
+	if (parse_source_date_epoch_from_env()) {
+		*cur++ = _time_;
+		*cur++ = _date_;
+		/* This has existed since gcc-4.4.0 */
+		*cur++ = "-Wno-builtin-macro-redefined";
+	}
+
 	paranoid_wrapper = getenv("BR_COMPILER_PARANOID_UNSAFE_PATH");
 	if (paranoid_wrapper && strlen(paranoid_wrapper) > 0)
 		paranoid = 1;
-- 
2.7.4

[Buildroot] [PATCH 2/2 next] toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc

[Thomas Petazzoni]

Hello,

On Tue, 21 Feb 2017 21:32:36 +0100, Yann E. MORIN wrote:
> +/* Returns false if SOURCE_DATE_EPOCH was not defined in the environment.
> + *
> + * Returns true if SOURCE_DATE_EPOCH is in the environment and represent
> + * a valid timestamp, in which case the timestamp is formatted into the
> + * global variables _source_ and _date_.

Shouldn't this be:

	"global variables _date_ and _time_"

instead ?

If you confirm, then I can fixup this when applying.

Thanks!

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH 2/2 next] toolchain/wrapper: fake __DATE_ and __TIME__ for older gcc

[Yann E. MORIN]

Thomas, All,

On 2017-03-01 23:12 +0100, Thomas Petazzoni spake thusly:
> On Tue, 21 Feb 2017 21:32:36 +0100, Yann E. MORIN wrote:
> > +/* Returns false if SOURCE_DATE_EPOCH was not defined in the environment.
> > + *
> > + * Returns true if SOURCE_DATE_EPOCH is in the environment and represent
> > + * a valid timestamp, in which case the timestamp is formatted into the
> > + * global variables _source_ and _date_.
> 
> Shouldn't this be:
> 
> 	"global variables _date_ and _time_"
> 
> instead ?
> 
> If you confirm, then I can fixup this when applying.

Yes, obviously it's _time_ and _date_.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'