[PATCH 0/3] reproducible builds

[PATCH 0/3] reproducible builds

[Alexander Couzens]

Hi,

I would like to build grub reproducible for coreboot.
The mkrescue patch is not needed for reproducible builds
in coreboot and might dropped, because someone want to create a direct
override argument to set the uuid reproducible.

Best,
lynxis

Alexander Couzens (3):
  mkstandalone: add argument --fixed-time to override mtime of files
  mkrescue: add argument --fixed-time to get reproducible uuids
  Makefile: use FIXED_TIMESTAMP for mkstandalone if set

 Makefile.am              |  2 +-
 util/grub-mkrescue.c     | 15 ++++++++++++++-
 util/grub-mkstandalone.c | 14 +++++++++++++-
 3 files changed, 28 insertions(+), 3 deletions(-)

-- 
2.6.3

[PATCH 1/3] mkstandalone: add argument --fixed-time to override mtime of files

[Alexander Couzens]

mkstandalone adds several files to an archive. Doing this it uses the
mtime to give these files a timestamp.
--fixed-time <TIME_EPOCH> overrides these timestamps with a given.

Replacing all timestamps with a specific one is required
to get reproducible builds. See source epoch specification of
reproducible-builds.org
---
 util/grub-mkstandalone.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/util/grub-mkstandalone.c b/util/grub-mkstandalone.c
index 4907d44..047f0cd 100644
--- a/util/grub-mkstandalone.c
+++ b/util/grub-mkstandalone.c
@@ -30,6 +30,7 @@
 #pragma GCC diagnostic error "-Wmissing-prototypes"
 #pragma GCC diagnostic error "-Wmissing-declarations"
 
+static time_t fixed_time;
 static char *output_image;
 static char **files;
 static int nfiles;
@@ -48,6 +49,7 @@ static struct argp_option options[] = {
    0, N_("save output in FILE [required]"), 2},
   {"format", 'O', N_("FILE"), 0, 0, 2},
   {"compression", 'C', "xz|none|auto", OPTION_HIDDEN, 0, 2},
+  {"fixed-time", 't', N_("TIMEEPOCH"), 0, N_("Use a fixed timestamp to override mtime of all files. Time since epoch is used."), 2},
   {0, 0, 0, 0, 0, 0}
 };
 
@@ -72,6 +74,7 @@ help_filter (int key, const char *text, void *input __attribute__ ((unused)))
 static error_t
 argp_parser (int key, char *arg, struct argp_state *state)
 {
+  char *b;
   if (key == 'C')
     key = GRUB_INSTALL_OPTIONS_INSTALL_CORE_COMPRESS;
 
@@ -80,6 +83,14 @@ argp_parser (int key, char *arg, struct argp_state *state)
 
   switch (key)
     {
+    case 't':
+      fixed_time = strtoll (arg, &b, 10);
+      if (*b !='\0') {
+        printf (_("invalid fixed time number: %s\n"), arg);
+        argp_usage (state);
+        exit (1);
+      }
+      break;
 
     case 'o':
       if (output_image)
@@ -192,7 +203,8 @@ add_tar_file (const char *from,
   if (grub_util_is_special_file (from))
     return;
 
-  mtime = grub_util_get_mtime (from);
+  /* use fixed_time if given for mtime */
+  mtime = fixed_time != -1 ? fixed_time : grub_util_get_mtime (from);
 
   optr = tcn = xmalloc (strlen (to) + 1);
   for (iptr = to; *iptr == '/'; iptr++);
-- 
2.6.3

Re: [PATCH 1/3] mkstandalone: add argument --fixed-time to override mtime of files

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

On 04.12.2015 17:10, Alexander Couzens wrote:
> mkstandalone adds several files to an archive. Doing this it uses the
> mtime to give these files a timestamp.
> --fixed-time <TIME_EPOCH> overrides these timestamps with a given.
> 
> Replacing all timestamps with a specific one is required
> to get reproducible builds. See source epoch specification of
> reproducible-builds.org
Patch in general looks good. I'm unsure about which way the timestamp
should be passed and parsed. I see 3 solutions:
1) Argument and use some standard function to parse date supply argument
+<value>
2) Essentially what you have done. It feels a bit ugly but not too much
3) Read directly from variable.
WDYT?
> +  {"fixed-time", 't', N_("TIMEEPOCH"), 0, N_("Use a fixed timestamp to override mtime of all files. Time since epoch is used."), 2},
It's not worth spending a letter on this. Please keep only long version.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

Re: [PATCH 1/3] mkstandalone: add argument --fixed-time to override mtime of files

[Andrei Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 1414 bytes --]

04.12.2015 21:00, Vladimir 'φ-coder/phcoder' Serbinenko пишет:
> On 04.12.2015 17:10, Alexander Couzens wrote:
>> mkstandalone adds several files to an archive. Doing this it uses the
>> mtime to give these files a timestamp.
>> --fixed-time <TIME_EPOCH> overrides these timestamps with a given.
>>
>> Replacing all timestamps with a specific one is required
>> to get reproducible builds. See source epoch specification of
>> reproducible-builds.org
> Patch in general looks good. I'm unsure about which way the timestamp
> should be passed and parsed. I see 3 solutions:
> 1) Argument and use some standard function to parse date supply argument
> +<value>

As long as it is user-facing option, more human oriented timestamp
parsing would be nice. One option would be strptime() with some
predefined formats or may be even getdate() which allows user to define
own input formats.

> 2) Essentially what you have done. It feels a bit ugly but not too much
> 3) Read directly from variable.
> WDYT?
>> +  {"fixed-time", 't', N_("TIMEEPOCH"), 0, N_("Use a fixed timestamp to override mtime of all files. Time since epoch is used."), 2},
> It's not worth spending a letter on this. Please keep only long version.
> 
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

[PATCH 2/3] mkrescue: add argument --fixed-time to get reproducible uuids

[Alexander Couzens]

The uuid generation is based on the time.
---
 util/grub-mkrescue.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/util/grub-mkrescue.c b/util/grub-mkrescue.c
index 4511826..164c4e1 100644
--- a/util/grub-mkrescue.c
+++ b/util/grub-mkrescue.c
@@ -52,6 +52,7 @@ static int xorriso_arg_alloc;
 static char **xorriso_argv;
 static char *iso_uuid;
 static char *iso9660_dir;
+static time_t fixed_time;
 
 static void
 xorriso_push (const char *val)
@@ -110,6 +111,7 @@ static struct argp_option options[] = {
   {"product-version", OPTION_PRODUCT_VERSION, N_("STRING"), 0, N_("use STRING as product version"), 2},
   {"sparc-boot", OPTION_SPARC_BOOT, 0, 0, N_("enable sparc boot. Disables HFS+, APM, ARCS and boot as disk image for i386-pc"), 2},
   {"arcs-boot", OPTION_ARCS_BOOT, 0, 0, N_("enable ARCS (big-endian mips machines, mostly SGI) boot. Disables HFS+, APM, sparc64 and boot as disk image for i386-pc"), 2},
+  {"fixed-time", 't', N_("TIMEEPOCH"), 0, N_("use a fixed timestamp for uuid generation"), 2},
   {0, 0, 0, 0, 0, 0}
 };
 
@@ -153,6 +155,8 @@ enum {
 static error_t 
 argp_parser (int key, char *arg, struct argp_state *state)
 {
+  char *b;
+
   if (grub_install_parse (key, arg))
     return 0;
   switch (key)
@@ -212,6 +216,15 @@ argp_parser (int key, char *arg, struct argp_state *state)
       xorriso = xstrdup (arg);
       return 0;
 
+    case 't':
+      fixed_time = strtoll (arg, &b, 10);
+      if (*b !='\0') {
+        printf (_("invalid fixed time number: %s\n"), arg);
+        argp_usage (state);
+        exit (1);
+      }
+      return 0;
+
     default:
       return ARGP_ERR_UNKNOWN;
     }
@@ -541,7 +554,7 @@ main (int argc, char *argv[])
   {
     time_t tim;
     struct tm *tmm;
-    tim = time (NULL);
+    tim = fixed_time != -1 ? fixed_time : time (NULL);
     tmm = gmtime (&tim);
     iso_uuid = xmalloc (55);
     grub_snprintf (iso_uuid, 50,
-- 
2.6.3

[PATCH 3/3] Makefile: use FIXED_TIMESTAMP for mkstandalone if set

[Alexander Couzens]

mkstandalone sets timestamps for files which can be overriden by a fixed_timestamp.
This makes it possible to build reproducible builds for coreboot.

To build a reproducible build of grub for coreboot do:
make default_payload.elf FIXED_TIMESTAMP=1134242
---
 Makefile.am | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile.am b/Makefile.am
index 994ebbd..37a7cc4 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -403,7 +403,7 @@ bootcheck: $(BOOTCHECKS)
 
 if COND_i386_coreboot
 default_payload.elf: grub-mkstandalone grub-mkimage
-	pkgdatadir=. ./grub-mkstandalone --grub-mkimage=./grub-mkimage -O i386-coreboot -o $@ --modules='ahci pata ehci uhci ohci usb_keyboard usbms part_msdos xfs ext2 fat at_keyboard part_gpt usbserial_usbdebug cbfs' --install-modules='ls linux search configfile normal cbtime cbls memrw iorw minicmd lsmmap lspci halt reboot hexdump pcidump regexp setpci lsacpi chain test serial multiboot cbmemc linux16 gzio echo help' --fonts= --themes= --locales= -d grub-core/ /boot/grub/grub.cfg=$(srcdir)/coreboot.cfg
+	pkgdatadir=. ./grub-mkstandalone --grub-mkimage=./grub-mkimage -O i386-coreboot -o $@ --modules='ahci pata ehci uhci ohci usb_keyboard usbms part_msdos xfs ext2 fat at_keyboard part_gpt usbserial_usbdebug cbfs' --install-modules='ls linux search configfile normal cbtime cbls memrw iorw minicmd lsmmap lspci halt reboot hexdump pcidump regexp setpci lsacpi chain test serial multiboot cbmemc linux16 gzio echo help' --fonts= --themes= --locales= -d grub-core/ /boot/grub/grub.cfg=$(srcdir)/coreboot.cfg $(if $(FIXED_TIMESTAMP),-t $(FIXED_TIMESTAMP))
 endif
 
 endif
-- 
2.6.3

Re: [PATCH 3/3] Makefile: use FIXED_TIMESTAMP for mkstandalone if set

[Alexander Couzens]

[-- Attachment #1: Type: text/plain, Size: 402 bytes --]

On Fri,  4 Dec 2015 17:10:44 +0100
Alexander Couzens <lynxis@fe80.eu> wrote:

> mkstandalone sets timestamps for files which can be overriden by a
> fixed_timestamp. This makes it possible to build reproducible builds
> for coreboot.
> 
> To build a reproducible build of grub for coreboot do:
> make default_payload.elf FIXED_TIMESTAMP=1134242

Please ignore this patch.

I'll send a v2.

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[PATCH] Makefile/coreboot use SOURCE_DATE_EPOCH as time source if set

[Alexander Couzens]

mkstandalone sets timestamps for files which can be overriden by a fixed_timestamp.
This makes it possible to build reproducible builds for coreboot.

To build a reproducible build of grub for coreboot do:
export SOURCE_DATE_EPOCH=1134242
make default_payload.elf
---
 Makefile.am | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile.am b/Makefile.am
index 994ebbd..5c756d7 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -403,7 +403,7 @@ bootcheck: $(BOOTCHECKS)
 
 if COND_i386_coreboot
 default_payload.elf: grub-mkstandalone grub-mkimage
-	pkgdatadir=. ./grub-mkstandalone --grub-mkimage=./grub-mkimage -O i386-coreboot -o $@ --modules='ahci pata ehci uhci ohci usb_keyboard usbms part_msdos xfs ext2 fat at_keyboard part_gpt usbserial_usbdebug cbfs' --install-modules='ls linux search configfile normal cbtime cbls memrw iorw minicmd lsmmap lspci halt reboot hexdump pcidump regexp setpci lsacpi chain test serial multiboot cbmemc linux16 gzio echo help' --fonts= --themes= --locales= -d grub-core/ /boot/grub/grub.cfg=$(srcdir)/coreboot.cfg
+	pkgdatadir=. ./grub-mkstandalone --grub-mkimage=./grub-mkimage -O i386-coreboot -o $@ --modules='ahci pata ehci uhci ohci usb_keyboard usbms part_msdos xfs ext2 fat at_keyboard part_gpt usbserial_usbdebug cbfs' --install-modules='ls linux search configfile normal cbtime cbls memrw iorw minicmd lsmmap lspci halt reboot hexdump pcidump regexp setpci lsacpi chain test serial multiboot cbmemc linux16 gzio echo help' --fonts= --themes= --locales= -d grub-core/ /boot/grub/grub.cfg=$(srcdir)/coreboot.cfg $(if $(SOURCE_DATE_EPOCH),-t $(SOURCE_DATE_EPOCH))
 endif
 
 endif
-- 
2.6.3

Re: [PATCH 3/3] Makefile: use FIXED_TIMESTAMP for mkstandalone if set

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 426 bytes --]

On 04.12.2015 17:10, Alexander Couzens wrote:
> mkstandalone sets timestamps for files which can be overriden by a fixed_timestamp.
> This makes it possible to build reproducible builds for coreboot.
> 
> To build a reproducible build of grub for coreboot do:
> make default_payload.elf FIXED_TIMESTAMP=1134242
Why FIXED_TIMESTAMP and not SOURCE_DATE_EPOCH ?
https://reproducible-builds.org/specs/source-date-epoch/


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

[PATCH v3 0/3] reproducible builds

[Alexander Couzens]

Changelog:

- v2:
 - 3/3: rewrite commit message
 - 3/3: rename variable into SOURCE_DATE_EPOCH
- v3:
 - */3: initialize fixed_time argument
 - 1+2/3: remove argument letter '-t' only --fixed-time works
 - 1+2/3: initialize fixed_time = -1

Alexander Couzens (3):
  mkstandalone: add argument --fixed-time to override mtime of files
  mkrescue: add argument --fixed-time to get reproducible uuids
  Makefile/coreboot use SOURCE_DATE_EPOCH as time source if set

 Makefile.am              |  2 +-
 util/grub-mkrescue.c     | 16 +++++++++++++++-
 util/grub-mkstandalone.c | 15 ++++++++++++++-
 3 files changed, 30 insertions(+), 3 deletions(-)

-- 
2.6.3

Re: [PATCH v3 0/3] reproducible builds

[Andrei Borzenkov]

04.12.2015 21:32, Alexander Couzens пишет:
> Changelog:
> 
> - v2:
>  - 3/3: rewrite commit message
>  - 3/3: rename variable into SOURCE_DATE_EPOCH
> - v3:
>  - */3: initialize fixed_time argument
>  - 1+2/3: remove argument letter '-t' only --fixed-time works
>  - 1+2/3: initialize fixed_time = -1
> 
> Alexander Couzens (3):
>   mkstandalone: add argument --fixed-time to override mtime of files
>   mkrescue: add argument --fixed-time to get reproducible uuids
>   Makefile/coreboot use SOURCE_DATE_EPOCH as time source if set
> 

Should not we convert 85a7be2414c4718e96d81a2ebaa70d0d42152e62 to use it
then instead of hardcoding time stamp? Then --fixed-time should probably
be promoted to common installer option.

>  Makefile.am              |  2 +-
>  util/grub-mkrescue.c     | 16 +++++++++++++++-
>  util/grub-mkstandalone.c | 15 ++++++++++++++-
>  3 files changed, 30 insertions(+), 3 deletions(-)
> 

Re: [PATCH v3 0/3] reproducible builds

[Alexander Couzens]

[-- Attachment #1: Type: text/plain, Size: 381 bytes --]

> Should not we convert 85a7be2414c4718e96d81a2ebaa70d0d42152e62 to use
> it then instead of hardcoding time stamp? Then --fixed-time should
> probably be promoted to common installer option.

Sounds good.

Best,
lynxis
-- 
Alexander Couzens

mail: lynxis@fe80.eu
jabber: lynxis@fe80.eu
mobile: +4915123277221
gpg: 390D CF78 8BF9 AA50 4F8F  F1E2 C29E 9DA6 A0DF 8604

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

[PATCH v3 1/3] mkstandalone: add argument --fixed-time to override mtime of files

[Alexander Couzens]

mkstandalone adds several files to an archive. Doing this it uses the
mtime to give these files a timestamp.
--fixed-time <TIME_EPOCH> overrides these timestamps with a given.

Replacing all timestamps with a specific one is required
to get reproducible builds. See source epoch specification of
reproducible-builds.org
---
 util/grub-mkstandalone.c | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/util/grub-mkstandalone.c b/util/grub-mkstandalone.c
index 4907d44..779c13c 100644
--- a/util/grub-mkstandalone.c
+++ b/util/grub-mkstandalone.c
@@ -30,6 +30,7 @@
 #pragma GCC diagnostic error "-Wmissing-prototypes"
 #pragma GCC diagnostic error "-Wmissing-declarations"
 
+static time_t fixed_time;
 static char *output_image;
 static char **files;
 static int nfiles;
@@ -48,6 +49,7 @@ static struct argp_option options[] = {
    0, N_("save output in FILE [required]"), 2},
   {"format", 'O', N_("FILE"), 0, 0, 2},
   {"compression", 'C', "xz|none|auto", OPTION_HIDDEN, 0, 2},
+  {"fixed-time", 0, N_("TIMEEPOCH"), 0, N_("Use a fixed timestamp to override mtime of all files. Time since epoch is used."), 2},
   {0, 0, 0, 0, 0, 0}
 };
 
@@ -72,6 +74,7 @@ help_filter (int key, const char *text, void *input __attribute__ ((unused)))
 static error_t
 argp_parser (int key, char *arg, struct argp_state *state)
 {
+  char *b;
   if (key == 'C')
     key = GRUB_INSTALL_OPTIONS_INSTALL_CORE_COMPRESS;
 
@@ -80,6 +83,14 @@ argp_parser (int key, char *arg, struct argp_state *state)
 
   switch (key)
     {
+    case 't':
+      fixed_time = strtoll (arg, &b, 10);
+      if (*b !='\0') {
+        printf (_("invalid fixed time number: %s\n"), arg);
+        argp_usage (state);
+        exit (1);
+      }
+      break;
 
     case 'o':
       if (output_image)
@@ -192,7 +203,8 @@ add_tar_file (const char *from,
   if (grub_util_is_special_file (from))
     return;
 
-  mtime = grub_util_get_mtime (from);
+  /* use fixed_time if given for mtime */
+  mtime = fixed_time != -1 ? fixed_time : grub_util_get_mtime (from);
 
   optr = tcn = xmalloc (strlen (to) + 1);
   for (iptr = to; *iptr == '/'; iptr++);
@@ -293,6 +305,7 @@ main (int argc, char *argv[])
   const char *pkglibdir;
   int i;
 
+  fixed_time = -1;
   grub_util_host_init (&argc, &argv);
   grub_util_disable_fd_syncs ();
 
-- 
2.6.3

Re: [PATCH v3 1/3] mkstandalone: add argument --fixed-time to override mtime of files

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 118 bytes --]

> +  fixed_time = -1;
-1 is actually perfectly valid. Can we have a second boolean to avoid
special-casing -1?



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

[PATCH v3 2/3] mkrescue: add argument --fixed-time to get reproducible uuids

[Alexander Couzens]

The uuid generation is based on the time.
---
 util/grub-mkrescue.c | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/util/grub-mkrescue.c b/util/grub-mkrescue.c
index 4511826..1af1da2 100644
--- a/util/grub-mkrescue.c
+++ b/util/grub-mkrescue.c
@@ -52,6 +52,7 @@ static int xorriso_arg_alloc;
 static char **xorriso_argv;
 static char *iso_uuid;
 static char *iso9660_dir;
+static time_t fixed_time;
 
 static void
 xorriso_push (const char *val)
@@ -110,6 +111,7 @@ static struct argp_option options[] = {
   {"product-version", OPTION_PRODUCT_VERSION, N_("STRING"), 0, N_("use STRING as product version"), 2},
   {"sparc-boot", OPTION_SPARC_BOOT, 0, 0, N_("enable sparc boot. Disables HFS+, APM, ARCS and boot as disk image for i386-pc"), 2},
   {"arcs-boot", OPTION_ARCS_BOOT, 0, 0, N_("enable ARCS (big-endian mips machines, mostly SGI) boot. Disables HFS+, APM, sparc64 and boot as disk image for i386-pc"), 2},
+  {"fixed-time", 0, N_("TIMEEPOCH"), 0, N_("use a fixed timestamp for uuid generation"), 2},
   {0, 0, 0, 0, 0, 0}
 };
 
@@ -153,6 +155,8 @@ enum {
 static error_t 
 argp_parser (int key, char *arg, struct argp_state *state)
 {
+  char *b;
+
   if (grub_install_parse (key, arg))
     return 0;
   switch (key)
@@ -212,6 +216,15 @@ argp_parser (int key, char *arg, struct argp_state *state)
       xorriso = xstrdup (arg);
       return 0;
 
+    case 't':
+      fixed_time = strtoll (arg, &b, 10);
+      if (*b !='\0') {
+        printf (_("invalid fixed time number: %s\n"), arg);
+        argp_usage (state);
+        exit (1);
+      }
+      return 0;
+
     default:
       return ARGP_ERR_UNKNOWN;
     }
@@ -431,6 +444,7 @@ main (int argc, char *argv[])
 
   pkgdatadir = grub_util_get_pkgdatadir ();
 
+  fixed_time = -1;
   product_name = xstrdup (PACKAGE_NAME);
   product_version = xstrdup (PACKAGE_VERSION);
   xorriso = xstrdup ("xorriso");
@@ -541,7 +555,7 @@ main (int argc, char *argv[])
   {
     time_t tim;
     struct tm *tmm;
-    tim = time (NULL);
+    tim = fixed_time != -1 ? fixed_time : time (NULL);
     tmm = gmtime (&tim);
     iso_uuid = xmalloc (55);
     grub_snprintf (iso_uuid, 50,
-- 
2.6.3

Re: [PATCH v3 2/3] mkrescue: add argument --fixed-time to get reproducible uuids

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 2762 bytes --]

On 04.12.2015 19:32, Alexander Couzens wrote:
> The uuid generation is based on the time.
> ---
>  util/grub-mkrescue.c | 16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
> 
This breaks uniqueness assumptions for UUID and we use UUID to find the
right disk, as it's not possible to rely on passed boot disk on some
platforms (I've just documented it in grub.texi and pushed it). Also for
mkrescue we always use UUID. We need to find a way to reliably find boot
disk without depending on current time.
> diff --git a/util/grub-mkrescue.c b/util/grub-mkrescue.c
> index 4511826..1af1da2 100644
> --- a/util/grub-mkrescue.c
> +++ b/util/grub-mkrescue.c
> @@ -52,6 +52,7 @@ static int xorriso_arg_alloc;
>  static char **xorriso_argv;
>  static char *iso_uuid;
>  static char *iso9660_dir;
> +static time_t fixed_time;
>  
>  static void
>  xorriso_push (const char *val)
> @@ -110,6 +111,7 @@ static struct argp_option options[] = {
>    {"product-version", OPTION_PRODUCT_VERSION, N_("STRING"), 0, N_("use STRING as product version"), 2},
>    {"sparc-boot", OPTION_SPARC_BOOT, 0, 0, N_("enable sparc boot. Disables HFS+, APM, ARCS and boot as disk image for i386-pc"), 2},
>    {"arcs-boot", OPTION_ARCS_BOOT, 0, 0, N_("enable ARCS (big-endian mips machines, mostly SGI) boot. Disables HFS+, APM, sparc64 and boot as disk image for i386-pc"), 2},
> +  {"fixed-time", 0, N_("TIMEEPOCH"), 0, N_("use a fixed timestamp for uuid generation"), 2},
>    {0, 0, 0, 0, 0, 0}
>  };
>  
> @@ -153,6 +155,8 @@ enum {
>  static error_t 
>  argp_parser (int key, char *arg, struct argp_state *state)
>  {
> +  char *b;
> +
>    if (grub_install_parse (key, arg))
>      return 0;
>    switch (key)
> @@ -212,6 +216,15 @@ argp_parser (int key, char *arg, struct argp_state *state)
>        xorriso = xstrdup (arg);
>        return 0;
>  
> +    case 't':
> +      fixed_time = strtoll (arg, &b, 10);
> +      if (*b !='\0') {
> +        printf (_("invalid fixed time number: %s\n"), arg);
> +        argp_usage (state);
> +        exit (1);
> +      }
> +      return 0;
> +
>      default:
>        return ARGP_ERR_UNKNOWN;
>      }
> @@ -431,6 +444,7 @@ main (int argc, char *argv[])
>  
>    pkgdatadir = grub_util_get_pkgdatadir ();
>  
> +  fixed_time = -1;
>    product_name = xstrdup (PACKAGE_NAME);
>    product_version = xstrdup (PACKAGE_VERSION);
>    xorriso = xstrdup ("xorriso");
> @@ -541,7 +555,7 @@ main (int argc, char *argv[])
>    {
>      time_t tim;
>      struct tm *tmm;
> -    tim = time (NULL);
> +    tim = fixed_time != -1 ? fixed_time : time (NULL);
>      tmm = gmtime (&tim);
>      iso_uuid = xmalloc (55);
>      grub_snprintf (iso_uuid, 50,
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

Re: [PATCH v3 2/3] mkrescue: add argument --fixed-time to get reproducible uuids

[Andrei Borzenkov]

[-- Attachment #1: Type: text/plain, Size: 3591 bytes --]

14.12.2015 18:22, Vladimir 'φ-coder/phcoder' Serbinenko пишет:
> On 04.12.2015 19:32, Alexander Couzens wrote:
>> The uuid generation is based on the time.
>> ---
>>  util/grub-mkrescue.c | 16 +++++++++++++++-
>>  1 file changed, 15 insertions(+), 1 deletion(-)
>>
> This breaks uniqueness assumptions for UUID and we use UUID to find the
> right disk, as it's not possible to rely on passed boot disk on some
> platforms (I've just documented it in grub.texi and pushed it). Also for
> mkrescue we always use UUID. We need to find a way to reliably find boot
> disk without depending on current time.

Well, UUID of isofs used by GRUB is not unique in any sense, so it is
not really much worse than it was before.

Having reliable way to identify boot device imply some unique property
of boot device which automatically conflict with idea of identical images.

But I am not sure we should stretch reproducible builds that far. ISO
image created by grub-mkrescue is not binary. It even does not have well
defined content, user is free to change modules list and other files
that are part of ISO.

>> diff --git a/util/grub-mkrescue.c b/util/grub-mkrescue.c
>> index 4511826..1af1da2 100644
>> --- a/util/grub-mkrescue.c
>> +++ b/util/grub-mkrescue.c
>> @@ -52,6 +52,7 @@ static int xorriso_arg_alloc;
>>  static char **xorriso_argv;
>>  static char *iso_uuid;
>>  static char *iso9660_dir;
>> +static time_t fixed_time;
>>  
>>  static void
>>  xorriso_push (const char *val)
>> @@ -110,6 +111,7 @@ static struct argp_option options[] = {
>>    {"product-version", OPTION_PRODUCT_VERSION, N_("STRING"), 0, N_("use STRING as product version"), 2},
>>    {"sparc-boot", OPTION_SPARC_BOOT, 0, 0, N_("enable sparc boot. Disables HFS+, APM, ARCS and boot as disk image for i386-pc"), 2},
>>    {"arcs-boot", OPTION_ARCS_BOOT, 0, 0, N_("enable ARCS (big-endian mips machines, mostly SGI) boot. Disables HFS+, APM, sparc64 and boot as disk image for i386-pc"), 2},
>> +  {"fixed-time", 0, N_("TIMEEPOCH"), 0, N_("use a fixed timestamp for uuid generation"), 2},
>>    {0, 0, 0, 0, 0, 0}
>>  };
>>  
>> @@ -153,6 +155,8 @@ enum {
>>  static error_t 
>>  argp_parser (int key, char *arg, struct argp_state *state)
>>  {
>> +  char *b;
>> +
>>    if (grub_install_parse (key, arg))
>>      return 0;
>>    switch (key)
>> @@ -212,6 +216,15 @@ argp_parser (int key, char *arg, struct argp_state *state)
>>        xorriso = xstrdup (arg);
>>        return 0;
>>  
>> +    case 't':
>> +      fixed_time = strtoll (arg, &b, 10);
>> +      if (*b !='\0') {
>> +        printf (_("invalid fixed time number: %s\n"), arg);
>> +        argp_usage (state);
>> +        exit (1);
>> +      }
>> +      return 0;
>> +
>>      default:
>>        return ARGP_ERR_UNKNOWN;
>>      }
>> @@ -431,6 +444,7 @@ main (int argc, char *argv[])
>>  
>>    pkgdatadir = grub_util_get_pkgdatadir ();
>>  
>> +  fixed_time = -1;
>>    product_name = xstrdup (PACKAGE_NAME);
>>    product_version = xstrdup (PACKAGE_VERSION);
>>    xorriso = xstrdup ("xorriso");
>> @@ -541,7 +555,7 @@ main (int argc, char *argv[])
>>    {
>>      time_t tim;
>>      struct tm *tmm;
>> -    tim = time (NULL);
>> +    tim = fixed_time != -1 ? fixed_time : time (NULL);
>>      tmm = gmtime (&tim);
>>      iso_uuid = xmalloc (55);
>>      grub_snprintf (iso_uuid, 50,
>>
> 
> 
> 
> 
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 198 bytes --]

Re: [PATCH v3 2/3] mkrescue: add argument --fixed-time to get reproducible uuids

[Vladimir 'phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 4358 bytes --]

Le 15 déc. 2015 4:46 PM, "Andrei Borzenkov" <arvidjaar@gmail.com> a écrit :

>

> 14.12.2015 18:22, Vladimir 'φ-coder/phcoder' Serbinenko пишет:

> > On 04.12.2015 19:32, Alexander Couzens wrote:

> >> The uuid generation is based on the time.

> >> ---

> >>  util/grub-mkrescue.c | 16 +++++++++++++++-

> >>  1 file changed, 15 insertions(+), 1 deletion(-)

> >>

> > This breaks uniqueness assumptions for UUID and we use UUID to find the

> > right disk, as it's not possible to rely on passed boot disk on some

> > platforms (I've just documented it in grub.texi and pushed it). Also for

> > mkrescue we always use UUID. We need to find a way to reliably find boot

> > disk without depending on current time.

>

> Well, UUID of isofs used by GRUB is not unique in any sense, so it is

> not really much worse than it was before.

>

> Having reliable way to identify boot device imply some unique property

> of boot device which automatically conflict with idea of identical images.

>

It's not mutually exclusive. We could e.g. feed all files with full names
in ISO into sha512sum, then stuff it into the ISO somehow

> But I am not sure we should stretch reproducible builds that far. ISO

> image created by grub-mkrescue is not binary. It even does not have well

> defined content, user is free to change modules list and other files

> that are part of ISO.

>

> >> diff --git a/util/grub-mkrescue.c b/util/grub-mkrescue.c

> >> index 4511826..1af1da2 100644

> >> --- a/util/grub-mkrescue.c

> >> +++ b/util/grub-mkrescue.c

> >> @@ -52,6 +52,7 @@ static int xorriso_arg_alloc;

> >>  static char **xorriso_argv;

> >>  static char *iso_uuid;

> >>  static char *iso9660_dir;

> >> +static time_t fixed_time;

> >>

> >>  static void

> >>  xorriso_push (const char *val)

> >> @@ -110,6 +111,7 @@ static struct argp_option options[] = {

> >>    {"product-version", OPTION_PRODUCT_VERSION, N_("STRING"), 0,
N_("use STRING as product version"), 2},

> >>    {"sparc-boot", OPTION_SPARC_BOOT, 0, 0, N_("enable sparc boot.
Disables HFS+, APM, ARCS and boot as disk image for i386-pc"), 2},

> >>    {"arcs-boot", OPTION_ARCS_BOOT, 0, 0, N_("enable ARCS (big-endian
mips machines, mostly SGI) boot. Disables HFS+, APM, sparc64 and boot as
disk image for i386-pc"), 2},

> >> +  {"fixed-time", 0, N_("TIMEEPOCH"), 0, N_("use a fixed timestamp for
uuid generation"), 2},

> >>    {0, 0, 0, 0, 0, 0}

> >>  };

> >>

> >> @@ -153,6 +155,8 @@ enum {

> >>  static error_t

> >>  argp_parser (int key, char *arg, struct argp_state *state)

> >>  {

> >> +  char *b;

> >> +

> >>    if (grub_install_parse (key, arg))

> >>      return 0;

> >>    switch (key)

> >> @@ -212,6 +216,15 @@ argp_parser (int key, char *arg, struct
argp_state *state)

> >>        xorriso = xstrdup (arg);

> >>        return 0;

> >>

> >> +    case 't':

> >> +      fixed_time = strtoll (arg, &b, 10);

> >> +      if (*b !='\0') {

> >> +        printf (_("invalid fixed time number: %s\n"), arg);

> >> +        argp_usage (state);

> >> +        exit (1);

> >> +      }

> >> +      return 0;

> >> +

> >>      default:

> >>        return ARGP_ERR_UNKNOWN;

> >>      }

> >> @@ -431,6 +444,7 @@ main (int argc, char *argv[])

> >>

> >>    pkgdatadir = grub_util_get_pkgdatadir ();

> >>

> >> +  fixed_time = -1;

> >>    product_name = xstrdup (PACKAGE_NAME);

> >>    product_version = xstrdup (PACKAGE_VERSION);

> >>    xorriso = xstrdup ("xorriso");

> >> @@ -541,7 +555,7 @@ main (int argc, char *argv[])

> >>    {

> >>      time_t tim;

> >>      struct tm *tmm;

> >> -    tim = time (NULL);

> >> +    tim = fixed_time != -1 ? fixed_time : time (NULL);

> >>      tmm = gmtime (&tim);

> >>      iso_uuid = xmalloc (55);

> >>      grub_snprintf (iso_uuid, 50,

> >>

> >

> >

> >

> >

> > _______________________________________________

> > Grub-devel mailing list

> > Grub-devel@gnu.org

> > https://lists.gnu.org/mailman/listinfo/grub-devel

> >

>

>

>

> _______________________________________________

> Grub-devel mailing list

> Grub-devel@gnu.org

> https://lists.gnu.org/mailman/listinfo/grub-devel

>

[-- Attachment #2: Type: text/html, Size: 10032 bytes --]

Re: [PATCH v3 2/3] mkrescue: add argument --fixed-time to get reproducible uuids

[Thomas Schmitt]

Hi,

Andrei Borzenkov wrote:
> I am not sure we should stretch reproducible builds that far. ISO
> image created by grub-mkrescue is not binary.

I was approached by Debian's reproducible-builds project because
they wanted to be able to create reproducible test ISOs.
  http://lists.alioth.debian.org/pipermail/reproducible-builds/Week-of-Mon-20150601/001693.html


Vladimir Serbinenko wrote:
> > We need to find a way to reliably find boot
> > disk without depending on current time.

Andrei Borzenkov wrote:
> Well, UUID of isofs used by GRUB is not unique in any sense,

I understand that we rely on the improbability that two
competing ISOs got created in the same second.
So an explicitely chosen "UUID" must be sufficiently random
on the first production to distinguish non-identical images.
Re-productions should then use the same "UUID".

The "UUID" is stored in the ISO as timestamp string of form
YYYYMMDDhhmmsscc with decimal digits. E.g. 2015121517395800
"cc" means centi-seconds, which would be usable to expand the
"UUID" space by a factor of 100.
It cannot be forwarded as time_t, though.
One would need finer time granularity or a second integer variable
which would bring the "cc" part down to the composition of the
xorriso command. (xorrisofs option --modification-date, i assume)


> so it is not really much worse than it was before.
> Having reliable way to identify boot device imply some unique property
> of boot device which automatically conflict with idea of identical images.

Yep. Having identical images would mean that they are the same
in any aspect. So here we should have no problem, i think.
(Adventurous testers could now try what happens if they present
 their machine two copies of the same ISO on two devices.)


Have a nice day :)

Thomas

[PATCH v3 3/3] Makefile/coreboot use SOURCE_DATE_EPOCH as time source if set

[Alexander Couzens]

mkstandalone sets timestamps for files which can be overriden by a fixed_timestamp.
This makes it possible to build reproducible builds for coreboot.

To build a reproducible build of grub for coreboot do:
export SOURCE_DATE_EPOCH=1134242
make default_payload.elf
---
 Makefile.am | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Makefile.am b/Makefile.am
index 994ebbd..5c756d7 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -403,7 +403,7 @@ bootcheck: $(BOOTCHECKS)
 
 if COND_i386_coreboot
 default_payload.elf: grub-mkstandalone grub-mkimage
-	pkgdatadir=. ./grub-mkstandalone --grub-mkimage=./grub-mkimage -O i386-coreboot -o $@ --modules='ahci pata ehci uhci ohci usb_keyboard usbms part_msdos xfs ext2 fat at_keyboard part_gpt usbserial_usbdebug cbfs' --install-modules='ls linux search configfile normal cbtime cbls memrw iorw minicmd lsmmap lspci halt reboot hexdump pcidump regexp setpci lsacpi chain test serial multiboot cbmemc linux16 gzio echo help' --fonts= --themes= --locales= -d grub-core/ /boot/grub/grub.cfg=$(srcdir)/coreboot.cfg
+	pkgdatadir=. ./grub-mkstandalone --grub-mkimage=./grub-mkimage -O i386-coreboot -o $@ --modules='ahci pata ehci uhci ohci usb_keyboard usbms part_msdos xfs ext2 fat at_keyboard part_gpt usbserial_usbdebug cbfs' --install-modules='ls linux search configfile normal cbtime cbls memrw iorw minicmd lsmmap lspci halt reboot hexdump pcidump regexp setpci lsacpi chain test serial multiboot cbmemc linux16 gzio echo help' --fonts= --themes= --locales= -d grub-core/ /boot/grub/grub.cfg=$(srcdir)/coreboot.cfg $(if $(SOURCE_DATE_EPOCH),-t $(SOURCE_DATE_EPOCH))
 endif
 
 endif
-- 
2.6.3

Re: [PATCH v3 3/3] Makefile/coreboot use SOURCE_DATE_EPOCH as time source if set

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1777 bytes --]

Looks good
On 04.12.2015 19:32, Alexander Couzens wrote:
> mkstandalone sets timestamps for files which can be overriden by a fixed_timestamp.
> This makes it possible to build reproducible builds for coreboot.
> 
> To build a reproducible build of grub for coreboot do:
> export SOURCE_DATE_EPOCH=1134242
> make default_payload.elf
> ---
>  Makefile.am | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/Makefile.am b/Makefile.am
> index 994ebbd..5c756d7 100644
> --- a/Makefile.am
> +++ b/Makefile.am
> @@ -403,7 +403,7 @@ bootcheck: $(BOOTCHECKS)
>  
>  if COND_i386_coreboot
>  default_payload.elf: grub-mkstandalone grub-mkimage
> -	pkgdatadir=. ./grub-mkstandalone --grub-mkimage=./grub-mkimage -O i386-coreboot -o $@ --modules='ahci pata ehci uhci ohci usb_keyboard usbms part_msdos xfs ext2 fat at_keyboard part_gpt usbserial_usbdebug cbfs' --install-modules='ls linux search configfile normal cbtime cbls memrw iorw minicmd lsmmap lspci halt reboot hexdump pcidump regexp setpci lsacpi chain test serial multiboot cbmemc linux16 gzio echo help' --fonts= --themes= --locales= -d grub-core/ /boot/grub/grub.cfg=$(srcdir)/coreboot.cfg
> +	pkgdatadir=. ./grub-mkstandalone --grub-mkimage=./grub-mkimage -O i386-coreboot -o $@ --modules='ahci pata ehci uhci ohci usb_keyboard usbms part_msdos xfs ext2 fat at_keyboard part_gpt usbserial_usbdebug cbfs' --install-modules='ls linux search configfile normal cbtime cbls memrw iorw minicmd lsmmap lspci halt reboot hexdump pcidump regexp setpci lsacpi chain test serial multiboot cbmemc linux16 gzio echo help' --fonts= --themes= --locales= -d grub-core/ /boot/grub/grub.cfg=$(srcdir)/coreboot.cfg $(if $(SOURCE_DATE_EPOCH),-t $(SOURCE_DATE_EPOCH))
>  endif
>  
>  endif
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 213 bytes --]