Deterministic grub-mkimage

Deterministic grub-mkimage

[Andrew Clausen]

Hi all,

Deterministic software builds are helpful for spotting and preventing
malicious modifications such as inserting back-doors.

At the moment, grub builds are mostly deterministic.  However,
grub-mkimage does not deterministically build EFI binaries.  This is
because the PE/COFF headers include timestamps.  This is a widespread
problem in the Windows world -- see for example a discussion of
deterministically building TrueCrypt. [1]

One solution would be to:
 * build deterministically by default by using a constant timestamp, and
 * add a --with-timestamps option (disabled by default), which would
enable honest timestamps.

What do you think?  Are you accepting patches?

Cheers,
Andrew

[1] https://madiba.encs.concordia.ca/~x_decarn/truecrypt-binaries-analysis/

Re: Deterministic grub-mkimage

[Jonathan McCune]

[-- Attachment #1: Type: text/plain, Size: 1437 bytes --]

On Sun, Dec 28, 2014 at 3:24 AM, Andrew Clausen <andrew.p.clausen@gmail.com>
wrote:

> Hi all,
>
> Deterministic software builds are helpful for spotting and preventing
> malicious modifications such as inserting back-doors.
>

Agree.


> At the moment, grub builds are mostly deterministic.  However,
> grub-mkimage does not deterministically build EFI binaries.  This is
> because the PE/COFF headers include timestamps.  This is a widespread
> problem in the Windows world -- see for example a discussion of
> deterministically building TrueCrypt. [1]
>
> One solution would be to:
>  * build deterministically by default by using a constant timestamp, and
>

I think doing this by default would be a poor choice, as most of the time
during development it is very useful to easily identify which version /
build / experiment / etc is in use.

 * add a --with-timestamps option (disabled by default), which would
> enable honest timestamps.
>
> What do you think?  Are you accepting patches?
>
>
The availability of a flag to explicitly set a specific timestamp for the
purpose of reproducing a build, seems sane to me. I don't think I would
enable it by default.

/$0.02
-Jon




> Cheers,
> Andrew
>
> [1]
> https://madiba.encs.concordia.ca/~x_decarn/truecrypt-binaries-analysis/
>
> _______________________________________________
> Grub-devel mailing list
> Grub-devel@gnu.org
> https://lists.gnu.org/mailman/listinfo/grub-devel
>

[-- Attachment #2: Type: text/html, Size: 2812 bytes --]

Re: Deterministic grub-mkimage

[Andrew Clausen]

Hi Jonathan,

On 29 December 2014 at 06:29, Jonathan McCune <jonmccune@google.com> wrote:
>> One solution would be to:
>>  * build deterministically by default by using a constant timestamp, and
>
> I think doing this by default would be a poor choice, as most of the time
> during development it is very useful to easily identify which version /
> build / experiment / etc is in use.

I agree that during development, timestamps might be useful.  Although
I've never found them particularly helpful myself -- they aren't as
easy as, say, having a text file sitting in the same directory saying
which git commit it is.  In fact, including the git commit somewhere
in the binary would be both more helpful and deterministic.  (I am
happy to supply a patch for this.)  Have you ever used time stamps?

>>  * add a --with-timestamps option (disabled by default), which would
>> enable honest timestamps.
>>
>> What do you think?  Are you accepting patches?
>
> The availability of a flag to explicitly set a specific timestamp for the
> purpose of reproducing a build, seems sane to me. I don't think I would
> enable it by default.

Sorry to be stubborn on this point, but I think it's quite important.
If most people are using deterministic builds, then it becomes much
easier for people to audit against each other's computers.  At the
moment, when I do audits with Grub, I have to ask my
colleagues/friends to zero out the timestamp.  It makes the
conversation longer, which makes me feel reluctant to inconvenience
them.  So I end up doing a less thorough audit.  This kind of audit
scenario arises frequently (or at least, it ought to) in work with
NGOs, journalists, law firms, etc.

Bottom line: I think there is an important social benefit to dropping
timestamps by default.  I'm not convinced timestamps are used much by
developers, and there are better alternatives such as git-commits.

Cheers,
Andrew

Re: Deterministic grub-mkimage

[Jonathan McCune]

[-- Attachment #1: Type: text/plain, Size: 2115 bytes --]

On Mon, Dec 29, 2014 at 3:08 AM, Andrew Clausen <andrew.p.clausen@gmail.com>
wrote:

> Hi Jonathan,
>
> On 29 December 2014 at 06:29, Jonathan McCune <jonmccune@google.com>
> wrote:
> >> One solution would be to:
> >>  * build deterministically by default by using a constant timestamp, and
> >
> > I think doing this by default would be a poor choice, as most of the time
> > during development it is very useful to easily identify which version /
> > build / experiment / etc is in use.
>
> I agree that during development, timestamps might be useful.  Although
> I've never found them particularly helpful myself -- they aren't as
> easy as, say, having a text file sitting in the same directory saying
> which git commit it is.  In fact, including the git commit somewhere
> in the binary would be both more helpful and deterministic.  (I am
> happy to supply a patch for this.)  Have you ever used time stamps?
>
> >>  * add a --with-timestamps option (disabled by default), which would
> >> enable honest timestamps.
> >>
> >> What do you think?  Are you accepting patches?
> >
> > The availability of a flag to explicitly set a specific timestamp for the
> > purpose of reproducing a build, seems sane to me. I don't think I would
> > enable it by default.
>
> Sorry to be stubborn on this point, but I think it's quite important.
> If most people are using deterministic builds, then it becomes much
> easier for people to audit against each other's computers.  At the
> moment, when I do audits with Grub, I have to ask my
> colleagues/friends to zero out the timestamp.  It makes the
> conversation longer, which makes me feel reluctant to inconvenience
> them.  So I end up doing a less thorough audit.  This kind of audit
> scenario arises frequently (or at least, it ought to) in work with
> NGOs, journalists, law firms, etc.
>
> Bottom line: I think there is an important social benefit to dropping
> timestamps by default.  I'm not convinced timestamps are used much by
> developers, and there are better alternatives such as git-commits.
>

No objection from me, though I'm not a maintainer.

-Jon

[-- Attachment #2: Type: text/html, Size: 2781 bytes --]

Re: Deterministic grub-mkimage

[Vladimir 'φ-coder/phcoder' Serbinenko]

[-- Attachment #1: Type: text/plain, Size: 1343 bytes --]

On 29.12.2014 12:08, Andrew Clausen wrote:
> Hi Jonathan,
> 
> On 29 December 2014 at 06:29, Jonathan McCune <jonmccune@google.com> wrote:
>>> One solution would be to:
>>>  * build deterministically by default by using a constant timestamp, and
>>
>> I think doing this by default would be a poor choice, as most of the time
>> during development it is very useful to easily identify which version /
>> build / experiment / etc is in use.
> 
> I agree that during development, timestamps might be useful.  Although
> I've never found them particularly helpful myself -- they aren't as
> easy as, say, having a text file sitting in the same directory saying
> which git commit it is.  In fact, including the git commit somewhere
> in the binary would be both more helpful and deterministic.  (I am
> happy to supply a patch for this.)  Have you ever used time stamps?
> 
We already have modinfo.sh. It would be a good place to put commit. In
fact it already includes version. The best would be to add
+g<commit>[-dirty] to version when building from git.
My main concern is that EFI itself might use the timestamp for some
weird caching but this shouldn't be the case. Other than that I'd be ok
with hardcoding it to unix time
1420070400 (Jan 1, 2015, midnight UTC)
Can you prepare the patches for both things?




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

[PATCH] use stock embedded timestamp 2015-01-01T0000+0000

[Daniel Kahn Gillmor]

Variant timestamps make some grub platforms produce non-deterministic
core images.  This makes it difficult to use simple tools to audit the
stability of a system with grub installed.

This patch selects a single timestamp to use for these embedded
timestamps so that the core images will be replicable.
---
 util/mkimage.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/util/mkimage.c b/util/mkimage.c
index 7821dc5..adc1706 100644
--- a/util/mkimage.c
+++ b/util/mkimage.c
@@ -55,6 +55,9 @@
 
 #define TARGET_NO_FIELD 0xffffffff
 
+/* use 2015-01-01T00:00:00+0000 as a stock timestamp */
+#define STABLE_EMBEDDING_TIMESTAMP 1420070400
+
 struct grub_install_image_target_desc
 {
   const char *dirname;
@@ -1439,7 +1442,7 @@ grub_install_generate_image (const char *dir, const char *prefix,
 	c->machine = grub_host_to_target16 (image_target->pe_target);
 
 	c->num_sections = grub_host_to_target16 (4);
-	c->time = grub_host_to_target32 (time (0));
+	c->time = grub_host_to_target32 (STABLE_EMBEDDING_TIMESTAMP);
 	c->characteristics = grub_host_to_target16 (GRUB_PE32_EXECUTABLE_IMAGE
 						    | GRUB_PE32_LINE_NUMS_STRIPPED
 						    | ((image_target->voidp_sizeof == 4)
@@ -1782,7 +1785,7 @@ grub_install_generate_image (const char *dir, const char *prefix,
 
       memset (hdr, 0, sizeof (*hdr));
       hdr->ih_magic = grub_cpu_to_be32_compile_time (GRUB_UBOOT_IH_MAGIC);
-      hdr->ih_time = grub_cpu_to_be32 (time (0));
+      hdr->ih_time = grub_cpu_to_be32 (STABLE_EMBEDDING_TIMESTAMP);
       hdr->ih_size = grub_cpu_to_be32 (core_size);
       hdr->ih_load = grub_cpu_to_be32 (image_target->link_addr);
       hdr->ih_ep = grub_cpu_to_be32 (image_target->link_addr);
@@ -1856,7 +1859,7 @@ grub_install_generate_image (const char *dir, const char *prefix,
 	head->magic = image_target->bigendian ? grub_host_to_target16 (0x160)
 	  : grub_host_to_target16 (0x166);
 	head->nsec = grub_host_to_target16 (1);
-	head->time = grub_host_to_target32 (0);
+	head->time = grub_host_to_target32 (STABLE_EMBEDDING_TIMESTAMP);
 	head->opt = grub_host_to_target16 (0x38);
 	head->flags = image_target->bigendian
 	  ? grub_host_to_target16 (0x207)
-- 
2.1.4

Re: [PATCH] use stock embedded timestamp 2015-01-01T0000+0000

[Vladimir 'φ-coder/phcoder' Serbinenko]

On 22.03.2015 20:33, Daniel Kahn Gillmor wrote:
> Variant timestamps make some grub platforms produce non-deterministic
> core images.  This makes it difficult to use simple tools to audit the
> stability of a system with grub installed.
>
> This patch selects a single timestamp to use for these embedded
> timestamps so that the core images will be replicable.
> ---
>   util/mkimage.c | 9 ++++++---
>   1 file changed, 6 insertions(+), 3 deletions(-)
>
> diff --git a/util/mkimage.c b/util/mkimage.c
> index 7821dc5..adc1706 100644
> --- a/util/mkimage.c
> +++ b/util/mkimage.c
> @@ -55,6 +55,9 @@
>
>   #define TARGET_NO_FIELD 0xffffffff
>
> +/* use 2015-01-01T00:00:00+0000 as a stock timestamp */
> +#define STABLE_EMBEDDING_TIMESTAMP 1420070400
> +
>   struct grub_install_image_target_desc
>   {
>     const char *dirname;
> @@ -1439,7 +1442,7 @@ grub_install_generate_image (const char *dir, const char *prefix,
>   	c->machine = grub_host_to_target16 (image_target->pe_target);
>
>   	c->num_sections = grub_host_to_target16 (4);
> -	c->time = grub_host_to_target32 (time (0));
> +	c->time = grub_host_to_target32 (STABLE_EMBEDDING_TIMESTAMP);
>   	c->characteristics = grub_host_to_target16 (GRUB_PE32_EXECUTABLE_IMAGE
>   						    | GRUB_PE32_LINE_NUMS_STRIPPED
>   						    | ((image_target->voidp_sizeof == 4)
> @@ -1782,7 +1785,7 @@ grub_install_generate_image (const char *dir, const char *prefix,
>
>         memset (hdr, 0, sizeof (*hdr));
>         hdr->ih_magic = grub_cpu_to_be32_compile_time (GRUB_UBOOT_IH_MAGIC);
> -      hdr->ih_time = grub_cpu_to_be32 (time (0));
> +      hdr->ih_time = grub_cpu_to_be32 (STABLE_EMBEDDING_TIMESTAMP);
>         hdr->ih_size = grub_cpu_to_be32 (core_size);
>         hdr->ih_load = grub_cpu_to_be32 (image_target->link_addr);
>         hdr->ih_ep = grub_cpu_to_be32 (image_target->link_addr);
> @@ -1856,7 +1859,7 @@ grub_install_generate_image (const char *dir, const char *prefix,
>   	head->magic = image_target->bigendian ? grub_host_to_target16 (0x160)
>   	  : grub_host_to_target16 (0x166);
>   	head->nsec = grub_host_to_target16 (1);
> -	head->time = grub_host_to_target32 (0);
> +	head->time = grub_host_to_target32 (STABLE_EMBEDDING_TIMESTAMP);
I dropped this hunk as it's just changing one const to another.
>   	head->opt = grub_host_to_target16 (0x38);
>   	head->flags = image_target->bigendian
>   	  ? grub_host_to_target16 (0x207)
>

Re: [PATCH] use stock embedded timestamp 2015-01-01T0000+0000

[Daniel Kahn Gillmor]

On Fri 2015-03-27 08:27:42 -0400, Vladimir 'φ-coder/phcoder' Serbinenko wrote:
>> @@ -1856,7 +1859,7 @@ grub_install_generate_image (const char *dir, const char *prefix,
>>   	head->magic = image_target->bigendian ? grub_host_to_target16 (0x160)
>>   	  : grub_host_to_target16 (0x166);
>>   	head->nsec = grub_host_to_target16 (1);
>> -	head->time = grub_host_to_target32 (0);
>> +	head->time = grub_host_to_target32 (STABLE_EMBEDDING_TIMESTAMP);
> I dropped this hunk as it's just changing one const to another.
>>   	head->opt = grub_host_to_target16 (0x38);
>>   	head->flags = image_target->bigendian
>>   	  ? grub_host_to_target16 (0x207)

Sure, that's fine with me.  I supplied it there so that we could say
"any grub image on any platform with an embedded timestamp will use the
same timestamp", but if you don't think that's a necessary or useful
statement to make, i have no interest in pushing it separately.

My main goal is image reproducibility, and the MIPS_ARC builds were
already reproducible.  Thanks for adopting the fixed timestamp for the
other two platforms!

          --dkg