[Buildroot] [PATCH] fs/cpio: make cpio rootfs reproducible

[Buildroot] [PATCH] fs/cpio: make cpio rootfs reproducible

[Atharva Lele]

Pass the recommended argument in the CPIO manual to make cpio archives
reproducible.

Reference: https://www.gnu.org/software/cpio/manual/cpio.html#Copy_002dpass-mode

Pre-patch diffoscope output: https://gitlab.com/snippets/1874745
Post-patch: https://gitlab.com/snippets/1874746

We can see that post-patch, the archive related differences are removed.
The differences are arising from utils/bin/getconf. This will have to
be investigated further. However, that is unrelated to cpio.

Signed-off-by: Atharva Lele <itsatharva@gmail.com>
---
 fs/cpio/cpio.mk | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/fs/cpio/cpio.mk b/fs/cpio/cpio.mk
index 1f0cafe62f..4847f56d53 100644
--- a/fs/cpio/cpio.mk
+++ b/fs/cpio/cpio.mk
@@ -29,8 +29,12 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC
 
 ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT
 
+ifeq ($(BR2_REPRODUCIBLE), y)
+CPIO_OPTS = --reproducible
+endif
+
 define ROOTFS_CPIO_CMD
-	cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@
+	cd $(TARGET_DIR) && find . | cpio $(CPIO_OPTS) --quiet -o -H newc > $@
 endef
 
 ifeq ($(BR2_TARGET_ROOTFS_CPIO_UIMAGE),y)
-- 
2.22.0

[Buildroot] [PATCH] fs/cpio: make cpio rootfs reproducible

[Arnout Vandecappelle]

 Hi Atharva,

On 12/07/2019 19:23, Atharva Lele wrote:
> Pass the recommended argument in the CPIO manual to make cpio archives
> reproducible.
> 
> Reference: https://www.gnu.org/software/cpio/manual/cpio.html#Copy_002dpass-mode
> 
> Pre-patch diffoscope output: https://gitlab.com/snippets/1874745
> Post-patch: https://gitlab.com/snippets/1874746
> 
> We can see that post-patch, the archive related differences are removed.
> The differences are arising from utils/bin/getconf. This will have to
> be investigated further. However, that is unrelated to cpio.
> 
> Signed-off-by: Atharva Lele <itsatharva@gmail.com>
> ---
>  fs/cpio/cpio.mk | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/fs/cpio/cpio.mk b/fs/cpio/cpio.mk
> index 1f0cafe62f..4847f56d53 100644
> --- a/fs/cpio/cpio.mk
> +++ b/fs/cpio/cpio.mk
> @@ -29,8 +29,12 @@ endif # BR2_ROOTFS_DEVICE_CREATION_STATIC
>  
>  ROOTFS_CPIO_PRE_GEN_HOOKS += ROOTFS_CPIO_ADD_INIT
>  
> +ifeq ($(BR2_REPRODUCIBLE), y)
> +CPIO_OPTS = --reproducible

 Unfortunately, this option was only introduced in cpio 2.12 (2015). Older
distros, including Debian oldstable and Ubuntu 2016.04, both of which are still
being maintained, have an earlier cpio version (2.11) so don't have this option.

 Therefore, I think we have to check for a suitable system cpio and build
host-cpio if it is not suitable. Or maybe, for reproducible, always build
host-cpio?

 Yann, what do you think?

 Otherwise, the patch looks good. But I've marked it as Changes Requested in
pathchwork.

 Regards,
 Arnout

> +endif
> +
>  define ROOTFS_CPIO_CMD
> -	cd $(TARGET_DIR) && find . | cpio --quiet -o -H newc > $@
> +	cd $(TARGET_DIR) && find . | cpio $(CPIO_OPTS) --quiet -o -H newc > $@
>  endef
>  
>  ifeq ($(BR2_TARGET_ROOTFS_CPIO_UIMAGE),y)
> 

[Buildroot] [PATCH] fs/cpio: make cpio rootfs reproducible

[Yann E. MORIN]

Arnout, Atharva, All,

On 2019-07-14 00:34 +0200, Arnout Vandecappelle spake thusly:
> On 12/07/2019 19:23, Atharva Lele wrote:
> > Pass the recommended argument in the CPIO manual to make cpio archives
> > reproducible.
[--SNIP--]
> > +ifeq ($(BR2_REPRODUCIBLE), y)
> > +CPIO_OPTS = --reproducible
> 
>  Unfortunately, this option was only introduced in cpio 2.12 (2015). Older
> distros, including Debian oldstable and Ubuntu 2016.04, both of which are still
> being maintained, have an earlier cpio version (2.11) so don't have this option.
> 
>  Therefore, I think we have to check for a suitable system cpio and build
> host-cpio if it is not suitable. Or maybe, for reproducible, always build
> host-cpio?

If we are reproducible, then always build our cpio version.

We currently do not have a host-cpio package, but it is relatively
trivial to add (patch pending). Yet, I tried and it adds about 25s to
the build, somewhat comparable to the overhead for host-tar...

If someone gets sufficiently annoyed by the overhead, they'll provide
a patch to switch to checking for a suitable cpio.

But in any case, I think that people who do care about reproducibility,
do care more about it than they do about time.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'