[PATCH] fix: reproducible builds for initramfs and UKI img

[PATCH] fix: reproducible builds for initramfs and UKI img

[Frieder Paape]

I've encountered issues reproducing initramfs and UKI image builds,
which will be fixed with this patch.

1. initramfs
There's a symbolic link to /sbin/init, which is appended to the cpio archive after creation.
The links timestamp needs to be static and the cpio append command needs the '--reproducible' flag to produce deterministic outcomes.

2. Unified Kernel Image
'--preserve-dates' is required for a static 'Time/Date' entry.
I've added '--enable-deterministic-archives' although in my case this
didn't change anything.

Signed-off-by: Frieder Paape <frieder@konvera.io>
---
 meta/classes-recipe/image_types.bbclass       | 5 +++--
 scripts/lib/wic/plugins/source/bootimg-efi.py | 2 ++
 2 files changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
index bbddfaf272..f73b4d965e 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -148,10 +148,11 @@ IMAGE_CMD:cpio () {
 		if [ ! -L ${IMAGE_ROOTFS}/init ] && [ ! -e ${IMAGE_ROOTFS}/init ]; then
 			if [ -L ${IMAGE_ROOTFS}/sbin/init ] || [ -e ${IMAGE_ROOTFS}/sbin/init ]; then
 				ln -sf /sbin/init ${WORKDIR}/cpio_append/init
+                                touch -h -r ${IMAGE_ROOTFS}/sbin/init ${WORKDIR}/cpio_append/init
 			else
-				touch ${WORKDIR}/cpio_append/init
+                                touch -h -r ${IMAGE_ROOTFS} ${WORKDIR}/cpio_append/init
 			fi
-			(cd  ${WORKDIR}/cpio_append && echo ./init | cpio -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
+			(cd  ${WORKDIR}/cpio_append && echo ./init | cpio --reproducible -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
 		fi
 	fi
 }
diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index 43c6fd94d9..2bf7375887 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -351,6 +351,8 @@ class BootimgEFIPlugin(SourcePlugin):
 
                 # https://www.freedesktop.org/software/systemd/man/systemd-stub.html
                 objcopy_cmd = "%s-objcopy" % target_sys
+                objcopy_cmd += " --enable-deterministic-archives"
+                objcopy_cmd += " --preserve-dates"
                 objcopy_cmd += " --add-section .osrel=%s/usr/lib/os-release" % staging_dir_host
                 objcopy_cmd += " --change-section-vma .osrel=0x20000"
                 objcopy_cmd += " --add-section .cmdline=%s" % cmdline.name
-- 
2.39.2 (Apple Git-143)

Re: [OE-core] [PATCH] fix: reproducible builds for initramfs and UKI img

[Richard Purdie]

On Mon, 2023-06-05 at 09:00 +0000, Frieder Paape wrote:
> I've encountered issues reproducing initramfs and UKI image builds,
> which will be fixed with this patch.
> 
> 1. initramfs
> There's a symbolic link to /sbin/init, which is appended to the cpio archive after creation.
> The links timestamp needs to be static and the cpio append command needs the '--reproducible' flag to produce deterministic outcomes.
> 
> 2. Unified Kernel Image
> '--preserve-dates' is required for a static 'Time/Date' entry.
> I've added '--enable-deterministic-archives' although in my case this
> didn't change anything.
> 
> Signed-off-by: Frieder Paape <frieder@konvera.io>
> ---
>  meta/classes-recipe/image_types.bbclass       | 5 +++--
>  scripts/lib/wic/plugins/source/bootimg-efi.py | 2 ++
>  2 files changed, 5 insertions(+), 2 deletions(-)
> 
> diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
> index bbddfaf272..f73b4d965e 100644
> --- a/meta/classes-recipe/image_types.bbclass
> +++ b/meta/classes-recipe/image_types.bbclass
> @@ -148,10 +148,11 @@ IMAGE_CMD:cpio () {
>  		if [ ! -L ${IMAGE_ROOTFS}/init ] && [ ! -e ${IMAGE_ROOTFS}/init ]; then
>  			if [ -L ${IMAGE_ROOTFS}/sbin/init ] || [ -e ${IMAGE_ROOTFS}/sbin/init ]; then
>  				ln -sf /sbin/init ${WORKDIR}/cpio_append/init
> +                                touch -h -r ${IMAGE_ROOTFS}/sbin/init ${WORKDIR}/cpio_append/init
>  			else
> -				touch ${WORKDIR}/cpio_append/init
> +                                touch -h -r ${IMAGE_ROOTFS} ${WORKDIR}/cpio_append/init
>  			fi
> -			(cd  ${WORKDIR}/cpio_append && echo ./init | cpio -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> +			(cd  ${WORKDIR}/cpio_append && echo ./init | cpio --reproducible -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
>  		fi
>  	fi
>  }
> diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
> index 43c6fd94d9..2bf7375887 100644
> --- a/scripts/lib/wic/plugins/source/bootimg-efi.py
> +++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
> @@ -351,6 +351,8 @@ class BootimgEFIPlugin(SourcePlugin):
>  
>                  # https://www.freedesktop.org/software/systemd/man/systemd-stub.html
>                  objcopy_cmd = "%s-objcopy" % target_sys
> +                objcopy_cmd += " --enable-deterministic-archives"
> +                objcopy_cmd += " --preserve-dates"
>                  objcopy_cmd += " --add-section .osrel=%s/usr/lib/os-release" % staging_dir_host
>                  objcopy_cmd += " --change-section-vma .osrel=0x20000"
>                  objcopy_cmd += " --add-section .cmdline=%s" % cmdline.name

I like the idea of this and agree it is something we should improve.
Unfortunately it caused testing failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5236/steps/14/logs/stdio

oe-selftest -r fitimage.FitImageTests.test_initramfs_bundle

should reproduce.

Cheers,

Richard

Re: [PATCH] fixup! fix: reproducible builds for initramfs and UKI img

[Frieder Paape]

[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]

The failure happens because `touch` doesn't create a file if called with the no-dereference option `-h`.
Removing `-h` from affected touch command.

Signed-off-by: Frieder Paape <frieder@konvera.io>
---
meta/classes-recipe/image_types.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
index f73b4d965e..023eb87537 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -150,7 +150,7 @@ IMAGE_CMD:cpio () {
ln -sf /sbin/init ${WORKDIR}/cpio_append/init
touch -h -r ${IMAGE_ROOTFS}/sbin/init ${WORKDIR}/cpio_append/init
else
-                                touch -h -r ${IMAGE_ROOTFS} ${WORKDIR}/cpio_append/init
+                                touch -r ${IMAGE_ROOTFS} ${WORKDIR}/cpio_append/init
fi
(cd  ${WORKDIR}/cpio_append && echo ./init | cpio --reproducible -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
fi
--
2.39.2 (Apple Git-143)

[-- Attachment #2: Type: text/html, Size: 1981 bytes --]

Re: [OE-core] [PATCH] fixup! fix: reproducible builds for initramfs and UKI img

[Alexander Kanavin]

It's better to resend the whole patchset as v2.

Alex

On Tue, 6 Jun 2023 at 10:01, Frieder Paape <frieder@konvera.io> wrote:
>
> The failure happens because `touch` doesn't create a file if called with the no-dereference option `-h`.
> Removing `-h` from affected touch command.
>
> Signed-off-by: Frieder Paape <frieder@konvera.io>
> ---
>  meta/classes-recipe/image_types.bbclass | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
> index f73b4d965e..023eb87537 100644
> --- a/meta/classes-recipe/image_types.bbclass
> +++ b/meta/classes-recipe/image_types.bbclass
> @@ -150,7 +150,7 @@ IMAGE_CMD:cpio () {
>                  ln -sf /sbin/init ${WORKDIR}/cpio_append/init
>                                  touch -h -r ${IMAGE_ROOTFS}/sbin/init ${WORKDIR}/cpio_append/init
>              else
> -                                touch -h -r ${IMAGE_ROOTFS} ${WORKDIR}/cpio_append/init
> +                                touch -r ${IMAGE_ROOTFS} ${WORKDIR}/cpio_append/init
>              fi
>              (cd  ${WORKDIR}/cpio_append && echo ./init | cpio --reproducible -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
>          fi
> --
> 2.39.2 (Apple Git-143)
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#182416): https://lists.openembedded.org/g/openembedded-core/message/182416
> Mute This Topic: https://lists.openembedded.org/mt/99359051/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] [PATCH] fixup! fix: reproducible builds for initramfs and UKI img

[Richard Purdie]

On Wed, 2023-06-07 at 11:00 +0200, Alexander Kanavin wrote:
> It's better to resend the whole patchset as v2.

I did squash this one into the original patch. 

For future reference, I did also tweak the shortlog to mention
"image_types:" as the prefix so the area of code changing was clearer
and matches the format we usually use.

It is nice to see work on reproducibility!

Cheers,

Richard