* [PATCH] fix: reproducible builds for initramfs and UKI img
@ 2023-06-05 9:00 Frieder Paape
2023-06-06 6:32 ` [OE-core] " Richard Purdie
0 siblings, 1 reply; 5+ messages in thread
From: Frieder Paape @ 2023-06-05 9:00 UTC (permalink / raw)
To: openembedded-core@lists.openembedded.org
I've encountered issues reproducing initramfs and UKI image builds,
which will be fixed with this patch.
1. initramfs
There's a symbolic link to /sbin/init, which is appended to the cpio archive after creation.
The links timestamp needs to be static and the cpio append command needs the '--reproducible' flag to produce deterministic outcomes.
2. Unified Kernel Image
'--preserve-dates' is required for a static 'Time/Date' entry.
I've added '--enable-deterministic-archives' although in my case this
didn't change anything.
Signed-off-by: Frieder Paape <frieder@konvera.io>
---
meta/classes-recipe/image_types.bbclass | 5 +++--
scripts/lib/wic/plugins/source/bootimg-efi.py | 2 ++
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
index bbddfaf272..f73b4d965e 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -148,10 +148,11 @@ IMAGE_CMD:cpio () {
if [ ! -L ${IMAGE_ROOTFS}/init ] && [ ! -e ${IMAGE_ROOTFS}/init ]; then
if [ -L ${IMAGE_ROOTFS}/sbin/init ] || [ -e ${IMAGE_ROOTFS}/sbin/init ]; then
ln -sf /sbin/init ${WORKDIR}/cpio_append/init
+ touch -h -r ${IMAGE_ROOTFS}/sbin/init ${WORKDIR}/cpio_append/init
else
- touch ${WORKDIR}/cpio_append/init
+ touch -h -r ${IMAGE_ROOTFS} ${WORKDIR}/cpio_append/init
fi
- (cd ${WORKDIR}/cpio_append && echo ./init | cpio -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
+ (cd ${WORKDIR}/cpio_append && echo ./init | cpio --reproducible -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
fi
fi
}
diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
index 43c6fd94d9..2bf7375887 100644
--- a/scripts/lib/wic/plugins/source/bootimg-efi.py
+++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
@@ -351,6 +351,8 @@ class BootimgEFIPlugin(SourcePlugin):
# https://www.freedesktop.org/software/systemd/man/systemd-stub.html
objcopy_cmd = "%s-objcopy" % target_sys
+ objcopy_cmd += " --enable-deterministic-archives"
+ objcopy_cmd += " --preserve-dates"
objcopy_cmd += " --add-section .osrel=%s/usr/lib/os-release" % staging_dir_host
objcopy_cmd += " --change-section-vma .osrel=0x20000"
objcopy_cmd += " --add-section .cmdline=%s" % cmdline.name
--
2.39.2 (Apple Git-143)
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [OE-core] [PATCH] fix: reproducible builds for initramfs and UKI img
2023-06-05 9:00 [PATCH] fix: reproducible builds for initramfs and UKI img Frieder Paape
@ 2023-06-06 6:32 ` Richard Purdie
2023-06-06 8:01 ` [PATCH] fixup! " Frieder Paape
0 siblings, 1 reply; 5+ messages in thread
From: Richard Purdie @ 2023-06-06 6:32 UTC (permalink / raw)
To: Frieder Paape, openembedded-core@lists.openembedded.org
On Mon, 2023-06-05 at 09:00 +0000, Frieder Paape wrote:
> I've encountered issues reproducing initramfs and UKI image builds,
> which will be fixed with this patch.
>
> 1. initramfs
> There's a symbolic link to /sbin/init, which is appended to the cpio archive after creation.
> The links timestamp needs to be static and the cpio append command needs the '--reproducible' flag to produce deterministic outcomes.
>
> 2. Unified Kernel Image
> '--preserve-dates' is required for a static 'Time/Date' entry.
> I've added '--enable-deterministic-archives' although in my case this
> didn't change anything.
>
> Signed-off-by: Frieder Paape <frieder@konvera.io>
> ---
> meta/classes-recipe/image_types.bbclass | 5 +++--
> scripts/lib/wic/plugins/source/bootimg-efi.py | 2 ++
> 2 files changed, 5 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
> index bbddfaf272..f73b4d965e 100644
> --- a/meta/classes-recipe/image_types.bbclass
> +++ b/meta/classes-recipe/image_types.bbclass
> @@ -148,10 +148,11 @@ IMAGE_CMD:cpio () {
> if [ ! -L ${IMAGE_ROOTFS}/init ] && [ ! -e ${IMAGE_ROOTFS}/init ]; then
> if [ -L ${IMAGE_ROOTFS}/sbin/init ] || [ -e ${IMAGE_ROOTFS}/sbin/init ]; then
> ln -sf /sbin/init ${WORKDIR}/cpio_append/init
> + touch -h -r ${IMAGE_ROOTFS}/sbin/init ${WORKDIR}/cpio_append/init
> else
> - touch ${WORKDIR}/cpio_append/init
> + touch -h -r ${IMAGE_ROOTFS} ${WORKDIR}/cpio_append/init
> fi
> - (cd ${WORKDIR}/cpio_append && echo ./init | cpio -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> + (cd ${WORKDIR}/cpio_append && echo ./init | cpio --reproducible -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> fi
> fi
> }
> diff --git a/scripts/lib/wic/plugins/source/bootimg-efi.py b/scripts/lib/wic/plugins/source/bootimg-efi.py
> index 43c6fd94d9..2bf7375887 100644
> --- a/scripts/lib/wic/plugins/source/bootimg-efi.py
> +++ b/scripts/lib/wic/plugins/source/bootimg-efi.py
> @@ -351,6 +351,8 @@ class BootimgEFIPlugin(SourcePlugin):
>
> # https://www.freedesktop.org/software/systemd/man/systemd-stub.html
> objcopy_cmd = "%s-objcopy" % target_sys
> + objcopy_cmd += " --enable-deterministic-archives"
> + objcopy_cmd += " --preserve-dates"
> objcopy_cmd += " --add-section .osrel=%s/usr/lib/os-release" % staging_dir_host
> objcopy_cmd += " --change-section-vma .osrel=0x20000"
> objcopy_cmd += " --add-section .cmdline=%s" % cmdline.name
I like the idea of this and agree it is something we should improve.
Unfortunately it caused testing failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/5236/steps/14/logs/stdio
oe-selftest -r fitimage.FitImageTests.test_initramfs_bundle
should reproduce.
Cheers,
Richard
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH] fixup! fix: reproducible builds for initramfs and UKI img
2023-06-06 6:32 ` [OE-core] " Richard Purdie
@ 2023-06-06 8:01 ` Frieder Paape
2023-06-07 9:00 ` [OE-core] " Alexander Kanavin
0 siblings, 1 reply; 5+ messages in thread
From: Frieder Paape @ 2023-06-06 8:01 UTC (permalink / raw)
To: openembedded-core
[-- Attachment #1: Type: text/plain, Size: 1106 bytes --]
The failure happens because `touch` doesn't create a file if called with the no-dereference option `-h`.
Removing `-h` from affected touch command.
Signed-off-by: Frieder Paape <frieder@konvera.io>
---
meta/classes-recipe/image_types.bbclass | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
index f73b4d965e..023eb87537 100644
--- a/meta/classes-recipe/image_types.bbclass
+++ b/meta/classes-recipe/image_types.bbclass
@@ -150,7 +150,7 @@ IMAGE_CMD:cpio () {
ln -sf /sbin/init ${WORKDIR}/cpio_append/init
touch -h -r ${IMAGE_ROOTFS}/sbin/init ${WORKDIR}/cpio_append/init
else
- touch -h -r ${IMAGE_ROOTFS} ${WORKDIR}/cpio_append/init
+ touch -r ${IMAGE_ROOTFS} ${WORKDIR}/cpio_append/init
fi
(cd ${WORKDIR}/cpio_append && echo ./init | cpio --reproducible -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
fi
--
2.39.2 (Apple Git-143)
[-- Attachment #2: Type: text/html, Size: 1981 bytes --]
^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [OE-core] [PATCH] fixup! fix: reproducible builds for initramfs and UKI img
2023-06-06 8:01 ` [PATCH] fixup! " Frieder Paape
@ 2023-06-07 9:00 ` Alexander Kanavin
2023-06-07 9:15 ` Richard Purdie
0 siblings, 1 reply; 5+ messages in thread
From: Alexander Kanavin @ 2023-06-07 9:00 UTC (permalink / raw)
To: Frieder Paape; +Cc: openembedded-core
It's better to resend the whole patchset as v2.
Alex
On Tue, 6 Jun 2023 at 10:01, Frieder Paape <frieder@konvera.io> wrote:
>
> The failure happens because `touch` doesn't create a file if called with the no-dereference option `-h`.
> Removing `-h` from affected touch command.
>
> Signed-off-by: Frieder Paape <frieder@konvera.io>
> ---
> meta/classes-recipe/image_types.bbclass | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/meta/classes-recipe/image_types.bbclass b/meta/classes-recipe/image_types.bbclass
> index f73b4d965e..023eb87537 100644
> --- a/meta/classes-recipe/image_types.bbclass
> +++ b/meta/classes-recipe/image_types.bbclass
> @@ -150,7 +150,7 @@ IMAGE_CMD:cpio () {
> ln -sf /sbin/init ${WORKDIR}/cpio_append/init
> touch -h -r ${IMAGE_ROOTFS}/sbin/init ${WORKDIR}/cpio_append/init
> else
> - touch -h -r ${IMAGE_ROOTFS} ${WORKDIR}/cpio_append/init
> + touch -r ${IMAGE_ROOTFS} ${WORKDIR}/cpio_append/init
> fi
> (cd ${WORKDIR}/cpio_append && echo ./init | cpio --reproducible -oA -H newc -F ${IMGDEPLOYDIR}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.cpio)
> fi
> --
> 2.39.2 (Apple Git-143)
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#182416): https://lists.openembedded.org/g/openembedded-core/message/182416
> Mute This Topic: https://lists.openembedded.org/mt/99359051/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [OE-core] [PATCH] fixup! fix: reproducible builds for initramfs and UKI img
2023-06-07 9:00 ` [OE-core] " Alexander Kanavin
@ 2023-06-07 9:15 ` Richard Purdie
0 siblings, 0 replies; 5+ messages in thread
From: Richard Purdie @ 2023-06-07 9:15 UTC (permalink / raw)
To: Alexander Kanavin, Frieder Paape; +Cc: openembedded-core
On Wed, 2023-06-07 at 11:00 +0200, Alexander Kanavin wrote:
> It's better to resend the whole patchset as v2.
I did squash this one into the original patch.
For future reference, I did also tweak the shortlog to mention
"image_types:" as the prefix so the area of code changing was clearer
and matches the format we usually use.
It is nice to see work on reproducibility!
Cheers,
Richard
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2023-06-07 9:15 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2023-06-05 9:00 [PATCH] fix: reproducible builds for initramfs and UKI img Frieder Paape
2023-06-06 6:32 ` [OE-core] " Richard Purdie
2023-06-06 8:01 ` [PATCH] fixup! " Frieder Paape
2023-06-07 9:00 ` [OE-core] " Alexander Kanavin
2023-06-07 9:15 ` Richard Purdie
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox