[cip-dev][isar-cip-core][PATCH] initramfs-verity-hook: Ensure sync on rebuild

[cip-dev][isar-cip-core][PATCH] initramfs-verity-hook: Ensure sync on rebuild

[Quirin Gylstorff]

From: Quirin Gylstorff <quirin.gylstorff@siemens.com>

Disable the timestamp of the do_install() task to
re-execute when ever a dependency on do_install
is executed.

Without this change the sstate needs to be clean for
every rebuild.

Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
---
 .../initramfs-verity-hook/initramfs-verity-hook_0.1.bb         | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb b/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb
index 3fc63ed..7435649 100644
--- a/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb
+++ b/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb
@@ -40,6 +40,9 @@ do_install[cleandirs] += " \
     ${D}/usr/share/initramfs-tools/scripts/local-top \
     ${D}/usr/share/initramfs-tools/conf-hooks.d"
 
+# Ensure VERITY_ENV_FILE is updated also when rebuilding initramfs-verity-hook
+do_install[nostamp] = "1"
+
 do_install() {
     # Insert the veritysetup commandline into the script
     if [ -f "${VERITY_ENV_FILE}" ]; then
-- 
2.40.1

Re: [cip-dev][isar-cip-core][PATCH] initramfs-verity-hook: Ensure sync on rebuild

[Jan Kiszka]

On 23.08.23 15:28, Quirin Gylstorff wrote:
> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
> 
> Disable the timestamp of the do_install() task to
> re-execute when ever a dependency on do_install
> is executed.
> 
> Without this change the sstate needs to be clean for
> every rebuild.
> 
> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
> ---
>  .../initramfs-verity-hook/initramfs-verity-hook_0.1.bb         | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb b/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb
> index 3fc63ed..7435649 100644
> --- a/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb
> +++ b/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb
> @@ -40,6 +40,9 @@ do_install[cleandirs] += " \
>      ${D}/usr/share/initramfs-tools/scripts/local-top \
>      ${D}/usr/share/initramfs-tools/conf-hooks.d"
>  
> +# Ensure VERITY_ENV_FILE is updated also when rebuilding initramfs-verity-hook
> +do_install[nostamp] = "1"
> +
>  do_install() {
>      # Insert the veritysetup commandline into the script
>      if [ -f "${VERITY_ENV_FILE}" ]; then

No other dpkg-raw packages require this, do they? What makes initramfs
packages special here? And what about those in isar upstream?

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [cip-dev][isar-cip-core][PATCH] initramfs-verity-hook: Ensure sync on rebuild

[Gylstorff Quirin]

On 8/28/23 08:47, Jan Kiszka wrote:
> On 23.08.23 15:28, Quirin Gylstorff wrote:
>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>
>> Disable the timestamp of the do_install() task to
>> re-execute when ever a dependency on do_install
>> is executed.
>>
>> Without this change the sstate needs to be clean for
>> every rebuild.
>>
>> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>> ---
>>   .../initramfs-verity-hook/initramfs-verity-hook_0.1.bb         | 3 +++
>>   1 file changed, 3 insertions(+)
>>
>> diff --git a/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb b/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb
>> index 3fc63ed..7435649 100644
>> --- a/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb
>> +++ b/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb
>> @@ -40,6 +40,9 @@ do_install[cleandirs] += " \
>>       ${D}/usr/share/initramfs-tools/scripts/local-top \
>>       ${D}/usr/share/initramfs-tools/conf-hooks.d"
>>   
>> +# Ensure VERITY_ENV_FILE is updated also when rebuilding initramfs-verity-hook
>> +do_install[nostamp] = "1"
>> +
>>   do_install() {
>>       # Insert the veritysetup commandline into the script
>>       if [ -f "${VERITY_ENV_FILE}" ]; then
> 
> No other dpkg-raw packages require this, do they? What makes initramfs
> packages special here? And what about those in isar upstream?
>

This is necessary for the initramfs package which contain the 
checksum/uuid from the root file system build.
This ensure that the checksum/uuid is refreshed.

It fixes the problem that an rebuild of a verity image leads to a broken 
boot after the build.

I will send a v2 to clarify.

Quirin

Re: [cip-dev][isar-cip-core][PATCH] initramfs-verity-hook: Ensure sync on rebuild

[Jan Kiszka]

On 12.09.23 15:37, Gylstorff Quirin wrote:
> 
> 
> On 8/28/23 08:47, Jan Kiszka wrote:
>> On 23.08.23 15:28, Quirin Gylstorff wrote:
>>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>
>>> Disable the timestamp of the do_install() task to
>>> re-execute when ever a dependency on do_install
>>> is executed.
>>>
>>> Without this change the sstate needs to be clean for
>>> every rebuild.
>>>
>>> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>> ---
>>>   .../initramfs-verity-hook/initramfs-verity-hook_0.1.bb         | 3 +++
>>>   1 file changed, 3 insertions(+)
>>>
>>> diff --git
>>> a/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb b/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb
>>> index 3fc63ed..7435649 100644
>>> ---
>>> a/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb
>>> +++
>>> b/recipes-initramfs/initramfs-verity-hook/initramfs-verity-hook_0.1.bb
>>> @@ -40,6 +40,9 @@ do_install[cleandirs] += " \
>>>       ${D}/usr/share/initramfs-tools/scripts/local-top \
>>>       ${D}/usr/share/initramfs-tools/conf-hooks.d"
>>>   +# Ensure VERITY_ENV_FILE is updated also when rebuilding
>>> initramfs-verity-hook
>>> +do_install[nostamp] = "1"
>>> +
>>>   do_install() {
>>>       # Insert the veritysetup commandline into the script
>>>       if [ -f "${VERITY_ENV_FILE}" ]; then
>>
>> No other dpkg-raw packages require this, do they? What makes initramfs
>> packages special here? And what about those in isar upstream?
>>
> 
> This is necessary for the initramfs package which contain the
> checksum/uuid from the root file system build.
> This ensure that the checksum/uuid is refreshed.
> 
> It fixes the problem that an rebuild of a verity image leads to a broken
> boot after the build.

And what will happen when the user decides to pin the UUID to fix value
in order to have reproducibility? Won't that cause a needless re-run?
Shouldn't this recipe rather make sure to depend on the UUID value?

Jan

-- 
Siemens AG, Technology
Linux Expert Center

Re: [cip-dev][isar-cip-core][PATCH] initramfs-verity-hook: Ensure sync on rebuild

[MOESSBAUER, Felix]

On Tue, 2023-09-12 at 15:55 +0530, Jan Kiszka wrote:
> On 12.09.23 15:37, Gylstorff Quirin wrote:
> > 
> > 
> > On 8/28/23 08:47, Jan Kiszka wrote:
> > > On 23.08.23 15:28, Quirin Gylstorff wrote:
> > > > From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
> > > > 
> > > > Disable the timestamp of the do_install() task to
> > > > re-execute when ever a dependency on do_install
> > > > is executed.
> > > > 
> > > > Without this change the sstate needs to be clean for
> > > > every rebuild.
> > > > 
> > > > Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
> > > > ---
> > > >   .../initramfs-verity-hook/initramfs-verity-
> > > > hook_0.1.bb         | 3 +++
> > > >   1 file changed, 3 insertions(+)
> > > > 
> > > > diff --git
> > > > a/recipes-initramfs/initramfs-verity-hook/initramfs-verity-
> > > > hook_0.1.bb b/recipes-initramfs/initramfs-verity-
> > > > hook/initramfs-verity-hook_0.1.bb
> > > > index 3fc63ed..7435649 100644
> > > > ---
> > > > a/recipes-initramfs/initramfs-verity-hook/initramfs-verity-
> > > > hook_0.1.bb
> > > > +++
> > > > b/recipes-initramfs/initramfs-verity-hook/initramfs-verity-
> > > > hook_0.1.bb
> > > > @@ -40,6 +40,9 @@ do_install[cleandirs] += " \
> > > >       ${D}/usr/share/initramfs-tools/scripts/local-top \
> > > >       ${D}/usr/share/initramfs-tools/conf-hooks.d"
> > > >   +# Ensure VERITY_ENV_FILE is updated also when rebuilding
> > > > initramfs-verity-hook
> > > > +do_install[nostamp] = "1"
> > > > +
> > > >   do_install() {
> > > >       # Insert the veritysetup commandline into the script
> > > >       if [ -f "${VERITY_ENV_FILE}" ]; then
> > > 
> > > No other dpkg-raw packages require this, do they? What makes
> > > initramfs
> > > packages special here? And what about those in isar upstream?
> > > 
> > 
> > This is necessary for the initramfs package which contain the
> > checksum/uuid from the root file system build.
> > This ensure that the checksum/uuid is refreshed.
> > 
> > It fixes the problem that an rebuild of a verity image leads to a
> > broken
> > boot after the build.
> 
> And what will happen when the user decides to pin the UUID to fix
> value
> in order to have reproducibility? Won't that cause a needless re-run?
> Shouldn't this recipe rather make sure to depend on the UUID value?

This is a general reproducibility topic: The task checksums are
computed at parsing time and by that the tasks that need to be re-run
are selected. However, this is based on the assumption that tasks are
deterministic, i.e. same input data also generates same output data.

Unfortunately we are still not 100% reproducible. By that this pre-
condition is not met. For normal packages this does not really matter,
but for the dm-verity checksum this very well matters.

I'm just wondering if we could work around that by making this
component non-cacheable. Putting Adriaan in CC for the sstate part.

Felix

> 
> Jan
> 

Re: [cip-dev][isar-cip-core][PATCH] initramfs-verity-hook: Ensure sync on rebuild

[Gylstorff Quirin]

On 9/13/23 03:26, MOESSBAUER, Felix (T CED INW-CN) wrote:
> On Tue, 2023-09-12 at 15:55 +0530, Jan Kiszka wrote:
>> On 12.09.23 15:37, Gylstorff Quirin wrote:
>>>
>>>
>>> On 8/28/23 08:47, Jan Kiszka wrote:
>>>> On 23.08.23 15:28, Quirin Gylstorff wrote:
>>>>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>>>
>>>>> Disable the timestamp of the do_install() task to
>>>>> re-execute when ever a dependency on do_install
>>>>> is executed.
>>>>>
>>>>> Without this change the sstate needs to be clean for
>>>>> every rebuild.
>>>>>
>>>>> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>>> ---
>>>>>    .../initramfs-verity-hook/initramfs-verity-
>>>>> hook_0.1.bb         | 3 +++
>>>>>    1 file changed, 3 insertions(+)
>>>>>
>>>>> diff --git
>>>>> a/recipes-initramfs/initramfs-verity-hook/initramfs-verity-
>>>>> hook_0.1.bb b/recipes-initramfs/initramfs-verity-
>>>>> hook/initramfs-verity-hook_0.1.bb
>>>>> index 3fc63ed..7435649 100644
>>>>> ---
>>>>> a/recipes-initramfs/initramfs-verity-hook/initramfs-verity-
>>>>> hook_0.1.bb
>>>>> +++
>>>>> b/recipes-initramfs/initramfs-verity-hook/initramfs-verity-
>>>>> hook_0.1.bb
>>>>> @@ -40,6 +40,9 @@ do_install[cleandirs] += " \
>>>>>        ${D}/usr/share/initramfs-tools/scripts/local-top \
>>>>>        ${D}/usr/share/initramfs-tools/conf-hooks.d"
>>>>>    +# Ensure VERITY_ENV_FILE is updated also when rebuilding
>>>>> initramfs-verity-hook
>>>>> +do_install[nostamp] = "1"
>>>>> +
>>>>>    do_install() {
>>>>>        # Insert the veritysetup commandline into the script
>>>>>        if [ -f "${VERITY_ENV_FILE}" ]; then
>>>>
>>>> No other dpkg-raw packages require this, do they? What makes
>>>> initramfs
>>>> packages special here? And what about those in isar upstream?
>>>>
>>>
>>> This is necessary for the initramfs package which contain the
>>> checksum/uuid from the root file system build.
>>> This ensure that the checksum/uuid is refreshed.
>>>
>>> It fixes the problem that an rebuild of a verity image leads to a
>>> broken
>>> boot after the build.
>>
>> And what will happen when the user decides to pin the UUID to fix
>> value
>> in order to have reproducibility? Won't that cause a needless re-run?
>> Shouldn't this recipe rather make sure to depend on the UUID value?
>
Even with pinning and same salt and uuid veritysetup will generate a new 
roothash during the build.

Quirin


> This is a general reproducibility topic: The task checksums are
> computed at parsing time and by that the tasks that need to be re-run
> are selected. However, this is based on the assumption that tasks are
> deterministic, i.e. same input data also generates same output data.
> 
> Unfortunately we are still not 100% reproducible. By that this pre-
> condition is not met. For normal packages this does not really matter,
> but for the dm-verity checksum this very well matters.
> 

> I'm just wondering if we could work around that by making this
> component non-cacheable. Putting Adriaan in CC for the sstate part.

The same  solution as is used by meta-security for Yocto[1].

[1]: https://lists.yoctoproject.org/g/yocto/message/50624

Quirin

> 
> Felix
> 
>>
>> Jan
>>
> 

Re: [cip-dev][isar-cip-core][PATCH] initramfs-verity-hook: Ensure sync on rebuild

[Jan Kiszka]

On 13.09.23 18:21, Gylstorff Quirin wrote:
> 
> 
> On 9/13/23 03:26, MOESSBAUER, Felix (T CED INW-CN) wrote:
>> On Tue, 2023-09-12 at 15:55 +0530, Jan Kiszka wrote:
>>> On 12.09.23 15:37, Gylstorff Quirin wrote:
>>>>
>>>>
>>>> On 8/28/23 08:47, Jan Kiszka wrote:
>>>>> On 23.08.23 15:28, Quirin Gylstorff wrote:
>>>>>> From: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>>>>
>>>>>> Disable the timestamp of the do_install() task to
>>>>>> re-execute when ever a dependency on do_install
>>>>>> is executed.
>>>>>>
>>>>>> Without this change the sstate needs to be clean for
>>>>>> every rebuild.
>>>>>>
>>>>>> Signed-off-by: Quirin Gylstorff <quirin.gylstorff@siemens.com>
>>>>>> ---
>>>>>>    .../initramfs-verity-hook/initramfs-verity-
>>>>>> hook_0.1.bb         | 3 +++
>>>>>>    1 file changed, 3 insertions(+)
>>>>>>
>>>>>> diff --git
>>>>>> a/recipes-initramfs/initramfs-verity-hook/initramfs-verity-
>>>>>> hook_0.1.bb b/recipes-initramfs/initramfs-verity-
>>>>>> hook/initramfs-verity-hook_0.1.bb
>>>>>> index 3fc63ed..7435649 100644
>>>>>> ---
>>>>>> a/recipes-initramfs/initramfs-verity-hook/initramfs-verity-
>>>>>> hook_0.1.bb
>>>>>> +++
>>>>>> b/recipes-initramfs/initramfs-verity-hook/initramfs-verity-
>>>>>> hook_0.1.bb
>>>>>> @@ -40,6 +40,9 @@ do_install[cleandirs] += " \
>>>>>>        ${D}/usr/share/initramfs-tools/scripts/local-top \
>>>>>>        ${D}/usr/share/initramfs-tools/conf-hooks.d"
>>>>>>    +# Ensure VERITY_ENV_FILE is updated also when rebuilding
>>>>>> initramfs-verity-hook
>>>>>> +do_install[nostamp] = "1"
>>>>>> +
>>>>>>    do_install() {
>>>>>>        # Insert the veritysetup commandline into the script
>>>>>>        if [ -f "${VERITY_ENV_FILE}" ]; then
>>>>>
>>>>> No other dpkg-raw packages require this, do they? What makes
>>>>> initramfs
>>>>> packages special here? And what about those in isar upstream?
>>>>>
>>>>
>>>> This is necessary for the initramfs package which contain the
>>>> checksum/uuid from the root file system build.
>>>> This ensure that the checksum/uuid is refreshed.
>>>>
>>>> It fixes the problem that an rebuild of a verity image leads to a
>>>> broken
>>>> boot after the build.
>>>
>>> And what will happen when the user decides to pin the UUID to fix
>>> value
>>> in order to have reproducibility? Won't that cause a needless re-run?
>>> Shouldn't this recipe rather make sure to depend on the UUID value?
>>
> Even with pinning and same salt and uuid veritysetup will generate a new
> roothash during the build.
> 

That sounds like other reproducibility issues we have, eg. with mkfs.*
Can't that be fixed? I think it has to, otherwise dm-verity images will
never be fully reproducible.

> Quirin
> 
> 
>> This is a general reproducibility topic: The task checksums are
>> computed at parsing time and by that the tasks that need to be re-run
>> are selected. However, this is based on the assumption that tasks are
>> deterministic, i.e. same input data also generates same output data.
>>
>> Unfortunately we are still not 100% reproducible. By that this pre-
>> condition is not met. For normal packages this does not really matter,
>> but for the dm-verity checksum this very well matters.
>>
> 
>> I'm just wondering if we could work around that by making this
>> component non-cacheable. Putting Adriaan in CC for the sstate part.
> 
> The same  solution as is used by meta-security for Yocto[1].
> 
> [1]: https://lists.yoctoproject.org/g/yocto/message/50624

Interesting - Yocto is NOT reproducible when it comes to verity...

Jan

-- 
Siemens AG, Technology
Linux Expert Center