license manifest files effectively disappear when do_rootfs is not rerun

license manifest files effectively disappear when do_rootfs is not rerun

[Christian Thießen]

(Cross-post: I first posted to https://lists.yoctoproject.org/g/yocto/message/58392, then realized that according to the poky readme this list is more appropriate. Sorry, beginner's mistake)

Dear list,
when building an image with a Yocto setup based on kirkstone, it seems I've come across an interesting corner case: The image built successfully, the build/tmp/deploy/licenses/${IMAGE}-${MACHINE} symlink got updated as expected, but that new directory contained only image_license.manifest. license.manifest and package.manifest were missing.
I've discovered that these latter files are generated by the image's do_rootfs task (ROOTFS_POSTPROCESS_COMMAND:prepend = "write_package_manifest; license_create_manifest; " in poky/meta/classes/license_image.bbclass). This task apparently did not need to be rerun in that particular build.
This behavior is at least a bit surprising to the user; bitbake claims to have run everything that needed to be run, completes successfully and then some files are just not there. It might be what happened here https://lists.yoctoproject.org/g/yocto/message/49600 too. For me, it broke a CI build that ran due to an environment change that didn't affect the image and then failed to deploy the manifest files. I could run bitbake with -C rootfs as a workaround, but that causes a warning and the problem could probably better be fixed within the oe-core classes. I'm not sure how though: Could the manifest generation be moved to a different point in the build, e.g. where image_license.manifest is generated? Or should build/tmp/deploy/licenses/${IMAGE}-${MACHINE} become a directory containing symlinks to files, instead of a symlink to a new directory, so that things that don't need to be updated can be retained? Maybe you have a completely different idea? I'll leave that up for discussion.
Thank you,

Christian Thießen

Re: [OE-core] license manifest files effectively disappear when do_rootfs is not rerun

[Alexander Kanavin]

Generally, you should not be deleting things from tmp/deploy (or tmp/
for that matter) - either delete all of tmp/ and rely on sstate or
don't touch it at all, and just re-run the build.

It helps if you provide steps to reproduce this behavior.

Alex

On Mon, 24 Oct 2022 at 10:36, Christian Thießen via
lists.openembedded.org
<christian.thiessen=airbus.com@lists.openembedded.org> wrote:
>
> (Cross-post: I first posted to https://lists.yoctoproject.org/g/yocto/message/58392, then realized that according to the poky readme this list is more appropriate. Sorry, beginner's mistake)
>
> Dear list,
> when building an image with a Yocto setup based on kirkstone, it seems I've come across an interesting corner case: The image built successfully, the build/tmp/deploy/licenses/${IMAGE}-${MACHINE} symlink got updated as expected, but that new directory contained only image_license.manifest. license.manifest and package.manifest were missing.
> I've discovered that these latter files are generated by the image's do_rootfs task (ROOTFS_POSTPROCESS_COMMAND:prepend = "write_package_manifest; license_create_manifest; " in poky/meta/classes/license_image.bbclass). This task apparently did not need to be rerun in that particular build.
> This behavior is at least a bit surprising to the user; bitbake claims to have run everything that needed to be run, completes successfully and then some files are just not there. It might be what happened here https://lists.yoctoproject.org/g/yocto/message/49600 too. For me, it broke a CI build that ran due to an environment change that didn't affect the image and then failed to deploy the manifest files. I could run bitbake with -C rootfs as a workaround, but that causes a warning and the problem could probably better be fixed within the oe-core classes. I'm not sure how though: Could the manifest generation be moved to a different point in the build, e.g. where image_license.manifest is generated? Or should build/tmp/deploy/licenses/${IMAGE}-${MACHINE} become a directory containing symlinks to files, instead of a symlink to a new directory, so that things that don't need to be updated can be retained? Maybe you have a completely different idea? I'll leave that up for discussion.
> Thank you,
>
> Christian Thießen
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#172077): https://lists.openembedded.org/g/openembedded-core/message/172077
> Mute This Topic: https://lists.openembedded.org/mt/94530086/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: license manifest files effectively disappear when do_rootfs is not rerun

[Christian Thießen]

I'm not deleting anything.
Steps to reproduce would be:
Build just about any image for any machine, as long as it uses the standard classes like license_image.bbclass.
You'll get a build/tmp/deploy/licenses/${IMAGE}-${MACHINE} symlink pointing to build/tmp/deploy/licenses/${IMAGE}-${MACHINE}-<datetime> containing all the expected manifests.
Run the same build again. Bitbake will rerun do_populate_lic_deploy, but not do_rootfs.
build/tmp/deploy/licenses/${IMAGE}-${MACHINE} now points to a new directory that contains only image_license.manifest.

Christian

Re: [OE-core] license manifest files effectively disappear when do_rootfs is not rerun

[Alexander Kanavin]

If I run the same build again, no tasks will be executed, and
accordingly nothing in tmp/deploy will change:

alex@Zen2:/srv/work/alex/poky/build-64$ bitbake core-image-minimal
...
NOTE: Tasks Summary: Attempted 2862 tasks of which 2862 didn't need to
be rerun and all succeeded.

alex@Zen2:/srv/work/alex/poky/build-64$ ls -l
tmp/deploy/licenses/core-image-minimal-qemux86-64/
total 16
-rw-r--r-- 1 alex alex  256 Oct 24 11:33 image_license.manifest
-rw-r--r-- 1 alex alex 6451 Oct 24 11:32 license.manifest
-rw-r--r-- 1 alex alex  958 Oct 24 11:32 package.manifest


So something happens between builds on your side, and there should be
a reproducer for it.

Alex


On Mon, 24 Oct 2022 at 10:58, Christian Thießen via
lists.openembedded.org
<christian.thiessen=airbus.com@lists.openembedded.org> wrote:
>
> I'm not deleting anything.
> Steps to reproduce would be:
> Build just about any image for any machine, as long as it uses the standard classes like license_image.bbclass.
> You'll get a build/tmp/deploy/licenses/${IMAGE}-${MACHINE} symlink pointing to build/tmp/deploy/licenses/${IMAGE}-${MACHINE}-<datetime> containing all the expected manifests.
> Run the same build again. Bitbake will rerun do_populate_lic_deploy, but not do_rootfs.
> build/tmp/deploy/licenses/${IMAGE}-${MACHINE} now points to a new directory that contains only image_license.manifest.
>
> Christian
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#172079): https://lists.openembedded.org/g/openembedded-core/message/172079
> Mute This Topic: https://lists.openembedded.org/mt/94530086/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: license manifest files effectively disappear when do_rootfs is not rerun

[Christian Thießen]

Thanks, you're right; a normal rebuild doesn't execute any tasks. I've now found that this is related to rm_work: If you enable it with INHERIT+=" rm_work" in local.conf, bitbake will rerun three tasks on each rebuild: populate_lic_deploy, rm_work and build. After that you get the missing manifests.
So another option for a workaround would be to exclude the image recipe from rm_work.

Christian

Re: [OE-core] license manifest files effectively disappear when do_rootfs is not rerun

[Quentin Schulz]

Hi Christian,

On 10/24/22 13:05, Christian Thießen via lists.openembedded.org wrote:
> Thanks, you're right; a normal rebuild doesn't execute any tasks. I've now found that this is related to rm_work: If you enable it with INHERIT+=" rm_work" in local.conf, bitbake will rerun three tasks on each rebuild: populate_lic_deploy, rm_work and build. After that you get the missing manifests.
> So another option for a workaround would be to exclude the image recipe from rm_work.
> 

Good job at identifying the culprit, would you mind sending us a patch 
and/or open a bug on our Bugzilla to keep track of that?

Cheers,
Quentin

Re: license manifest files effectively disappear when do_rootfs is not rerun

[Christian Thießen]

I don't feel familiar enough with the inner workings of oe-core to decide what a good (robust, maintainable) solution would be, but I've created a bug report with some suggestions:
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14946