Re: [OE-core] [PATCH v2 0/6] Use the kernel from sstate when building fitImages

Alexandre Belloni
2024-07-24T15:54:50Z

Hello,

I'm pretty sure this causes this failure:

https://autobuilder.yoctoproject.org/typhoon/#/builders/151/builds/2005/steps/12/logs/stdio

And also those oe-selftest failures:

https://autobuilder.yoctoproject.org/typhoon/#/builders/80/builds/6945/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/79/builds/6987/steps/15/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/7000/steps/14/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/86/builds/7010/steps/14/logs/stdio

On 15/07/2024 16:10:37+0200, Adrian Freihofer wrote:
> Changes in comparison to v1:
> - Add the missing dependency from do_image_wic to
> initramfs.do_image_complete which caused the build failure on the AB.
> - Support symbolic links for $dtb_path again, as suggested by Mark.
> - Simplify the refactoring of the uboot_prep_kimage function.
> - Evaluate staging via sysroot instead of taking the kernel artifacts
> from the deploy folder. But there is an issue with this approach: It
> does not allow to remove the dependency from do_deploy on do_install.
> - Rebasing to latest master
>
> If the KERNEL_IMAGETYPES(S) contains fitImage, the kernel is always
> rebuilt when something changes in the initramfs frequently.
> This is even worse if the build runs from an empty TMPDIR. The kernel
> re-build starts by fetching the large kernel git repository and
> recompiling it from scratch.
>
> This cannot be improved if INITRAMFS_IMAGE_BUNDLE = "1". If the kernel
> Makefile is needed to generate the initramfs bundle the kernel build
> folder is required.
> But for a build configuration with INITRAMFS_IMAGE_BUNDLE = "" the
> build folder is not needed. Creating the initramfs bundle requires:
> linux.bin, DTBs and the initramfs which are available in the deploy
> directory. That means creating the fitImage can be done with artifacts
> which are already cached by the sstate.
>
> There is an extra commit providing a html file. This html file provides
> some graphics which show the changes in the kernel's task dependencies.
> I hope this simplifies the re-view of this patch series.
>
> Testing:
> - oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg \
> reproducible -T yocto-mirrors -T machine -T toolchain-user \
> -T toolchain-system
> - Build for core-image-minimal for MACHINE = "genericarm64"
> (failure from AB was reproducible and is fixed now)
> - Build clean followed by build from sstate
> Note: Adding this test to oe-selftest failed because sstate is read only
> - Clean build config:
> KERNEL_IMAGETYPE = "Image"
> KERNEL_IMAGETYPES += " fitImage "
> KERNEL_CLASSES = " kernel-fitimage "
> IMAGE_FSTYPES += "cpio.gz"
> INITRAMFS_IMAGE = "core-image-minimal"
> IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""
> UBOOT_RD_LOADADDRESS = "0x88000000"
> UBOOT_RD_ENTRYPOINT = "0x88000000"
> UBOOT_LOADADDRESS = "0x80080000"
> UBOOT_ENTRYPOINT = "0x80080000"
> FIT_DESC = "A model description"
> FOO_VAR = "1"
> INHERIT += "image-buildinfo"
> IMAGE_BUILDINFO_VARS:append = " FOO_VAR"
> - Append the following and rebuild with sstate:
> FOO_VAR = "2"
> TMPDIR = "${TOPDIR}/tmp-2"
> - Check the log files:
> - tmp/log/cooker/qemux86-64 contains:
> linux-yocto-6.6.35+git-r0: task do_deploy: Succeeded
> linux-yocto-6.6.35+git-r0: task do_deploy_fitimage_unbundled: Succeeded
> - tmp-2/log/cooker/qemux86-64 contains:
> linux-yocto-6.6.35+git-r0: task do_deploy_setscene: Succeeded
> linux-yocto-6.6.35+git-r0: task do_deploy_fitimage_unbundled: Succeeded
> - To re-view the chagnes in the task dependencies, the script bellow
> has been used. It confirms:
> - For builds with fitImage and unbundled initramfs:
> - the do_deploy_fitimage_unbundled task runs after do_deploy
> - do_assemble_fitimage_initramfs is not executed
> - do_bundle_initramfs is not executed
> - It works for fitImage in KERNEL_IMAGETYPE as well as for fitImage
> in KERNEL_IMAGETYPES
> - For builds with fitImage and bundled initramfs: No changes
> - For builds with fitImage, without initramfs:
> - do_assemble_fitimage_initramfs is not executed
> - do_bundle_initramfs is not executed
>
> OUTPUT_FILE=task-depends.md
> GIT_BRANCH=master
> GIT_BRANCH_NEXT=adrianf/kernel-fitimage-sstate
>
> echo "# Task dependeny changes" > "$OUTPUT_FILE"
> echo "" >> "$OUTPUT_FILE"
>
> run_bitbake(){
> echo "$1" >> "$OUTPUT_FILE"
> echo "" >> "$OUTPUT_FILE"
>
> bitbake virtual/kernel -g
> grep -E '(digraph depends|linux-yocto|\})' task-depends.dot \
> | grep -v -E '(spdx|do_kernel_configcheck|do_prepare_recipe_sysroot|do_populate_lic|native.do_populate_sysroot)' \
> | grep -v -E '(depmodwrapper-cross.do_populate_sysroot|binutils-cross-x86_64.do_populate_sysroot|do_package)' \
> | grep -v -E '(do_validate_branches|do_unpack|do_recipe_qa|do_patch|do_kernel_checkout|do_kernel_configme|do_kernel_metadata)' \
> | grep -v -E '(do_sizecheck|do_strip|do_compile_kernelmodules|do_shared_workdir|do_transform_kernel|do_kernel_link_images)' \
> | grep -v -E '(do_kernel_version_sanity_check|do_symlink_kernsrc|do_deploy_source_date_epoch|do_configure|do_fetch)' \
> | sed -e 's;\\n\:.*.bb;;g' -e 's;linux-yocto[. ];;g' > task-depends-filtered.dot
>
> echo '```plantuml' >> "$OUTPUT_FILE"
> cat task-depends-filtered.dot >> "$OUTPUT_FILE"
> echo '```' >> "$OUTPUT_FILE"
> echo "" >> "$OUTPUT_FILE"
> }
>
> run_bitbake_per_branch(){
> echo "## Configuration: $1" >> "$OUTPUT_FILE"
> echo "" >> "$OUTPUT_FILE"
> echo '```raw' >> "$OUTPUT_FILE"
> cat build/conf/auto.conf >> "$OUTPUT_FILE"
> echo '```' >> "$OUTPUT_FILE"
> echo "" >> "$OUTPUT_FILE"
>
> git checkout $GIT_BRANCH
> run_bitbake "### branch: $GIT_BRANCH"
> mv -f task-depends-filtered.dot task-depends-filtered-old.dot
>
> git checkout $GIT_BRANCH_NEXT
> run_bitbake "### branch: $GIT_BRANCH_NEXT"
>
> echo "## Diff" >> "$OUTPUT_FILE"
> echo "" >> "$OUTPUT_FILE"
> echo '```patch' >> "$OUTPUT_FILE"
> diff task-depends-filtered-old.dot task-depends-filtered.dot >> "$OUTPUT_FILE"
> echo '```' >> "$OUTPUT_FILE"
> echo "" >> "$OUTPUT_FILE"
> }
>
> cat << EOF > build/conf/auto.conf
> KERNEL_IMAGETYPE = "Image"
> KERNEL_IMAGETYPES += " fitImage "
> KERNEL_CLASSES = " kernel-fitimage "
> IMAGE_FSTYPES += "cpio.gz"
> INITRAMFS_IMAGE = "core-image-minimal-initramfs"
> EOF
> run_bitbake_per_branch "image, fitimage, initramfs, unbundled"
>
> cat << EOF > build/conf/auto.conf
> KERNEL_IMAGETYPE:forcevariable = "fitImage"
> KERNEL_CLASSES = " kernel-fitimage "
> IMAGE_FSTYPES += "cpio.gz"
> INITRAMFS_IMAGE = "core-image-minimal-initramfs"
> EOF
> run_bitbake_per_branch "fitimage, initramfs, unbundled"
>
> cat << EOF > build/conf/auto.conf
> KERNEL_IMAGETYPE = "Image"
> KERNEL_IMAGETYPES += " fitImage "
> KERNEL_CLASSES = " kernel-fitimage "
> IMAGE_FSTYPES += "cpio.gz"
> INITRAMFS_IMAGE = "core-image-minimal-initramfs"
> INITRAMFS_IMAGE_BUNDLE = "1"
> EOF
> run_bitbake_per_branch "image, fitimage, initramfs, bundled"
>
> cat << EOF > build/conf/auto.conf
> KERNEL_IMAGETYPE = "Image"
> KERNEL_IMAGETYPES += " fitImage "
> KERNEL_CLASSES = " kernel-fitimage "
> EOF
> run_bitbake_per_branch "image, fitimage"
>
> rm -f build/conf/auto.conf task-depends-filtered-old.dot task-depends-filtered.dot task-depends.dot
>
> Adrian Freihofer (6):
> kernel-fitimage: fix intentation
> kernel-fitimage: fix external dtb check
> kernel: refactor linux compression
> kernel-fitimage: refactor fitimage_assemble
> kernel: refactor fitimage
> kernel-fitimage: run unbundled fitimage after deploy
>
> meta/classes-recipe/image.bbclass | 12 +-
> meta/classes-recipe/kernel-fitimage.bbclass | 188 ++++++++++++--------
> meta/classes-recipe/kernel-uboot.bbclass | 1 +
> meta/classes-recipe/kernel.bbclass | 31 ++--
> 4 files changed, 148 insertions(+), 84 deletions(-)
>
> --
> 2.45.2
>

>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#201926): https://lists.openembedded.org/g/openembedded-core/message/201926
> Mute This Topic: https://lists.openembedded.org/mt/107231736/3617179
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

--
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering