[PATCH v2 0/6] Use the kernel from sstate when building fitImages

[Adrian Freihofer <adrian.freihofer@gmail.com>]

Changes in comparison to v1:
- Add the missing dependency from do_image_wic to
  initramfs.do_image_complete which caused the build failure on the AB.
- Support symbolic links for $dtb_path again, as suggested by Mark.
- Simplify the refactoring of the uboot_prep_kimage function.
- Evaluate staging via sysroot instead of taking the kernel artifacts
  from the deploy folder. But there is an issue with this approach: It
  does not allow to remove the dependency from do_deploy on do_install.
- Rebasing to latest master

If the KERNEL_IMAGETYPES(S) contains fitImage, the kernel is always
rebuilt when something changes in the initramfs frequently.
This is even worse if the build runs from an empty TMPDIR. The kernel
re-build starts by fetching the large kernel git repository and
recompiling it from scratch.

This cannot be improved if INITRAMFS_IMAGE_BUNDLE = "1". If the kernel
Makefile is needed to generate the initramfs bundle the kernel build
folder is required.
But for a build configuration with INITRAMFS_IMAGE_BUNDLE = "" the
build folder is not needed. Creating the initramfs bundle requires:
linux.bin, DTBs and the initramfs which are available in the deploy
directory. That means creating the fitImage can be done with artifacts
which are already cached by the sstate.

There is an extra commit providing a html file. This html file provides
some graphics which show the changes in the kernel's task dependencies.
I hope this simplifies the re-view of this patch series.

Testing:
- oe-selftest -a --skip-tests distrodata.Distrodata.test_checkpkg \
     reproducible -T yocto-mirrors -T machine -T toolchain-user \
     -T toolchain-system
- Build for core-image-minimal for MACHINE = "genericarm64"
  (failure from AB was reproducible and is fixed now)
- Build clean followed by build from sstate
  Note: Adding this test to oe-selftest failed because sstate is read only
  - Clean build config:
    KERNEL_IMAGETYPE = "Image"
    KERNEL_IMAGETYPES += " fitImage "
    KERNEL_CLASSES = " kernel-fitimage "
    IMAGE_FSTYPES += "cpio.gz"
    INITRAMFS_IMAGE = "core-image-minimal"
    IMAGE_NAME_SUFFIX:pn-core-image-minimal = ""
    UBOOT_RD_LOADADDRESS = "0x88000000"
    UBOOT_RD_ENTRYPOINT = "0x88000000"
    UBOOT_LOADADDRESS = "0x80080000"
    UBOOT_ENTRYPOINT = "0x80080000"
    FIT_DESC = "A model description"
    FOO_VAR = "1"
    INHERIT += "image-buildinfo"
    IMAGE_BUILDINFO_VARS:append = " FOO_VAR"
  - Append the following and rebuild with sstate:
    FOO_VAR = "2"
    TMPDIR = "${TOPDIR}/tmp-2"
  - Check the log files:
    - tmp/log/cooker/qemux86-64 contains:
      linux-yocto-6.6.35+git-r0: task do_deploy: Succeeded
      linux-yocto-6.6.35+git-r0: task do_deploy_fitimage_unbundled: Succeeded
    - tmp-2/log/cooker/qemux86-64 contains:
      linux-yocto-6.6.35+git-r0: task do_deploy_setscene: Succeeded
      linux-yocto-6.6.35+git-r0: task do_deploy_fitimage_unbundled: Succeeded
- To re-view the chagnes in the task dependencies, the script bellow
  has been used. It confirms:
  - For builds with fitImage and unbundled initramfs:
    - the do_deploy_fitimage_unbundled task runs after do_deploy
    - do_assemble_fitimage_initramfs is not executed
    - do_bundle_initramfs is not executed
    - It works for fitImage in KERNEL_IMAGETYPE as well as for fitImage
      in KERNEL_IMAGETYPES
  - For builds with fitImage and bundled initramfs: No changes
  - For builds with fitImage, without initramfs:
    - do_assemble_fitimage_initramfs is not executed
    - do_bundle_initramfs is not executed

OUTPUT_FILE=task-depends.md
GIT_BRANCH=master
GIT_BRANCH_NEXT=adrianf/kernel-fitimage-sstate

echo "# Task dependeny changes" > "$OUTPUT_FILE"
echo "" >> "$OUTPUT_FILE"

run_bitbake(){
    echo "$1" >> "$OUTPUT_FILE"
    echo "" >> "$OUTPUT_FILE"

    bitbake virtual/kernel -g
    grep -E '(digraph depends|linux-yocto|\})' task-depends.dot \
    | grep -v -E '(spdx|do_kernel_configcheck|do_prepare_recipe_sysroot|do_populate_lic|native.do_populate_sysroot)' \
    | grep -v -E '(depmodwrapper-cross.do_populate_sysroot|binutils-cross-x86_64.do_populate_sysroot|do_package)' \
    | grep -v -E '(do_validate_branches|do_unpack|do_recipe_qa|do_patch|do_kernel_checkout|do_kernel_configme|do_kernel_metadata)' \
    | grep -v -E '(do_sizecheck|do_strip|do_compile_kernelmodules|do_shared_workdir|do_transform_kernel|do_kernel_link_images)' \
    | grep -v -E '(do_kernel_version_sanity_check|do_symlink_kernsrc|do_deploy_source_date_epoch|do_configure|do_fetch)' \
    | sed -e 's;\\n\:.*.bb;;g' -e 's;linux-yocto[. ];;g' > task-depends-filtered.dot

    echo '```plantuml' >> "$OUTPUT_FILE"
    cat task-depends-filtered.dot >> "$OUTPUT_FILE"
    echo '```' >> "$OUTPUT_FILE"
    echo "" >> "$OUTPUT_FILE"
}

run_bitbake_per_branch(){
    echo "## Configuration: $1" >> "$OUTPUT_FILE"
    echo "" >> "$OUTPUT_FILE"
    echo '```raw' >> "$OUTPUT_FILE"
    cat build/conf/auto.conf >> "$OUTPUT_FILE"
    echo '```' >> "$OUTPUT_FILE"
    echo "" >> "$OUTPUT_FILE"

    git checkout $GIT_BRANCH
    run_bitbake "### branch: $GIT_BRANCH"
    mv -f task-depends-filtered.dot task-depends-filtered-old.dot

    git checkout $GIT_BRANCH_NEXT
    run_bitbake "### branch: $GIT_BRANCH_NEXT"

    echo "## Diff" >> "$OUTPUT_FILE"
    echo "" >> "$OUTPUT_FILE"
    echo '```patch' >> "$OUTPUT_FILE"
    diff task-depends-filtered-old.dot task-depends-filtered.dot >> "$OUTPUT_FILE"
    echo '```' >> "$OUTPUT_FILE"
    echo "" >> "$OUTPUT_FILE"
}

cat << EOF > build/conf/auto.conf
KERNEL_IMAGETYPE = "Image"
KERNEL_IMAGETYPES += " fitImage "
KERNEL_CLASSES = " kernel-fitimage "
IMAGE_FSTYPES += "cpio.gz"
INITRAMFS_IMAGE = "core-image-minimal-initramfs"
EOF
run_bitbake_per_branch "image, fitimage, initramfs, unbundled"

cat << EOF > build/conf/auto.conf
KERNEL_IMAGETYPE:forcevariable = "fitImage"
KERNEL_CLASSES = " kernel-fitimage "
IMAGE_FSTYPES += "cpio.gz"
INITRAMFS_IMAGE = "core-image-minimal-initramfs"
EOF
run_bitbake_per_branch "fitimage, initramfs, unbundled"

cat << EOF > build/conf/auto.conf
KERNEL_IMAGETYPE = "Image"
KERNEL_IMAGETYPES += " fitImage "
KERNEL_CLASSES = " kernel-fitimage "
IMAGE_FSTYPES += "cpio.gz"
INITRAMFS_IMAGE = "core-image-minimal-initramfs"
INITRAMFS_IMAGE_BUNDLE = "1"
EOF
run_bitbake_per_branch "image, fitimage, initramfs, bundled"

cat << EOF > build/conf/auto.conf
KERNEL_IMAGETYPE = "Image"
KERNEL_IMAGETYPES += " fitImage "
KERNEL_CLASSES = " kernel-fitimage "
EOF
run_bitbake_per_branch "image, fitimage"

rm -f build/conf/auto.conf task-depends-filtered-old.dot task-depends-filtered.dot task-depends.dot

Adrian Freihofer (6):
  kernel-fitimage: fix intentation
  kernel-fitimage: fix external dtb check
  kernel: refactor linux compression
  kernel-fitimage: refactor fitimage_assemble
  kernel: refactor fitimage
  kernel-fitimage: run unbundled fitimage after deploy

 meta/classes-recipe/image.bbclass           |  12 +-
 meta/classes-recipe/kernel-fitimage.bbclass | 188 ++++++++++++--------
 meta/classes-recipe/kernel-uboot.bbclass    |   1 +
 meta/classes-recipe/kernel.bbclass          |  31 ++--
 4 files changed, 148 insertions(+), 84 deletions(-)

-- 
2.45.2