[yocto-autobuilder-helper][PATCH v4 00/11] Implement 'containers' jobs

[Tim Orling <ticotimo@gmail.com>]

This series builds on top of the patches already merged from v2 and
previously submitted as v3.

This series contains the 'yocto-autobuilder-helper' patches to enable:
* building, testing and publishing of 'vcontainer-tarball'
  - vcontainer-tests job
* building and pushing containers from:
  - meta-virtualization
* containers are pushed when the push_containers boolean is set in the
  AutoBuilder UI
  - regstries are set with CONTAINER_REGISTRIES variable
* authentication uses local user credentials (e.g. ~/.docker/config.json
  or .../auth.json for podman)
* containers are tagged with PV_MAJOR, PV_MAJOR.PV_MINOR, DISTRO_CODENAME
  and yocto-x.y release (yocto-x.y.z for release builds).
* multiarch containers built for x86-64 and aarch64
* signing of containers
* attaching SPDX 3.0 SBOM attestation (and signing it) for pushed
  containers
* 'dev' mode containers which run as 'root' user and in some cases add a
  shell. These are tagged as above but with a '-dev' suffix.

Additional features, such as attaching SLSA provenance attestations will
come in a future series.

Concerns about the possible infinite loop around the tarball extraction
"while True, try, expect OSError" code [1] are addressed separately [2],
since the original patch has been merged.

This series relies on the resolution of [3].

The 'vcontainer-tests' job is currently experiencing some failures [4],
which will have to be resolved separately.

[1] https://lore.kernel.org/yocto-patches/02f0c35f16a8a51bbe0e89145353008f67180a84.camel@pbarker.dev/
[2] https://lore.kernel.org/yocto-patches/20260606011918.61582-1-tim.orling@konsulko.com/
[3] https://lore.kernel.org/yocto-meta-virtualization/cover.1783356922.git.tim.orling@konsulko.com/
[4] https://autobuilder.yoctoproject.org/valkyrie/?#/builders/118/builds/4

Changes in v2:
* Simplify by merging vdkr-tests and vpdmn-tests into vcontainer-tests
* Simplify by building containers from meta-virtualization:
- Depends on resolution of: https://lists.yoctoproject.org/g/meta-virtualization/message/9826
* Workaround for recent (since Friday May 29, 2026) errors:
  Error: reading blob sha256:<hash>: file integrity checksum failed for
  "<file>"
  - This is probably related to either sstate changes or recent changes
    in vcontainer-common...

Changes in v3:
* vcontainer-tarball is installed in jobs that set "vcontainer" similar
  to "extratools", instead of the more global "buildtools" behavior.
* The (rather large amount of) code for pushing containers in run-config
  is broken out into a dedicated run-push-containers script.
* Workaround for pushing containers is moved to run-push-containers. The
  most recent behavior (on top of meta-virt master-next) showed a
  different error pattern:
  Error: reading blob sha256:<hash>: EOF
  The workaround simply removes all container images from the VM's
  container-registry before importing or pushing new container images.
  The commit message was reworded accordingly.
* Rather than installing latest versions of pip, setuptools and wheel,
  silence the pip warning with PIP_DISABLE_PIP_VERSION_CHECK=1
* Rather than installing the latest versions of pytest, pytest-timeout
  and pexpect, install from meta-virtualization/tests/requirements.txt.
* Refactor run-vcontainer-tests script to use optargs instead of a mix of
  positional args and env vars.

Changes in v4:
* Adjusted the 'vcontainer' installation of the vcontainer-tarball to
  only apply to 'push-containers' step, as we currently only need the
  functionality to 'vimport' and push single-arch containers.
* Added app-container-alpine as a single-arch container example
* Switched other container builds to multi-arch, building both x86-64 and
  aarch64 flavors
* Added conditional signing of containers in the run-push-containers
  script, this is gated by CONTAINER_COSIGN_KEY being set to the path to
  a 'cosign' private key.
* Added conditional attachment of SPDX 3.0 SBOM attestation (when
  CONTAINER_COSIGN_KEY is defined).
* Added conditional verification of the SPDX 3.0 SBOM attestation, which
  is gated by CONTAINER_COSIGN_PUB being set to the path to a 'cosign'
  public key.
* Added building of '-dev' mode containers, with PACKAGECONFIG 'dev',
  which run as 'root' user instead of the 'nonroot' user and in some
  cases conditionally add a shell (depends on the container image recipe).
* Added pushing of '-dev'containers adding a '-dev' suffix to the tags. 

Tim Orling (11):
  scripts: add vcontainer-tarball setup, integration, and publishing
  config.json: add vcontainer-tarball build target
  scripts: add run-vcontainer-tests for meta-virtualization
  scripts: add container registry push, auth, tagging, runtime selection
  config.json: add 'containers-library' build job
  scripts/run-push-containers: push multiarch containers with
    skopeo-native
  containers-library: switch to multiarch; add alpine single-arch
  run-push-containers: conditionally sign pushed containers with cosign
  scripts/run-push-containers: add SPDX SBOM attestation
  config.json: add -dev container builds to containers-library
  run-push-containers: optionally push -dev tagged containers

 config.json                  | 168 +++++++++++++
 scripts/publish-artefacts    |   5 +
 scripts/run-config           |  19 ++
 scripts/run-push-containers  | 471 +++++++++++++++++++++++++++++++++++
 scripts/run-vcontainer-tests | 212 ++++++++++++++++
 scripts/utils.py             |  20 +-
 6 files changed, 891 insertions(+), 4 deletions(-)
 create mode 100755 scripts/run-push-containers
 create mode 100755 scripts/run-vcontainer-tests

These changes can be found:
https://git.yoctoproject.org/yocto-autobuilder-helper/log/?h=contrib/timo/containers-upstream-v4

These changes had a successful build of containers-library job:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/117/builds/25
(with one additional commit to set the path to VCONTAINER_TARBALL_URL)

-- 
2.54.0