* [yocto-autobuilder2][PATCH v3 1/6] scripts: add vcontainer-tarball setup, integration, and publishing
2026-06-06 2:51 [yocto-autobuilder2][PATCH v3 0/6] Implement 'containers' jobs tim.orling
@ 2026-06-06 2:51 ` tim.orling
2026-06-06 2:51 ` [yocto-autobuilder2][PATCH v3 2/6] config.json: add vcontainer-tarball build target tim.orling
` (4 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: tim.orling @ 2026-06-06 2:51 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Introduce the vcontainer-tarball SDK plumbing used by container build
jobs. The vcontainer-tarball is a meta-virtualization-derived SDK
(modelled after buildtools-tarball) that ships the container build
toolchain so worker jobs do not need to rebuild it for every step.
* scripts/utils.py: add setup_vcontainer_tarball(), and add an
env_glob keyword argument to setup_tools_tarball() and
enable_tools_tarball() so the vcontainer-tarball can source its
specific environment-setup-ci file rather than the universal
glob.
* scripts/run-config: source the vcontainer-tarball environment
for build-targets when "vcontainer" parameter is passed in,
similar to "extratools" behavior.
* scripts/publish-artefacts: publish the vcontainer-tarball
artefact so downstream test jobs can fetch a stable SDK.
AI-Generated: Claude Cowork Opus 4.8
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
scripts/publish-artefacts | 5 +++++
scripts/run-config | 11 +++++++++++
scripts/utils.py | 20 ++++++++++++++++----
3 files changed, 32 insertions(+), 4 deletions(-)
diff --git a/scripts/publish-artefacts b/scripts/publish-artefacts
index e56e131..0e820e9 100755
--- a/scripts/publish-artefacts
+++ b/scripts/publish-artefacts
@@ -146,5 +146,10 @@ case "$target" in
sha256sums $TMPDIR/deploy/images/qemux86-64
cp -R --no-dereference --preserve=links $TMPDIR/deploy/images/qemux86-64/*qemux86* $DEST/patchtest
;;
+ "vcontainer-tarball")
+ mkdir -p $DEST/vcontainer-tarball
+ sha256sums $TMPDIR/deploy/sdk
+ cp -R --no-dereference --preserve=links $TMPDIR/deploy/sdk/*vcontainer* $DEST/vcontainer-tarball
+ ;;
esac
diff --git a/scripts/run-config b/scripts/run-config
index e896234..90a5996 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -153,6 +153,17 @@ else:
if args.phase == "init" and args.stepname == "buildtools":
sys.exit(0)
+vcontainer = utils.getconfigvar("vcontainer", ourconfig, args.target)
+if jcfg:
+ if vcontainer:
+ addentry("vcontainer", "Setup vcontainer tarball", "init")
+elif vcontainer:
+ # vcontainer is opt-in per target via the "vcontainer" config variable,
+ # so this is a no-op for targets which don't set it
+ utils.setup_vcontainer_tarball(ourconfig, args.target, args.builddir + "/../vcontainer-tarball")
+ if args.phase == "init" and args.stepname == "vcontainer":
+ sys.exit(0)
+
extratools = utils.getconfigvar("extratools", ourconfig, args.target)
if jcfg:
if extratools:
diff --git a/scripts/utils.py b/scripts/utils.py
index a4dd12e..bddc715 100644
--- a/scripts/utils.py
+++ b/scripts/utils.py
@@ -456,8 +456,8 @@ def sha256_file(filename):
pass
return method.hexdigest()
-def enable_tools_tarball(btdir, name):
- btenv = glob.glob(btdir + "/environment-setup*")
+def enable_tools_tarball(btdir, name, env_glob="/environment-setup*"):
+ btenv = glob.glob(btdir + env_glob)
print("Using %s %s" % (name, btenv))
# We either parse or wrap all our execution calls, rock and a hard place :(
with open(btenv[0], "r") as f:
@@ -474,6 +474,18 @@ def enable_tools_tarball(btdir, name):
if line in os.environ:
del os.environ[line]
+# Unlike buildtools (a host/worker property, keyed by worker name globs),
+# the vcontainer-tarball is only needed by specific jobs (e.g.
+# containers-library), so it is keyed off the target/builder via a
+# per-target "vcontainer" config variable, following the extratools pattern.
+def setup_vcontainer_tarball(ourconfig, target, vcdir, checkonly=False):
+ vctarball = getconfigvar("vcontainer", ourconfig, target) or None
+
+ if checkonly:
+ return vctarball
+
+ setup_tools_tarball(ourconfig, vcdir, vctarball, name="vcontainer-tarball", env_glob="/environment-setup-ci")
+
def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False):
bttarball = None
if "buildtools" in ourconfig and workername:
@@ -488,7 +500,7 @@ def setup_buildtools_tarball(ourconfig, workername, btdir, checkonly=False):
setup_tools_tarball(ourconfig, btdir, bttarball)
-def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
+def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools", env_glob="/environment-setup*"):
btenv = None
if bttarball:
@@ -557,7 +569,7 @@ def setup_tools_tarball(ourconfig, btdir, bttarball, name="buildtools"):
if not os.path.exists(btdir):
print("Extracting %s %s" % (name, bttarball))
subprocess.check_call(["bash", btdlpath, "-d", btdir, "-y"])
- enable_tools_tarball(btdir, name)
+ enable_tools_tarball(btdir, name, env_glob)
def get_string_from_version(version, milestone=None, rc=None):
""" Point releases finishing by 0 (e.g 4.0.0, 4.1.0) do no exists,
--
2.43.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [yocto-autobuilder2][PATCH v3 2/6] config.json: add vcontainer-tarball build target
2026-06-06 2:51 [yocto-autobuilder2][PATCH v3 0/6] Implement 'containers' jobs tim.orling
2026-06-06 2:51 ` [yocto-autobuilder2][PATCH v3 1/6] scripts: add vcontainer-tarball setup, integration, and publishing tim.orling
@ 2026-06-06 2:51 ` tim.orling
2026-06-06 2:51 ` [yocto-autobuilder2][PATCH v3 3/6] scripts: add run-vcontainer-tests for meta-virtualization tim.orling
` (3 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: tim.orling @ 2026-06-06 2:51 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Add the vcontainer-tarball build target which produces the
meta-virtualization SDK used by downstream container build
jobs. Modelled after the buildtools-tarball target.
The target uses EXTRACMDS to reset BBMULTICONFIG for the worker
shell, places the extravars in the per-step config, and includes
a publish-artefacts step so the resulting SDK tarball is staged
for reuse by container-tests and other consumers.
Since meta-virtualization is the first layer removed by
'remove-layers', our BBMULTICONFIGs become invalid immediately
after layer removal and throw an error.
In scripts/run-config, EXTRACMDS runs after BBTARGETS, but
before 'remove-layers'. Use 'sed' to reset BBMULTICONFIG to ''.
The error in the 'remove-layers' step can be summarized as:
bb.parse.ParseError: ParseError at /home/pokybuild/yocto-worker/vcontainer-tarball/build/layers/openembedded-core/meta/conf/bitbake.conf:824: Could not include required file conf/multiconfig/vruntime-aarch64.conf
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
config.json | 26 ++++++++++++++++++++++++++
1 file changed, 26 insertions(+)
diff --git a/config.json b/config.json
index 4904439..2971fe9 100644
--- a/config.json
+++ b/config.json
@@ -1827,6 +1827,32 @@
},
"toaster" : {
"EXTRACMDS" : ["${SCRIPTSDIR}/run-toaster-tests ${HELPERBUILDDIR} ${HELPERBUILDDIR}/../layers/bitbake"]
+ },
+ "vcontainer-tarball": {
+ "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"],
+ "ADDLAYER" : [
+ "${BUILDDIR}/../meta-openembedded/meta-oe",
+ "${BUILDDIR}/../meta-openembedded/meta-python",
+ "${BUILDDIR}/../meta-openembedded/meta-networking",
+ "${BUILDDIR}/../meta-openembedded/meta-filesystems",
+ "${BUILDDIR}/../meta-virtualization"
+ ],
+ "step1" : {
+ "shortname" : "Build vcontainer-tarballs",
+ "BBTARGETS" : "vcontainer-tarball",
+ "extravars" : [
+ "DISTRO_FEATURES:append = ' virtualization vcontainer'",
+ "BBMULTICONFIG = 'vruntime-aarch64 vruntime-x86-64'",
+ "INIT_MANAGER = 'systemd'"
+ ],
+ "EXTRACMDS" : ["sed -i '/vruntime-aarch64 vruntime-x86-64/d' ${HELPERBUILDDIR}/conf/auto.conf"]
+ },
+ "step2" : {
+ "shortname" : "Publish vcontainer SDK for test reuse",
+ "EXTRACMDS" : [
+ "install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh"
+ ]
+ }
}
},
"repo-defaults" : {
--
2.43.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [yocto-autobuilder2][PATCH v3 3/6] scripts: add run-vcontainer-tests for meta-virtualization
2026-06-06 2:51 [yocto-autobuilder2][PATCH v3 0/6] Implement 'containers' jobs tim.orling
2026-06-06 2:51 ` [yocto-autobuilder2][PATCH v3 1/6] scripts: add vcontainer-tarball setup, integration, and publishing tim.orling
2026-06-06 2:51 ` [yocto-autobuilder2][PATCH v3 2/6] config.json: add vcontainer-tarball build target tim.orling
@ 2026-06-06 2:51 ` tim.orling
2026-06-06 2:51 ` [yocto-autobuilder2][PATCH v3 4/6] scripts: add container registry push, auth, tagging, runtime selection tim.orling
` (2 subsequent siblings)
5 siblings, 0 replies; 7+ messages in thread
From: tim.orling @ 2026-06-06 2:51 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Add scripts/run-vcontainer-tests, the test runner used by the
vcontainers-test job. It sources the vcontainer-tarball SDK,
discovers the meta-virtualization pytest suite, and runs a
configurable set of suites (vdkr, vpdmn, memres) against the
checked-out layers. Suites can be selected per-step so the
top-level 'vcontainer-tests' job runs the container engine
agnostic tests:
- tests/test_container_cross_install.py
- tests/test_container_registry_script.py
- tests/test_vcontainer_auth_config.py
- tests/test_multiarch_oci.py
- tests/test_multilayer_oci.py
The 'vdkr-tests' and 'vpdmn-tests' jobs run only their respective
suites (including memres for each container engine):
- tests/test_vdkr.py
- tests/test_vdkr_registry.py
and
- tests/test_vpdmn.py
The purpose of this script is to test the just built vcontainer-tarball,
so that it can be considered "known-good" and be published for use by
container building jobs that need the vcontainer-tarball installed.
AI-Generated: Claude Cowork Opus 4.7
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
config.json | 31 +++++
scripts/run-vcontainer-tests | 212 +++++++++++++++++++++++++++++++++++
2 files changed, 243 insertions(+)
create mode 100755 scripts/run-vcontainer-tests
diff --git a/config.json b/config.json
index 2971fe9..3fccd83 100644
--- a/config.json
+++ b/config.json
@@ -1853,6 +1853,37 @@
"install -d ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest && install -m 0755 ${BUILDDIR}/tmp/deploy/sdk/vcontainer-standalone.sh ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new && mv -f ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh.new ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh"
]
}
+ },
+ "vcontainer-tests": {
+ "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"],
+ "ADDLAYER" : [
+ "${BUILDDIR}/../meta-openembedded/meta-oe",
+ "${BUILDDIR}/../meta-openembedded/meta-python",
+ "${BUILDDIR}/../meta-openembedded/meta-networking",
+ "${BUILDDIR}/../meta-openembedded/meta-filesystems",
+ "${BUILDDIR}/../meta-virtualization"
+ ],
+ "step1" : {
+ "shortname" : "Run vcontainer pytest suite",
+ "NOBUILDTOOLS" : 1,
+ "EXTRACMDS" : [
+ "${SCRIPTSDIR}/run-vcontainer-tests -s vcontainer -b ${BUILDDIR} -m ${BUILDDIR}/../meta-virtualization -S ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh -r ${HELPERRESULTSDIR}"
+ ]
+ },
+ "step2" : {
+ "shortname" : "Run vdkr pytest suite",
+ "NOBUILDTOOLS" : 1,
+ "EXTRACMDS" : [
+ "${SCRIPTSDIR}/run-vcontainer-tests -s vdkr -b ${BUILDDIR} -m ${BUILDDIR}/../meta-virtualization -S ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh -r ${HELPERRESULTSDIR}"
+ ]
+ },
+ "step3" : {
+ "shortname" : "Run vpdmn pytest suite",
+ "NOBUILDTOOLS" : 1,
+ "EXTRACMDS" : [
+ "${SCRIPTSDIR}/run-vcontainer-tests -s vpdmn -b ${BUILDDIR} -m ${BUILDDIR}/../meta-virtualization -S ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh -r ${HELPERRESULTSDIR}"
+ ]
+ }
}
},
"repo-defaults" : {
diff --git a/scripts/run-vcontainer-tests b/scripts/run-vcontainer-tests
new file mode 100755
index 0000000..a7dbab3
--- /dev/null
+++ b/scripts/run-vcontainer-tests
@@ -0,0 +1,212 @@
+#!/bin/bash
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Run meta-virtualization pytest test suites against the vcontainer
+# standalone SDK (vdkr/vpdmn) that was built in a previous bitbake
+# step.
+#
+# Usage:
+# run-vcontainer-tests -s <suite> -b <builddir> -m <metavirtdir> [options]
+#
+# Required:
+# -s <suite> suite name: one of "vcontainer", "vdkr", "vpdmn"
+# -b <builddir> bitbake build directory (${BUILDDIR})
+# -m <metavirtdir> path to the meta-virtualization layer
+#
+# Optional:
+# -S <sdk> path to the vcontainer standalone SDK installer. The
+# autobuilder -tests jobs share the SDK produced by the
+# separate vcontainer-tarball builder. When unset, falls
+# back to looking in <builddir>/tmp/deploy/sdk.
+# -e <extractdir> where to extract the standalone SDK tarball
+# (default: <builddir>/vcontainer-test-extracted)
+# -i <oci_image> path to an OCI image directory (enables vdkr/vpdmn
+# import tests)
+# -a <arch> target architecture for vdkr/vpdmn tests (default: x86_64)
+# -r <resultsdir> directory to copy pytest artefacts (junit xml / log) to
+# -k <markers> pytest marker filter (default excludes long running /
+# infrastructure dependent tests)
+# -h show this help and exit
+#
+# The script is intentionally conservative: any pytest tests that cannot run
+# in the CI environment (those marked "slow", "network", "boot") are skipped
+# so that the autobuilder step completes without needing network access. Those
+# can be re-enabled via -k before invocation.
+#
+# It is assumed that /dev/kvm is writable by the CI user running the tests,
+# since the performance is significantly faster with 'memres'.
+#
+
+set -e
+set -u
+set -o pipefail
+set -x
+
+usage() {
+ cat >&2 <<'EOF'
+Usage: run-vcontainer-tests -s <suite> -b <builddir> -m <metavirtdir> [options]
+
+Required:
+ -s <suite> vcontainer | vdkr | vpdmn
+ -b <builddir> bitbake build directory
+ -m <metavirtdir> path to the meta-virtualization layer
+
+Optional:
+ -S <sdk> path to the vcontainer standalone SDK installer
+ -e <extractdir> where to extract the standalone SDK tarball
+ -i <oci_image> path to an OCI image directory
+ -a <arch> target architecture (default: x86_64)
+ -r <resultsdir> directory to copy pytest artefacts to
+ -k <markers> pytest marker filter
+ -h show this help and exit
+EOF
+}
+
+suite=""
+builddir=""
+metavirtdir=""
+sdk_tarball=""
+extract_dir=""
+oci_image=""
+arch=""
+results_dir=""
+marker_filter="not slow and not network and not boot and not incus and not k3s"
+
+while getopts ":s:b:m:S:e:i:a:r:k:h" opt; do
+ case "$opt" in
+ s) suite="$OPTARG" ;;
+ b) builddir="$OPTARG" ;;
+ m) metavirtdir="$OPTARG" ;;
+ S) sdk_tarball="$OPTARG" ;;
+ e) extract_dir="$OPTARG" ;;
+ i) oci_image="$OPTARG" ;;
+ a) arch="$OPTARG" ;;
+ r) results_dir="$OPTARG" ;;
+ k) marker_filter="$OPTARG" ;;
+ h) usage; exit 0 ;;
+ :) echo "ERROR: option -$OPTARG requires an argument" >&2; usage; exit 2 ;;
+ \?) echo "ERROR: unknown option -$OPTARG" >&2; usage; exit 2 ;;
+ esac
+done
+
+if [ -z "$suite" ] || [ -z "$builddir" ] || [ -z "$metavirtdir" ]; then
+ echo "ERROR: -s, -b and -m are required" >&2
+ usage
+ exit 2
+fi
+
+builddir=$(realpath "$builddir")
+metavirtdir=$(realpath "$metavirtdir")
+
+if [ ! -d "$metavirtdir/tests" ]; then
+ echo "ERROR: meta-virtualization tests directory not found at $metavirtdir/tests" >&2
+ exit 1
+fi
+
+# Locate the vcontainer standalone SDK tarball. Prefer an explicitly-provided
+# SDK (-S), and fall back to looking in the local build's
+# deploy/sdk directory when running stand-alone.
+if [ -n "$sdk_tarball" ]; then
+ if [ ! -f "$sdk_tarball" ]; then
+ echo "ERROR: SDK installer '$sdk_tarball' is set but not a file" >&2
+ exit 1
+ fi
+else
+ sdk_tarball="$builddir/tmp/deploy/sdk/vcontainer-standalone.sh"
+ if [ ! -f "$sdk_tarball" ]; then
+ # Try to find any matching tarball in case naming changed (e.g. versioned)
+ alt=$(ls -1 "$builddir"/tmp/deploy/sdk/vcontainer-*.sh 2>/dev/null | head -n1 || true)
+ if [ -n "$alt" ]; then
+ sdk_tarball="$alt"
+ else
+ echo "ERROR: vcontainer standalone SDK not found." >&2
+ echo " Pass -S with an existing SDK installer, or" >&2
+ echo " build vcontainer-tarball so $builddir/tmp/deploy/sdk/vcontainer-standalone.sh exists." >&2
+ exit 1
+ fi
+ fi
+fi
+
+extract_dir="${extract_dir:-$builddir/vcontainer-test-extracted}"
+rm -rf "$extract_dir"
+mkdir -p "$(dirname "$extract_dir")"
+
+# Self-extracting installer (silent, -y agrees to license, -d picks dir)
+"$sdk_tarball" -d "$extract_dir" -y
+
+# Prepare a Python venv so we don't pollute the worker's system packages.
+python3 -m venv "$builddir/meta-virt-test-venv"
+# shellcheck disable=SC1091
+source "$builddir/meta-virt-test-venv/bin/activate"
+# Avoid warnings by upgrading pip; install pytest/pexpect into the venv via pip.
+export PIP_DISABLE_PIP_VERSION_CHECK=1
+python3 -m pip install --quiet -r "$metavirtdir/tests/requirements.txt"
+
+# Per-suite test file selection. Uses -k/-m for fine-grained filtering and
+# keeps the CLI small for logging clarity.
+case "$suite" in
+ vdkr)
+ test_files=(
+ "tests/test_vdkr.py"
+ "tests/test_vdkr_registry.py"
+ )
+ ;;
+ vpdmn)
+ test_files=(
+ "tests/test_vpdmn.py"
+ )
+ ;;
+ vcontainer)
+ # Broad vcontainer/bbclass/tooling coverage that doesn't require the
+ # vdkr/vpdmn CLI harness to be running.
+ test_files=(
+ "tests/test_container_cross_install.py"
+ "tests/test_container_registry_script.py"
+ "tests/test_vcontainer_auth_config.py"
+ "tests/test_multiarch_oci.py"
+ "tests/test_multilayer_oci.py"
+ )
+ ;;
+ *)
+ echo "ERROR: unknown suite '$suite' (expected vcontainer|vdkr|vpdmn)" >&2
+ exit 2
+ ;;
+esac
+
+pytest_args=(
+ -v
+ --tb=short
+ -m "$marker_filter"
+ --vdkr-dir "$extract_dir"
+ --junitxml="$builddir/pytest-$suite-results.xml"
+)
+
+# Allow tests that consume an OCI image (import/save/load) to find one.
+if [ -n "$oci_image" ] && [ -d "$oci_image" ]; then
+ pytest_args+=(--oci-image "$oci_image")
+fi
+
+# Pass architecture through when set (default is x86_64).
+if [ -n "$arch" ]; then
+ pytest_args+=(--arch "$arch")
+fi
+
+cd "$metavirtdir"
+# Don't let a single failing test kill the whole step - collect the junit
+# report, then surface the exit code via the junit file + exit status.
+set +e
+python3 -m pytest "${pytest_args[@]}" "${test_files[@]}"
+rc=$?
+set -e
+
+# Copy artefacts to the results dir if one was provided.
+if [ -n "$results_dir" ]; then
+ mkdir -p "$results_dir"
+ cp -f "$builddir/pytest-$suite-results.xml" "$results_dir/" 2>/dev/null || true
+ if [ -f /tmp/pytest-vcontainer.log ]; then
+ cp -f /tmp/pytest-vcontainer.log "$results_dir/pytest-$suite.log" || true
+ fi
+fi
+
+exit $rc
--
2.43.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [yocto-autobuilder2][PATCH v3 4/6] scripts: add container registry push, auth, tagging, runtime selection
2026-06-06 2:51 [yocto-autobuilder2][PATCH v3 0/6] Implement 'containers' jobs tim.orling
` (2 preceding siblings ...)
2026-06-06 2:51 ` [yocto-autobuilder2][PATCH v3 3/6] scripts: add run-vcontainer-tests for meta-virtualization tim.orling
@ 2026-06-06 2:51 ` tim.orling
2026-06-06 2:51 ` [yocto-autobuilder2][PATCH v3 5/6] config.json: add 'containers-library' build job tim.orling
2026-06-06 2:51 ` [yocto-autobuilder2][PATCH v3 6/6] scripts/run-push-containers: remove all images before push tim.orling
5 siblings, 0 replies; 7+ messages in thread
From: tim.orling @ 2026-06-06 2:51 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Add the push-containers infrastructure that drives the
post-build steps for the 'containers-library' job. After
each build step the runtime container store is harvested
and pushed to one or more registries with derived per-step
tags.
* config.json: add CONTAINER_REGISTRIES, CONTAINER_AUTH_CONFIG,
CONTAINER_RUNTIME, CONTAINER_TAG_CMDS, CONTAINER_VERSION_RECIPE
and CONTAINER_IMAGE_MAP configuration knobs.
* scripts/run-config: drive push-containers as a post-step
action (by running scripts/run-push-containers).
* scripts/run-push-containers: Tags are generated from recipe and
distro metadata (yocto- tag uses major.minor on snapshots and full
PV on releases) with CONTAINER_VERSION_RECIPE allowing a step to
source PV from a different recipe than the image itself.
* Registry auth is staged via .../config.json or podman
.../auth.json using CONTAINER_AUTH_CONFIG, replacing an
interactive login that could hang. CONTAINER_RUNTIME picks
between vdkr (Docker-compatible) and vpdmn (Podman) runtimes.
* Robustness: skip gracefully when no registries are configured,
fix the OCI directory path, handle memres already running,
and avoid hanging when memres has not yet come up.
AI-Generated: Claude Cowork Opus 4.8
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
config.json | 5 ++
scripts/run-config | 12 +++
scripts/run-push-containers | 165 ++++++++++++++++++++++++++++++++++++
3 files changed, 182 insertions(+)
create mode 100755 scripts/run-push-containers
diff --git a/config.json b/config.json
index 3fccd83..e1ea01a 100644
--- a/config.json
+++ b/config.json
@@ -43,6 +43,11 @@
"BUILDINFOVARS" : ["INHERIT += 'image-buildinfo'", "IMAGE_BUILDINFO_VARS:append = ' IMAGE_BASENAME IMAGE_NAME'"],
"WRITECONFIG" : true,
"SENDERRORS" : true,
+ "CONTAINER_RUNTIME" : "vpdmn",
+ "CONTAINER_REGISTRIES" : [],
+ "CONTAINER_TAGS" : ["latest"],
+ "CONTAINER_TAG_CMDS" : [],
+ "CONTAINER_IMAGE_MAP" : {},
"extravars" : [
"SANITY_TESTED_DISTROS = ''",
"BB_HASHSERVE = '${AUTOBUILDER_HASHSERV}'",
diff --git a/scripts/run-config b/scripts/run-config
index 90a5996..31db212 100755
--- a/scripts/run-config
+++ b/scripts/run-config
@@ -195,6 +195,8 @@ utils.mkdir(errordir)
errorlogs = set()
+push_containers = properties.get("push_containers", False)
+
def log_file_contents(filename, builddir, stepnum, stepname):
logfile = logname(builddir, stepnum, stepname)
with open(logfile, "a") as outf, open(filename, "r") as f:
@@ -313,6 +315,16 @@ def handle_stepnum(stepnum):
hp.printheader("Step %s/%s: Running bitbake %s" % (stepnum, maxsteps, sanitytargets))
bitbakecmd(args.builddir, "bitbake %s -k" % (sanitytargets), report, stepnum, args.stepname)
+ # Push container images to registries when push_containers is enabled.
+ # The push logic itself lives in scripts/run-push-containers.
+ container_images = utils.getconfigdict("CONTAINER_IMAGE_MAP", ourconfig, args.target, stepnum)
+ if container_images and push_containers:
+ if jcfg:
+ addstepentry("push-containers", "Push containers", shortdesc, desc, str(container_images), str(stepnum))
+ elif args.stepname == "push-containers":
+ hp.printheader("Step %s/%s: Pushing container images %s" % (stepnum, maxsteps, list(container_images.keys())))
+ bitbakecmd(args.builddir, "%s/run-push-containers %s %s" % (scriptsdir, args.target, stepnum), report, stepnum, args.stepname)
+
# Run any extra commands specified
cmds = utils.getconfiglist("EXTRACMDS", ourconfig, args.target, stepnum)
if jcfg:
diff --git a/scripts/run-push-containers b/scripts/run-push-containers
new file mode 100755
index 0000000..cdef497
--- /dev/null
+++ b/scripts/run-push-containers
@@ -0,0 +1,165 @@
+#!/usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-only
+#
+# Push container images built by a target step to the configured registries.
+#
+# Invoked by run-config for 'push-containers' steps with the OE build
+# environment already sourced (bitbake and the vcontainer runtime wrappers
+# must be on PATH). All configuration comes from the target/step config:
+#
+# CONTAINER_IMAGE_MAP - dict of image recipe -> registry image name
+# CONTAINER_RUNTIME - vpdmn (default) or vdkr
+# CONTAINER_REGISTRIES - list of registries to push to
+# CONTAINER_TAGS - list of static tags (e.g. latest)
+# CONTAINER_TAG_CMDS - extra shell to populate _EXTRA_TAGS
+# CONTAINER_VERSION_RECIPE - recipe whose PV provides the version tag
+# CONTAINER_AUTH_CONFIG - registry auth file staged into the guest
+#
+
+import subprocess
+import sys
+
+import utils
+
+parser = utils.ArgParser(description='Pushes container images for a target step to the configured registries.')
+
+parser.add_argument('target',
+ help="The target build name")
+parser.add_argument('stepnum',
+ type=int,
+ help="The step number within the target")
+
+args = parser.parse_args()
+
+ourconfig = utils.loadconfig()
+
+container_images = utils.getconfigdict("CONTAINER_IMAGE_MAP", ourconfig, args.target, args.stepnum)
+if not container_images:
+ print("No CONTAINER_IMAGE_MAP for %s step %s, nothing to push" % (args.target, args.stepnum))
+ sys.exit(0)
+
+registries = utils.getconfiglist("CONTAINER_REGISTRIES", ourconfig, args.target, args.stepnum)
+if not registries:
+ utils.printheader("push-containers skipped — CONTAINER_REGISTRIES is empty, no containers pushed")
+ sys.exit(0)
+
+runtime = utils.getconfigvar("CONTAINER_RUNTIME", ourconfig, args.target, args.stepnum) or "vpdmn"
+static_tags = utils.getconfiglist("CONTAINER_TAGS", ourconfig, args.target, args.stepnum)
+auth_config = utils.getconfigvar("CONTAINER_AUTH_CONFIG", ourconfig, args.target, args.stepnum)
+if not auth_config:
+ if runtime == "vpdmn":
+ auth_config = "${HOME}/.config/containers/auth.json"
+ else:
+ auth_config = "${HOME}/.docker/config.json"
+
+utils.printheader("Pushing container images %s" % list(container_images.keys()))
+
+script = [
+ "set -e",
+ "test -w /dev/kvm || { echo 'ERROR: /dev/kvm is not writable, cannot push containers'; exit 1; }",
+ # Always bring up a fresh memres VM in the foreground.
+ #
+ # 'memres status' only checks that the QEMU PID in daemon.pid
+ # is alive (see daemon_is_running()/daemon_status() in
+ # meta-virtualization's vrunner.sh); it returns 0 as soon as
+ # QEMU forks, so a hung/partially-booted VM from a previous
+ # run — or a VM in mid-boot — is reported as healthy. The
+ # subsequent 'login'/'vimport'/'push' commands then hang on
+ # the unresponsive daemon socket.
+ #
+ # 'memres restart' is synchronous: it does stop+start and
+ # runs a PING/PONG readiness probe against the daemon socket
+ # (120s timeout), exiting non-zero if the VM never answers.
+ # Running it in the foreground gives us a trustworthy ready
+ # signal via its exit code, so we can drop the status-poll
+ # loop entirely.
+ #
+ # Install an EXIT trap first so we always tear the daemon
+ # down, even if bitbake -e / vimport / push fails mid-step
+ # under 'set -e'. The trap is armed before the restart so
+ # a restart failure also triggers cleanup.
+ #
+ # Registry auth is staged into the guest at VM boot via
+ # the global '--config' flag — vrunner.sh's setup_auth_share()
+ # copies $AUTH_CONFIG onto a read-only 9p share, and
+ # vdkr-init.sh / vpdmn-init.sh's install_auth_config()
+ # installs it at /root/.docker/config.json (vdkr) or
+ # /run/containers/0/auth.json (vpdmn) inside the guest.
+ # Subsequent 'push' calls use those creds directly, so no
+ # explicit 'login' step is needed. Calling 'login' would
+ # actually hang under the autobuilder (no PTY): when the
+ # memres daemon is running, vcontainer-common.sh dispatches
+ # login via '--daemon-interactive' and blocks reading the
+ # password from stdin (see login case in vcontainer-common.sh).
+ "trap '%s-$(arch) memres stop 2>/dev/null || true' EXIT" % runtime,
+ "%s-$(arch) --config %s memres restart </dev/null" % (runtime, auth_config),
+]
+tag_cmds = utils.getconfiglist("CONTAINER_TAG_CMDS", ourconfig, args.target, args.stepnum)
+version_recipe = utils.getconfigvar("CONTAINER_VERSION_RECIPE", ourconfig, args.target, args.stepnum)
+for recipe, image in container_images.items():
+ # Extract version metadata from the recipe and distro via
+ # bitbake -e. Steps that need additional derived tags (e.g.
+ # major, major.minor) populate _EXTRA_TAGS via
+ # CONTAINER_TAG_CMDS in their step config.
+ #
+ # PV is sanitized with 'sed s/+.*//' to drop Yocto's
+ # '+git<sha>' suffix on AUTOREV/dev recipes — Docker
+ # reference format does not allow '+' in tags, and the
+ # base PV is what consumers expect.
+ #
+ # DISTRO_VERSION needs context-sensitive handling. Poky's
+ # DISTRO_VERSION resolves to '${PV}+snapshot-${METADATA_REVISION}'
+ # off a tag and just '${PV}' on a release tag. The '+' in
+ # the snapshot form is illegal in a Docker tag, but more
+ # importantly the patch level on a snapshot build (e.g.
+ # '6.0.99' between 6.0 and 6.1) is a moving target that
+ # doesn't correspond to any real release — only the
+ # major.minor line is meaningful. So:
+ # - snapshot build (DISTRO_VERSION contains '+') → tag
+ # with major.minor only, e.g. 'yocto-6.0'.
+ # - release-tag build (no '+') → tag with the full
+ # version, e.g. 'yocto-5.0.5' from the yocto-5.0.5 tag.
+ script += [
+ "_BBENV=$(bitbake -e %s 2>/dev/null) || true" % recipe,
+ "_PV=$(echo \"$_BBENV\" | awk -F'\"' '/^PV=/{ print $2; exit }' | sed 's/+.*//')",
+ "_DISTRO_CODENAME=$(echo \"$_BBENV\" | awk -F'\"' '/^DISTRO_CODENAME=/{ print $2; exit }')",
+ "_DISTRO_VERSION_RAW=$(echo \"$_BBENV\" | awk -F'\"' '/^DISTRO_VERSION=/{ print $2; exit }')",
+ "case \"$_DISTRO_VERSION_RAW\" in",
+ " *+*) _DISTRO_VERSION=$(echo \"${_DISTRO_VERSION_RAW%%+*}\" | cut -d. -f1,2) ;;",
+ " *) _DISTRO_VERSION=\"$_DISTRO_VERSION_RAW\" ;;",
+ "esac",
+ "_DEPLOY_DIR_IMAGE=$(echo \"$_BBENV\" | awk -F'\"' '/^DEPLOY_DIR_IMAGE=/{ print $2; exit }')",
+ "_EXTRA_TAGS=\"\"",
+ ]
+ if version_recipe:
+ # When the image recipe's PV is a wrapper-style
+ # placeholder (e.g. app-container-python_1.0.0.bb,
+ # whose 1.0.0 is meaningless to a downstream user),
+ # CONTAINER_VERSION_RECIPE points at the recipe whose
+ # PV is actually meaningful for the resulting tag —
+ # typically the language runtime or app being packaged
+ # (e.g. python3 -> 3.14.x). Override _PV from that
+ # recipe; image-recipe state still drives
+ # DEPLOY_DIR_IMAGE and DISTRO_* since those are
+ # environment-wide.
+ script += [
+ "_VBBENV=$(bitbake -e %s 2>/dev/null) || true" % version_recipe,
+ "_PV=$(echo \"$_VBBENV\" | awk -F'\"' '/^PV=/{ print $2; exit }' | sed 's/+.*//')",
+ ]
+ script += tag_cmds
+ script.append(
+ "_TAGS=\"%s $_PV $_DISTRO_CODENAME yocto-$_DISTRO_VERSION $_EXTRA_TAGS\"" % " ".join(static_tags)
+ )
+ for registry in registries:
+ # No per-registry 'login': credentials were staged into
+ # the guest by '--config' on 'memres restart' above.
+ script += [
+ "for _tag in $_TAGS; do",
+ " %s-$(arch) vimport ${_DEPLOY_DIR_IMAGE}/%s-latest-oci %s/%s:${_tag}" % (runtime, recipe, registry, image),
+ " %s-$(arch) push %s/%s:${_tag}" % (runtime, registry, image),
+ "done",
+ ]
+# Tear-down is handled by the EXIT trap installed above.
+utils.flush()
+sys.exit(subprocess.call(["/bin/bash", "-c", "\n".join(script)]))
--
2.43.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [yocto-autobuilder2][PATCH v3 5/6] config.json: add 'containers-library' build job
2026-06-06 2:51 [yocto-autobuilder2][PATCH v3 0/6] Implement 'containers' jobs tim.orling
` (3 preceding siblings ...)
2026-06-06 2:51 ` [yocto-autobuilder2][PATCH v3 4/6] scripts: add container registry push, auth, tagging, runtime selection tim.orling
@ 2026-06-06 2:51 ` tim.orling
2026-06-06 2:51 ` [yocto-autobuilder2][PATCH v3 6/6] scripts/run-push-containers: remove all images before push tim.orling
5 siblings, 0 replies; 7+ messages in thread
From: tim.orling @ 2026-06-06 2:51 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Add 'containers-library' build job that builds container images on top
of the vcontainer-tarball SDK:
* original container images from the intial "container-cross-install"
branch of meta-virtualization
- container-base
- app-container-curl
* additional images modelled after docker.io/library/*
- app-container-python
- app-container-mosquitto
- app-container-valkey
- app-container-nginx
* Tag containers with versions based on the recipe to which they are
mapped, e.g. python:3, python:3.14, python:3.14.5. Also tag with
DISTRO_CODENAME and DISTRO_VERSION, e.g. wrynose and yocto-6.0.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
config.json | 56 +++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 56 insertions(+)
diff --git a/config.json b/config.json
index e1ea01a..18a2b6b 100644
--- a/config.json
+++ b/config.json
@@ -1889,6 +1889,62 @@
"${SCRIPTSDIR}/run-vcontainer-tests -s vpdmn -b ${BUILDDIR} -m ${BUILDDIR}/../meta-virtualization -S ${BASE_SHAREDDIR}/pub/vcontainer-tarball-latest/vcontainer-standalone.sh -r ${HELPERRESULTSDIR}"
]
}
+ },
+ "containers-library": {
+ "NEEDREPOS" : ["bitbake", "meta-openembedded", "meta-virtualization"],
+ "ADDLAYER" : [
+ "${BUILDDIR}/../meta-openembedded/meta-oe",
+ "${BUILDDIR}/../meta-openembedded/meta-python",
+ "${BUILDDIR}/../meta-openembedded/meta-networking",
+ "${BUILDDIR}/../meta-openembedded/meta-filesystems",
+ "${BUILDDIR}/../meta-openembedded/meta-webserver",
+ "${BUILDDIR}/../meta-virtualization"
+ ],
+ "extravars" : [
+ "DISTRO_FEATURES:append = ' virtualization vcontainer'"
+ ],
+ "vcontainer" : "${VCONTAINER_TARBALL_URL}",
+ "CONTAINER_TAG_CMDS" : [
+ "_PV_MAJOR=$(echo $_PV | cut -d. -f1)",
+ "_PV_MAJOR_MINOR=$(echo $_PV | cut -d. -f1,2)",
+ "_EXTRA_TAGS=\"$_PV_MAJOR $_PV_MAJOR_MINOR\""
+ ],
+ "step1" : {
+ "shortname" : "Build 'base' container",
+ "BBTARGETS" : "container-base",
+ "CONTAINER_IMAGE_MAP" : {"container-base": "base"},
+ "CONTAINER_VERSION_RECIPE" : "base-files"
+ },
+ "step2" : {
+ "shortname" : "Build 'curl' container",
+ "BBTARGETS" : "app-container-curl",
+ "CONTAINER_IMAGE_MAP" : {"app-container-curl": "curl"},
+ "CONTAINER_VERSION_RECIPE" : "curl"
+ },
+ "step3" : {
+ "shortname" : "Build 'python' container",
+ "BBTARGETS" : "app-container-python",
+ "CONTAINER_IMAGE_MAP" : {"app-container-python": "python"},
+ "CONTAINER_VERSION_RECIPE" : "python3"
+ },
+ "step4" : {
+ "shortname" : "Build 'mosquitto' container",
+ "BBTARGETS" : "app-container-mosquitto",
+ "CONTAINER_IMAGE_MAP" : {"app-container-mosquitto": "mosquitto"},
+ "CONTAINER_VERSION_RECIPE" : "mosquitto"
+ },
+ "step5" : {
+ "shortname" : "Build 'valkey' container",
+ "BBTARGETS" : "app-container-valkey",
+ "CONTAINER_IMAGE_MAP" : {"app-container-valkey": "valkey"},
+ "CONTAINER_VERSION_RECIPE" : "valkey"
+ },
+ "step6" : {
+ "shortname" : "Build 'nginx' container",
+ "BBTARGETS" : "app-container-nginx",
+ "CONTAINER_IMAGE_MAP" : {"app-container-nginx": "nginx"},
+ "CONTAINER_VERSION_RECIPE" : "nginx"
+ }
}
},
"repo-defaults" : {
--
2.43.0
^ permalink raw reply related [flat|nested] 7+ messages in thread* [yocto-autobuilder2][PATCH v3 6/6] scripts/run-push-containers: remove all images before push
2026-06-06 2:51 [yocto-autobuilder2][PATCH v3 0/6] Implement 'containers' jobs tim.orling
` (4 preceding siblings ...)
2026-06-06 2:51 ` [yocto-autobuilder2][PATCH v3 5/6] config.json: add 'containers-library' build job tim.orling
@ 2026-06-06 2:51 ` tim.orling
5 siblings, 0 replies; 7+ messages in thread
From: tim.orling @ 2026-06-06 2:51 UTC (permalink / raw)
To: yocto-patches
From: Tim Orling <tim.orling@konsulko.com>
Workaround for errors like:
Error: reading blob sha256:<hash>: file integrity checksum failed for "<file>"
Error: reading blob sha256:<hash>: EOF
The first time the blob (e.g. bin/bash.bash) is copied, it succeeds. All subsequent
tries fail. Might possibly be an issue with sstate or a recent change to vcontainer-common
to allow for multiarch containers.
Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
scripts/run-push-containers | 2 ++
1 file changed, 2 insertions(+)
diff --git a/scripts/run-push-containers b/scripts/run-push-containers
index cdef497..4f1ac53 100755
--- a/scripts/run-push-containers
+++ b/scripts/run-push-containers
@@ -94,6 +94,8 @@ script = [
# password from stdin (see login case in vcontainer-common.sh).
"trap '%s-$(arch) memres stop 2>/dev/null || true' EXIT" % runtime,
"%s-$(arch) --config %s memres restart </dev/null" % (runtime, auth_config),
+ # workaround for 'Error: reading blob sha256:<hash>: file integrity checksum failed for "<file>"'
+ "%s-$(arch) image rm --all 2>/dev/null" % (runtime),
]
tag_cmds = utils.getconfiglist("CONTAINER_TAG_CMDS", ourconfig, args.target, args.stepnum)
version_recipe = utils.getconfigvar("CONTAINER_VERSION_RECIPE", ourconfig, args.target, args.stepnum)
--
2.43.0
^ permalink raw reply related [flat|nested] 7+ messages in thread