[meta-virtualization][PATCH 1/2] image-oci: add OCI runtime bundle output

[meta-virtualization][PATCH 1/2] image-oci: add OCI runtime bundle output

[Tanguy Raufflet]

umoci currently uses an OCI runtime bundle as a temporary artifact
because umoci needs this intermediate representation to construct an OCI
image with the rootfs populated. After this step, the bundle is not used
anymore since umoci does not provide a mechanism to directly configure
the runtime bundle directly.

Because some utilities expect an OCI runtime bundle as input (e.g.,
crun, runc), it is useful to optionally retain the bundle after
constructing the OCI image. The comment [1] in the code was already
suggesting that retaining the OCI runtime bundle could be beneficial.

    "We could make this optional, since the bundle is directly runnable
    via runc"

This commit adds OCI_IMAGE_BUNDLE_OUTPUT to optionally deploy the
runtime bundle as a tar archive into IMGDEPLOYDIR. This bundle contains
the fully populated OCI runtime bundle, including the rootfs and the
config.json file reflecting the final OCI image configuration.

Also, a postprocess command can be specified to either modify the
runtime bundle's configuration or replace it entirely before it is
archived.

Note that image_bundle_name is not generated to the IMGDEPLOYDIR anymore
to avoid issues when BitBake processes and caches files under
IMGDEPLOYDIR.

Signed-off-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com>
---
 classes/image-oci-umoci.inc | 41 ++++++++++++++++++++++--
 classes/image-oci.bbclass   | 13 ++++++++
 docs/container-bundling.md  | 64 +++++++++++++++++++++++++++++++++++++
 3 files changed, 115 insertions(+), 3 deletions(-)

diff --git a/classes/image-oci-umoci.inc b/classes/image-oci-umoci.inc
index 67f6c6e5..ad103f16 100644
--- a/classes/image-oci-umoci.inc
+++ b/classes/image-oci-umoci.inc
@@ -421,6 +421,34 @@ PYEOF
     bbnote "OCI fix: Layer whiteout fix complete"
 }
 
+oci_archive_bundle() {
+    image_tag="${@d.getVar("OCI_IMAGE_TAG").replace(":", "_")}"
+
+    ${IMAGE_CMD_TAR} --sort=name --format=posix --pax-option=delete=atime,delete=ctime --numeric-owner \
+        -cf "${IMGDEPLOYDIR}/${IMAGE_NAME}.oci-bundle.tar" \
+        -C "${OCI_BUNDLE_DIR}" \
+        .
+
+    ln -sfn "${IMAGE_NAME}.oci-bundle.tar" \
+        "${IMGDEPLOYDIR}/${IMAGE_BASENAME}-$image_tag-oci-bundle.tar"
+
+    rm -rf "${OCI_BUNDLE_DIR}"
+}
+
+python oci_bundle_postprocess() {
+    from oe.utils import execute_pre_post_process
+
+    execute_pre_post_process(
+        d, d.getVar('OCI_BUNDLE_POSTPROCESS_COMMAND')
+    )
+    bb.build.exec_func('oci_archive_bundle', d)
+}
+
+oci_bundle_postprocess[dirs] = "${OCI_BUNDLE_DIR}"
+
+do_image_oci[postfuncs] += "${@'oci_bundle_postprocess' if bb.utils.to_boolean(d.getVar('OCI_IMAGE_BUNDLE_OUTPUT')) else ''}"
+do_image_oci[subimages] += "${@'oci-bundle.tar' if bb.utils.to_boolean(d.getVar('OCI_IMAGE_BUNDLE_OUTPUT')) else ''}"
+
 IMAGE_CMD:oci() {
     umoci_options=""
 
@@ -477,7 +505,7 @@ IMAGE_CMD:oci() {
 
     new_image=t
     image_name="${IMAGE_NAME}${IMAGE_NAME_SUFFIX}-oci"
-    image_bundle_name="${IMAGE_NAME}${IMAGE_NAME_SUFFIX}-oci-bundle"
+    image_bundle_name="${OCI_BUNDLE_DIR}"
     if [ -n "$OCI_REUSE_IMAGE" ]; then
 	if [ -d $image_name ]; then
 	    bbdebug 1 "OCI: reusing image directory"
@@ -816,8 +844,15 @@ IMAGE_CMD:oci() {
 	ln -sfn "$image_name-dir.tar" "${IMAGE_BASENAME}-$image_tag-oci-dir.tar"
     fi
 
-    # We could make this optional, since the bundle is directly runnable via runc
-    rm -rf $image_bundle_name
+    if [ "${OCI_IMAGE_BUNDLE_OUTPUT}" = "1" ]; then
+        # Refresh config.json from the finalized image before post-processing.
+        umoci raw runtime-config --rootless \
+            --image "$image_name:${OCI_IMAGE_TAG}" \
+            --rootfs "$image_bundle_name/rootfs" \
+            "$image_bundle_name/config.json"
+    else
+        rm -rf "$image_bundle_name"
+    fi
 
     # This is the OCI image directory, which is technically the "image" as specified
     # Use -n to avoid creating link inside existing symlink target directory
diff --git a/classes/image-oci.bbclass b/classes/image-oci.bbclass
index 141cd2a8..cbbaa388 100644
--- a/classes/image-oci.bbclass
+++ b/classes/image-oci.bbclass
@@ -280,6 +280,19 @@ OCI_LAYER_CACHE_DIR ?= "${TOPDIR}/oci-layer-cache/${MACHINE}"
 # bundled into a tarball.
 OCI_IMAGE_TAR_OUTPUT ?= "true"
 
+# Create a tar archive containing an OCI runtime bundle suitable for OCI runtimes such as crun.
+OCI_IMAGE_BUNDLE_OUTPUT ?= "0"
+
+# Functions appended here run after the final runtime configuration is generated
+# and before the bundle is archived. The bundle directory is available as
+# OCI_BUNDLE_DIR.
+OCI_BUNDLE_DIR = "${WORKDIR}/${IMAGE_BASENAME}-oci-bundle"
+OCI_BUNDLE_POSTPROCESS_COMMAND ?= ""
+
+python () {
+    d.setVarFlag('OCI_BUNDLE_POSTPROCESS_COMMAND', 'vardeps', d.getVar('OCI_BUNDLE_POSTPROCESS_COMMAND'))
+}
+
 # Generate a subarch that is appropriate to OCI image
 # types. This is typically only ARM architectures at the
 # moment.
diff --git a/docs/container-bundling.md b/docs/container-bundling.md
index 77aa09b0..bedd541e 100644
--- a/docs/container-bundling.md
+++ b/docs/container-bundling.md
@@ -288,6 +288,70 @@ Disable caching with `OCI_LAYER_CACHE = "0"` if you:
 - Need fully reproducible builds with no local state
 - Are debugging package installation issues
 
+### OCI Runtime Bundles
+
+The umoci backend can emit an OCI runtime bundle suitable for runtimes such
+as crun and runc:
+
+    OCI_IMAGE_BUNDLE_OUTPUT = "1"
+
+The `app-container-oci-bundle` demo image enables this output and can be
+built with `bitbake app-container-oci-bundle`.
+
+The runtime configuration is generated from the finalized OCI image, so its
+`config.json` includes settings applied through `OCI_IMAGE_ENV_VARS`,
+`OCI_IMAGE_CMD`, `OCI_IMAGE_ENTRYPOINT`, `OCI_IMAGE_RUNTIME_UID`, and
+`OCI_IMAGE_WORKINGDIR`. It is deployed as:
+
+    ${IMAGE_BASENAME}-${OCI_IMAGE_TAG}-oci-bundle.tar
+
+with the convenience symlink:
+
+    ${IMAGE_BASENAME}-oci-bundle.tar
+
+`OCI_IMAGE_BUNDLE_OUTPUT` is independent of `OCI_IMAGE_TAR_OUTPUT`. The
+latter controls tar output for the OCI image layout (`index.json` and
+`blobs/`), while the runtime bundle contains `config.json` and `rootfs/`.
+
+Extract and run the bundle with crun:
+
+    mkdir my-container-bundle
+    tar -xf my-container-oci-bundle.tar -C my-container-bundle
+    crun run --bundle my-container-bundle my-container
+
+#### Bundle Post-processing
+
+Functions listed in `OCI_BUNDLE_POSTPROCESS_COMMAND` run after the finalized
+image is unpacked and before the bundle is archived. They run with
+`OCI_BUNDLE_DIR` as their current working directory:
+
+```bitbake
+OCI_BUNDLE_POSTPROCESS_COMMAND += "configure_runtime_bundle;"
+
+configure_runtime_bundle() {
+    config="config.json"
+    jq '.process.terminal = false' "$config" > "$config.tmp"
+    mv "$config.tmp" "$config"
+}
+```
+
+Post-processing changes only the emitted runtime bundle. It does not modify
+the OCI image configuration or its filesystem layers.
+
+To replace `config.json` completely, add the replacement to `SRC_URI` and
+install it from a post-processing function:
+
+```bitbake
+SRC_URI += "file://config.json"
+
+OCI_BUNDLE_POSTPROCESS_COMMAND += "install_runtime_config;"
+
+install_runtime_config() {
+    install -m 0644 "${WORKDIR}/config.json" \
+    "config.json"
+}
+```
+
 ### OCI_IMAGE_CMD vs OCI_IMAGE_ENTRYPOINT
 
     # CMD (default) - replaced when user passes arguments
-- 
2.55.0

[meta-virtualization][PATCH 2/2] app-container-oci-bundle.bb: add a new demo recipe using an OCI bundle

[Tanguy Raufflet]

This new recipe can be used to generate an OCI runtime bundle.

Signed-off-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com>
---
 recipes-demo/images/app-container-oci-bundle.bb | 4 ++++
 1 file changed, 4 insertions(+)
 create mode 100644 recipes-demo/images/app-container-oci-bundle.bb

diff --git a/recipes-demo/images/app-container-oci-bundle.bb b/recipes-demo/images/app-container-oci-bundle.bb
new file mode 100644
index 00000000..f4b74b27
--- /dev/null
+++ b/recipes-demo/images/app-container-oci-bundle.bb
@@ -0,0 +1,4 @@
+require app-container.bb
+
+SUMMARY = "Application OCI runtime bundle demo image"
+OCI_IMAGE_BUNDLE_OUTPUT = "1"
-- 
2.55.0

Re: [meta-virtualization][PATCH 1/2] image-oci: add OCI runtime bundle output

[Bruce Ashfield]

Hi Tanguy,

I don't want to take this as it stands, and I want to be clear about
why. It isn't the mechanics of the change, it's the architectural
one.

But this is a good topic to discuss more, since there may be another
way to get what you ar elooking for.

Sorry for the long reply, this is something I'm currently working
through as part of my OSSEU presentation (and next release planning)
that I'm doing this week.

image-oci exists to build OCI images. If a use case wants an
OCI-targeted container, the "sanctioned" way is a real image recipe
with the appropriate image type set (IMAGE_FSTYPES = "oci") and the
OCI_IMAGE_* variables expressing what that container is. The recipe
is where the container's identity lives and where it can be reasoned
about, audited, patched, and rebuilt.

I'll risk saying the obvious, since this will go in the list archive
.. and it is worth being clear about what the two outputs actually
are. The existing -oci-dir[.tar] artifact is an OCI Image Layout
(blobs + manifests). The patch's new .oci-bundle.tar is an OCI Runtime
Bundle (flat rootfs + config.json). Same container in two spec
representations, aimed at different consumers. Image Layout for
interchange (registries, skopeo, cross-arch manifests), Runtime Bundle
for direct exec by runc/crun. `umoci unpack --image <layout>:tag
<bundle-dir>` converts one to the other in a step. So the patch's
output is a strictly-lossier view (no layer sharing, no multi-arch, no
registry push) of the artifact we already ship, produced at build time
in a way the deploy side could have produced itself.

I can see some value in emitting the bundle directly. But that piece
is trivially scriptable downstream from an OCI container that's
already built: one `umoci unpack` at deploy time gets you there, 
and doesn't have to be baked into image-oci as a build-time output
to be reachable.

The part I object to more strongly is the modification-to-make-it-
runnable half. OCI_BUNDLE_POSTPROCESS_COMMAND (jq-mutate config.json,
or drop config.json in from SRC_URI) is hidden logic that mutates
container output bytes after the recipe has finished expressing what
the container is. That can't be a first-class output of the layer.
First-class means the recipe/variables describe the container end-to-
end, and someone reading the recipe can tell what will land in
config.json without also reading a post-hook shell function that
patches it afterward. Layered together with the 2/2 pattern
("require app-container.bb; OCI_IMAGE_BUNDLE_OUTPUT = 1"), it's a
way to define effectively a new container by appending post-hooks
onto an unrelated image recipe. Every image-oci build silently
becomes two products with the second one shaped by hidden logic.
That's the side door.

One clarification on the code comment you cite as prior
encouragement:

    "We could make this optional, since the bundle is directly
     runnable via runc"

That was less about promoting the bundle as a first-class shippable
artifact and more a note about reuse/sharing between image builds --
"we could keep it around instead of rebuilding it every time" reuse
angle. Reading it as a plan for exposing bundles as standalone-
runnable outputs via crun/runc goes further than intended.

That's the reasoning, now to the path to hopefully a solution ....

One question that would help me understand the motivation of what
you're after: is the goal here to produce multiple containers from a
single image construction, and the bundle+postprocess pair is the
plumbing you found that gets close to that? If so, the current patch
is still 1-image-to-1-bundle so it doesn't actually get you there
either, and that's a different discussion worth having on its own.

If the goal is just "1 image, 1 bundle, but with config.json tweaks
the current OCI_IMAGE_* variables can't express", then let me know the
missing knobs and we can look at adding them as first-class variables
in image-oci (the recipe-level path), rather than as a post-hook that
sits outside the image-config machinery.

If there's something clunky or otherwise not working in the image-oci
output, then that is also something to look into fixing. But if we
still require a recipe -> produce one output, then I'm missing why
we need a change like this.

Cheers,

Bruce

Re: [meta-virtualization][PATCH 1/2] image-oci: add OCI runtime bundle output

[Tanguy Raufflet]

Hello Bruce,

Thanks for the review !

> I can see some value in emitting the bundle directly. But that piece
> is trivially scriptable downstream from an OCI container that's
> already built: one `umoci unpack` at deploy time gets you there,
> and doesn't have to be baked into image-oci as a build-time output
> to be reachable.

I am working with an embedded system that that will use container and for
which, we need to maximize the performance. This is why crun was chosen.
For this I need to generate an OCI bundle image, and we want it to be
fully handled by Yocto. Running umoci unpack during deployment would add
a preparation step outside our Yocto build, which does not meet our
project requirements.

> The part I object to more strongly is the modification-to-make-it-
> runnable half. OCI_BUNDLE_POSTPROCESS_COMMAND (jq-mutate config.json,
> or drop config.json in from SRC_URI) is hidden logic that mutates
> container output bytes after the recipe has finished expressing what
> the container is. That can't be a first-class output of the layer.
> First-class means the recipe/variables describe the container end-to-
> end, and someone reading the recipe can tell what will land in
> config.json without also reading a post-hook shell function that
> patches it afterward. Layered together with the 2/2 pattern
> ("require app-container.bb; OCI_IMAGE_BUNDLE_OUTPUT = 1"), it's a
> way to define effectively a new container by appending post-hooks
> onto an unrelated image recipe. Every image-oci build silently
> becomes two products with the second one shaped by hidden logic.
> That's the side door.

Yes, I agree.
I added the post-processing hook because we expect to need runtime
settings such as mounts, capabilities, and namespaces. But it lets the
bundle's configuration diverge through changes that are not expressed
as recipe-level settings. Would first-class, bundle-specific variables
in image-oci be an appropriate way to declare those runtime settings
while generating the bundle from the finalized image?

> One question that would help me understand the motivation of what
> you're after: is the goal here to produce multiple containers from a
> single image construction, and the bundle+postprocess pair is the
> plumbing you found that gets close to that? If so, the current patch
> is still 1-image-to-1-bundle so it doesn't actually get you there
> either, and that's a different discussion worth having on its own.

> If the goal is just "1 image, 1 bundle, but with config.json tweaks
> the current OCI_IMAGE_* variables can't express", then let me know the
> missing knobs and we can look at adding them as first-class variables
> in image-oci (the recipe-level path), rather than as a post-hook that
> sits outside the image-config machinery.

Yes, the goal is simply to produce one OCI bundle with one image recipe.
The OCI image that will also be generated will never be used in my case.
The settings I expect to need include mounts, capabilities, namespaces,
etc. As I understand it, these are runtime-spec settings in config.json,
not fields of the OCI image configuration, so the current OCI_IMAGE_*
variables cannot express them in the image itself

I would be interested to get feedback on the best approach to generate
the OCI bundle fully within Yocto, without relying on post-processing
hooks, so I can update the patch accordingly.

Best regards,
Tanguy Raufflet

Re: [meta-virtualization][PATCH 1/2] image-oci: add OCI runtime bundle output

[Bruce Ashfield]

Hi Tanguy,

Sorry for the slow reply, I've been busy since getting back from
OSSEU.

Dropping the postprocess-hook half clears the architectural concern,
and the real gap you've identified (runtime-spec fields with no
OCI_IMAGE_* equivalent) is worth addressing properly.

Context first, because it probably isn't in anyone else's memory
but mine (as it deals with the historcal path I took to the
current support and bundling):

Some is obvious, some isn't, but I'm stating it for the mailing
list archive.

image-oci already produces a runnable OCI runtime bundle during the
build. The umoci backend creates ${IMAGE_NAME}-oci-bundle/ with a
flat rootfs/ and config.json as a working intermediate (see
classes/image-oci-umoci.inc, the image_bundle_name variable). The
final rm at the end is the only thing that stops it being a
first-class output. The top-of-file docstring in
classes/image-oci.bbclass has walked through this flow since 2021,
including "runc run -b container-base-oci-bundle ctr-build" as the
canonical runnable form -- the capability has been there, it just
hasn't been exposed as a shippable artifact when using umoci.

So what I'm going to do:

1. Reinstate the historical dual-form tar convention described in the
   umoci backend's comment block at classes/image-oci-umoci.inc.  The
   .tar form (not -dir.tar) carries blobs/, oci-layout, index.json AND
   top-level rootfs/, config.json together, so extracting it gives
   both the OCI layout and a runnable bundle side by side. The current
   code only tars the layout contents; the comment is accurate for the
   intent, the code drifted. Fix: preserve $image_bundle_name through
   the tar step and include its contents in $image_name.tar.

   Both backends (umoci and sloci) will honor the same convention.
   Multi-layer squashing is a free consequence of umoci's own
   unpack-after-each-layer-repack flow -- the final bundle rootfs
   already reflects the full stacked tree flat.

   Rather than making this opt-in, this becomes the default output
   going forward for new builds, with the layout-only -dir.tar form
   preserved so existing users don't break.

2. Make bundles cross-installable, parallel to how
   container-cross-install.bbclass already treats docker and podman
   images. New BUNDLED_CONTAINERS flavor along the lines of
   "name:crun[:autostart]" (and/or ":runc" / ":bundle" -- naming
   TBD). The install step extracts rootfs/ + config.json from the
   .tar on-host at build time, lands them in the target rootfs at a
   stable path (/usr/share/oci-bundles/<name>/ or similar), and the
   first-boot generator writes a systemd unit that calls
   "crun run -b <path> <name>" ... similar to the existing
   generate_docker_service_from_bundle / generate_podman_service_
   from_bundle paths. Runtime (crun or runc) gets added to RDEPENDS
   when any BUNDLED_CONTAINERS entry uses the new flavor, same
   mechanism docker/podman use today.

3. First-class runtime-spec recipe variables for the fields you
   actually need to configure: mounts, capabilities, namespaces,
   readonly rootfs, no-new-privileges, resource limits as they
   surface, etc. These live in image-oci as regular recipe
   variables that control what gets emitted into config.json during
   the umoci / sloci build, so the recipe fully describes the
   container's identity end-to-end. We already have OCI_IMAGE_* for
   the image-spec fields (env, cmd, entrypoint, workingdir, user,
   labels, ports); the runtime-spec additions are the natural
   extension. Naming (keep OCI_IMAGE_* for coherence vs. introduce
   OCI_BUNDLE_* for strict spec-family hygiene) is still something
   I need to decide.

If this plan matches what you need, can you send a list of the
variables that you've identified as needed ? Along with expected
runtime results (if any) .. and with that, I can test what I've
pulled together in my WIP branches.

Cheers,

Bruce