[meta-virtualization][PATCH 1/2] image-oci: add OCI runtime bundle output

[meta-virtualization][PATCH 1/2] image-oci: add OCI runtime bundle output

[Tanguy Raufflet]

umoci currently uses an OCI runtime bundle as a temporary artifact
because umoci needs this intermediate representation to construct an OCI
image with the rootfs populated. After this step, the bundle is not used
anymore since umoci does not provide a mechanism to directly configure
the runtime bundle directly.

Because some utilities expect an OCI runtime bundle as input (e.g.,
crun, runc), it is useful to optionally retain the bundle after
constructing the OCI image. The comment [1] in the code was already
suggesting that retaining the OCI runtime bundle could be beneficial.

    "We could make this optional, since the bundle is directly runnable
    via runc"

This commit adds OCI_IMAGE_BUNDLE_OUTPUT to optionally deploy the
runtime bundle as a tar archive into IMGDEPLOYDIR. This bundle contains
the fully populated OCI runtime bundle, including the rootfs and the
config.json file reflecting the final OCI image configuration.

Also, a postprocess command can be specified to either modify the
runtime bundle's configuration or replace it entirely before it is
archived.

Note that image_bundle_name is not generated to the IMGDEPLOYDIR anymore
to avoid issues when BitBake processes and caches files under
IMGDEPLOYDIR.

Signed-off-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com>
---
 classes/image-oci-umoci.inc | 41 ++++++++++++++++++++++--
 classes/image-oci.bbclass   | 13 ++++++++
 docs/container-bundling.md  | 64 +++++++++++++++++++++++++++++++++++++
 3 files changed, 115 insertions(+), 3 deletions(-)

diff --git a/classes/image-oci-umoci.inc b/classes/image-oci-umoci.inc
index 67f6c6e5..ad103f16 100644
--- a/classes/image-oci-umoci.inc
+++ b/classes/image-oci-umoci.inc
@@ -421,6 +421,34 @@ PYEOF
     bbnote "OCI fix: Layer whiteout fix complete"
 }
 
+oci_archive_bundle() {
+    image_tag="${@d.getVar("OCI_IMAGE_TAG").replace(":", "_")}"
+
+    ${IMAGE_CMD_TAR} --sort=name --format=posix --pax-option=delete=atime,delete=ctime --numeric-owner \
+        -cf "${IMGDEPLOYDIR}/${IMAGE_NAME}.oci-bundle.tar" \
+        -C "${OCI_BUNDLE_DIR}" \
+        .
+
+    ln -sfn "${IMAGE_NAME}.oci-bundle.tar" \
+        "${IMGDEPLOYDIR}/${IMAGE_BASENAME}-$image_tag-oci-bundle.tar"
+
+    rm -rf "${OCI_BUNDLE_DIR}"
+}
+
+python oci_bundle_postprocess() {
+    from oe.utils import execute_pre_post_process
+
+    execute_pre_post_process(
+        d, d.getVar('OCI_BUNDLE_POSTPROCESS_COMMAND')
+    )
+    bb.build.exec_func('oci_archive_bundle', d)
+}
+
+oci_bundle_postprocess[dirs] = "${OCI_BUNDLE_DIR}"
+
+do_image_oci[postfuncs] += "${@'oci_bundle_postprocess' if bb.utils.to_boolean(d.getVar('OCI_IMAGE_BUNDLE_OUTPUT')) else ''}"
+do_image_oci[subimages] += "${@'oci-bundle.tar' if bb.utils.to_boolean(d.getVar('OCI_IMAGE_BUNDLE_OUTPUT')) else ''}"
+
 IMAGE_CMD:oci() {
     umoci_options=""
 
@@ -477,7 +505,7 @@ IMAGE_CMD:oci() {
 
     new_image=t
     image_name="${IMAGE_NAME}${IMAGE_NAME_SUFFIX}-oci"
-    image_bundle_name="${IMAGE_NAME}${IMAGE_NAME_SUFFIX}-oci-bundle"
+    image_bundle_name="${OCI_BUNDLE_DIR}"
     if [ -n "$OCI_REUSE_IMAGE" ]; then
 	if [ -d $image_name ]; then
 	    bbdebug 1 "OCI: reusing image directory"
@@ -816,8 +844,15 @@ IMAGE_CMD:oci() {
 	ln -sfn "$image_name-dir.tar" "${IMAGE_BASENAME}-$image_tag-oci-dir.tar"
     fi
 
-    # We could make this optional, since the bundle is directly runnable via runc
-    rm -rf $image_bundle_name
+    if [ "${OCI_IMAGE_BUNDLE_OUTPUT}" = "1" ]; then
+        # Refresh config.json from the finalized image before post-processing.
+        umoci raw runtime-config --rootless \
+            --image "$image_name:${OCI_IMAGE_TAG}" \
+            --rootfs "$image_bundle_name/rootfs" \
+            "$image_bundle_name/config.json"
+    else
+        rm -rf "$image_bundle_name"
+    fi
 
     # This is the OCI image directory, which is technically the "image" as specified
     # Use -n to avoid creating link inside existing symlink target directory
diff --git a/classes/image-oci.bbclass b/classes/image-oci.bbclass
index 141cd2a8..cbbaa388 100644
--- a/classes/image-oci.bbclass
+++ b/classes/image-oci.bbclass
@@ -280,6 +280,19 @@ OCI_LAYER_CACHE_DIR ?= "${TOPDIR}/oci-layer-cache/${MACHINE}"
 # bundled into a tarball.
 OCI_IMAGE_TAR_OUTPUT ?= "true"
 
+# Create a tar archive containing an OCI runtime bundle suitable for OCI runtimes such as crun.
+OCI_IMAGE_BUNDLE_OUTPUT ?= "0"
+
+# Functions appended here run after the final runtime configuration is generated
+# and before the bundle is archived. The bundle directory is available as
+# OCI_BUNDLE_DIR.
+OCI_BUNDLE_DIR = "${WORKDIR}/${IMAGE_BASENAME}-oci-bundle"
+OCI_BUNDLE_POSTPROCESS_COMMAND ?= ""
+
+python () {
+    d.setVarFlag('OCI_BUNDLE_POSTPROCESS_COMMAND', 'vardeps', d.getVar('OCI_BUNDLE_POSTPROCESS_COMMAND'))
+}
+
 # Generate a subarch that is appropriate to OCI image
 # types. This is typically only ARM architectures at the
 # moment.
diff --git a/docs/container-bundling.md b/docs/container-bundling.md
index 77aa09b0..bedd541e 100644
--- a/docs/container-bundling.md
+++ b/docs/container-bundling.md
@@ -288,6 +288,70 @@ Disable caching with `OCI_LAYER_CACHE = "0"` if you:
 - Need fully reproducible builds with no local state
 - Are debugging package installation issues
 
+### OCI Runtime Bundles
+
+The umoci backend can emit an OCI runtime bundle suitable for runtimes such
+as crun and runc:
+
+    OCI_IMAGE_BUNDLE_OUTPUT = "1"
+
+The `app-container-oci-bundle` demo image enables this output and can be
+built with `bitbake app-container-oci-bundle`.
+
+The runtime configuration is generated from the finalized OCI image, so its
+`config.json` includes settings applied through `OCI_IMAGE_ENV_VARS`,
+`OCI_IMAGE_CMD`, `OCI_IMAGE_ENTRYPOINT`, `OCI_IMAGE_RUNTIME_UID`, and
+`OCI_IMAGE_WORKINGDIR`. It is deployed as:
+
+    ${IMAGE_BASENAME}-${OCI_IMAGE_TAG}-oci-bundle.tar
+
+with the convenience symlink:
+
+    ${IMAGE_BASENAME}-oci-bundle.tar
+
+`OCI_IMAGE_BUNDLE_OUTPUT` is independent of `OCI_IMAGE_TAR_OUTPUT`. The
+latter controls tar output for the OCI image layout (`index.json` and
+`blobs/`), while the runtime bundle contains `config.json` and `rootfs/`.
+
+Extract and run the bundle with crun:
+
+    mkdir my-container-bundle
+    tar -xf my-container-oci-bundle.tar -C my-container-bundle
+    crun run --bundle my-container-bundle my-container
+
+#### Bundle Post-processing
+
+Functions listed in `OCI_BUNDLE_POSTPROCESS_COMMAND` run after the finalized
+image is unpacked and before the bundle is archived. They run with
+`OCI_BUNDLE_DIR` as their current working directory:
+
+```bitbake
+OCI_BUNDLE_POSTPROCESS_COMMAND += "configure_runtime_bundle;"
+
+configure_runtime_bundle() {
+    config="config.json"
+    jq '.process.terminal = false' "$config" > "$config.tmp"
+    mv "$config.tmp" "$config"
+}
+```
+
+Post-processing changes only the emitted runtime bundle. It does not modify
+the OCI image configuration or its filesystem layers.
+
+To replace `config.json` completely, add the replacement to `SRC_URI` and
+install it from a post-processing function:
+
+```bitbake
+SRC_URI += "file://config.json"
+
+OCI_BUNDLE_POSTPROCESS_COMMAND += "install_runtime_config;"
+
+install_runtime_config() {
+    install -m 0644 "${WORKDIR}/config.json" \
+    "config.json"
+}
+```
+
 ### OCI_IMAGE_CMD vs OCI_IMAGE_ENTRYPOINT
 
     # CMD (default) - replaced when user passes arguments
-- 
2.55.0

[meta-virtualization][PATCH 2/2] app-container-oci-bundle.bb: add a new demo recipe using an OCI bundle

[Tanguy Raufflet]

This new recipe can be used to generate an OCI runtime bundle.

Signed-off-by: Tanguy Raufflet <tanguy.raufflet@savoirfairelinux.com>
---
 recipes-demo/images/app-container-oci-bundle.bb | 4 ++++
 1 file changed, 4 insertions(+)
 create mode 100644 recipes-demo/images/app-container-oci-bundle.bb

diff --git a/recipes-demo/images/app-container-oci-bundle.bb b/recipes-demo/images/app-container-oci-bundle.bb
new file mode 100644
index 00000000..f4b74b27
--- /dev/null
+++ b/recipes-demo/images/app-container-oci-bundle.bb
@@ -0,0 +1,4 @@
+require app-container.bb
+
+SUMMARY = "Application OCI runtime bundle demo image"
+OCI_IMAGE_BUNDLE_OUTPUT = "1"
-- 
2.55.0

Re: [meta-virtualization][PATCH 1/2] image-oci: add OCI runtime bundle output

[Bruce Ashfield]

Hi Tanguy,

I don't want to take this as it stands, and I want to be clear about
why. It isn't the mechanics of the change, it's the architectural
one.

But this is a good topic to discuss more, since there may be another
way to get what you ar elooking for.

Sorry for the long reply, this is something I'm currently working
through as part of my OSSEU presentation (and next release planning)
that I'm doing this week.

image-oci exists to build OCI images. If a use case wants an
OCI-targeted container, the "sanctioned" way is a real image recipe
with the appropriate image type set (IMAGE_FSTYPES = "oci") and the
OCI_IMAGE_* variables expressing what that container is. The recipe
is where the container's identity lives and where it can be reasoned
about, audited, patched, and rebuilt.

I'll risk saying the obvious, since this will go in the list archive
.. and it is worth being clear about what the two outputs actually
are. The existing -oci-dir[.tar] artifact is an OCI Image Layout
(blobs + manifests). The patch's new .oci-bundle.tar is an OCI Runtime
Bundle (flat rootfs + config.json). Same container in two spec
representations, aimed at different consumers. Image Layout for
interchange (registries, skopeo, cross-arch manifests), Runtime Bundle
for direct exec by runc/crun. `umoci unpack --image <layout>:tag
<bundle-dir>` converts one to the other in a step. So the patch's
output is a strictly-lossier view (no layer sharing, no multi-arch, no
registry push) of the artifact we already ship, produced at build time
in a way the deploy side could have produced itself.

I can see some value in emitting the bundle directly. But that piece
is trivially scriptable downstream from an OCI container that's
already built: one `umoci unpack` at deploy time gets you there, 
and doesn't have to be baked into image-oci as a build-time output
to be reachable.

The part I object to more strongly is the modification-to-make-it-
runnable half. OCI_BUNDLE_POSTPROCESS_COMMAND (jq-mutate config.json,
or drop config.json in from SRC_URI) is hidden logic that mutates
container output bytes after the recipe has finished expressing what
the container is. That can't be a first-class output of the layer.
First-class means the recipe/variables describe the container end-to-
end, and someone reading the recipe can tell what will land in
config.json without also reading a post-hook shell function that
patches it afterward. Layered together with the 2/2 pattern
("require app-container.bb; OCI_IMAGE_BUNDLE_OUTPUT = 1"), it's a
way to define effectively a new container by appending post-hooks
onto an unrelated image recipe. Every image-oci build silently
becomes two products with the second one shaped by hidden logic.
That's the side door.

One clarification on the code comment you cite as prior
encouragement:

    "We could make this optional, since the bundle is directly
     runnable via runc"

That was less about promoting the bundle as a first-class shippable
artifact and more a note about reuse/sharing between image builds --
"we could keep it around instead of rebuilding it every time" reuse
angle. Reading it as a plan for exposing bundles as standalone-
runnable outputs via crun/runc goes further than intended.

That's the reasoning, now to the path to hopefully a solution ....

One question that would help me understand the motivation of what
you're after: is the goal here to produce multiple containers from a
single image construction, and the bundle+postprocess pair is the
plumbing you found that gets close to that? If so, the current patch
is still 1-image-to-1-bundle so it doesn't actually get you there
either, and that's a different discussion worth having on its own.

If the goal is just "1 image, 1 bundle, but with config.json tweaks
the current OCI_IMAGE_* variables can't express", then let me know the
missing knobs and we can look at adding them as first-class variables
in image-oci (the recipe-level path), rather than as a post-hook that
sits outside the image-config machinery.

If there's something clunky or otherwise not working in the image-oci
output, then that is also something to look into fixing. But if we
still require a recipe -> produce one output, then I'm missing why
we need a change like this.

Cheers,

Bruce

Re: [meta-virtualization][PATCH 1/2] image-oci: add OCI runtime bundle output

[Tanguy Raufflet]

Hello Bruce,

Thanks for the review !

> I can see some value in emitting the bundle directly. But that piece
> is trivially scriptable downstream from an OCI container that's
> already built: one `umoci unpack` at deploy time gets you there,
> and doesn't have to be baked into image-oci as a build-time output
> to be reachable.

I am working with an embedded system that that will use container and for
which, we need to maximize the performance. This is why crun was chosen.
For this I need to generate an OCI bundle image, and we want it to be
fully handled by Yocto. Running umoci unpack during deployment would add
a preparation step outside our Yocto build, which does not meet our
project requirements.

> The part I object to more strongly is the modification-to-make-it-
> runnable half. OCI_BUNDLE_POSTPROCESS_COMMAND (jq-mutate config.json,
> or drop config.json in from SRC_URI) is hidden logic that mutates
> container output bytes after the recipe has finished expressing what
> the container is. That can't be a first-class output of the layer.
> First-class means the recipe/variables describe the container end-to-
> end, and someone reading the recipe can tell what will land in
> config.json without also reading a post-hook shell function that
> patches it afterward. Layered together with the 2/2 pattern
> ("require app-container.bb; OCI_IMAGE_BUNDLE_OUTPUT = 1"), it's a
> way to define effectively a new container by appending post-hooks
> onto an unrelated image recipe. Every image-oci build silently
> becomes two products with the second one shaped by hidden logic.
> That's the side door.

Yes, I agree.
I added the post-processing hook because we expect to need runtime
settings such as mounts, capabilities, and namespaces. But it lets the
bundle's configuration diverge through changes that are not expressed
as recipe-level settings. Would first-class, bundle-specific variables
in image-oci be an appropriate way to declare those runtime settings
while generating the bundle from the finalized image?

> One question that would help me understand the motivation of what
> you're after: is the goal here to produce multiple containers from a
> single image construction, and the bundle+postprocess pair is the
> plumbing you found that gets close to that? If so, the current patch
> is still 1-image-to-1-bundle so it doesn't actually get you there
> either, and that's a different discussion worth having on its own.

> If the goal is just "1 image, 1 bundle, but with config.json tweaks
> the current OCI_IMAGE_* variables can't express", then let me know the
> missing knobs and we can look at adding them as first-class variables
> in image-oci (the recipe-level path), rather than as a post-hook that
> sits outside the image-config machinery.

Yes, the goal is simply to produce one OCI bundle with one image recipe.
The OCI image that will also be generated will never be used in my case.
The settings I expect to need include mounts, capabilities, namespaces,
etc. As I understand it, these are runtime-spec settings in config.json,
not fields of the OCI image configuration, so the current OCI_IMAGE_*
variables cannot express them in the image itself

I would be interested to get feedback on the best approach to generate
the OCI bundle fully within Yocto, without relying on post-processing
hooks, so I can update the patch accordingly.

Best regards,
Tanguy Raufflet