[meta-virtualization][PATCH 0/7] Container improvements

[meta-virtualization][PATCH 0/7] Container improvements

[Tim Orling]

This series:
  * Adds a class to create/run containers with a non-root user
  * Adds new containers:
    - app-container-python
    - app-containter-mosquitto
    - app-container-valkey
    - app-container-nginx
  * Modifies app-container-curl to be more like the upstream
    experience (and more like the above containers)
  * Allows meta-webserver/recipes-http to be parsed for
    vcontainer distro so we can build multiarch containers
    for app-container-nginx, etc.

Each of these containers was built in a MACHINE=qemuarm64
environment as well as mc:container-amd64+mc:container-arm64
multiarch environment.

The resulting containers were tested with simple command line
usage compared to Docker provided equivalents to ensure the
same expected behavior.

Tim Orling (7):
  classes: add container-nonroot-user.bbclass
  recipes-containers/images: add app-container-python
  recipes-containers/images: add app-container-mosquitto
  recipes-containers/images: add app-container-valkey
  recipes-containers/images: add app-container-nginx
  vcontainer-bbmask.inc: allow meta-webserver/recipes-httpd
  app-container-curl: use multilayer mode; container-nonroot-user

 classes/container-nonroot-user.bbclass        | 68 ++++++++++++++++
 conf/distro/include/vcontainer-bbmask.inc     |  2 +-
 conf/layer.conf                               |  1 +
 .../images/app-container-curl.bb              | 29 ++++++-
 .../images/app-container-mosquitto.bb         | 46 +++++++++++
 .../images/app-container-nginx.bb             | 77 +++++++++++++++++++
 .../images/app-container-python.bb            | 57 ++++++++++++++
 .../images/app-container-valkey.bb            | 61 +++++++++++++++
 8 files changed, 336 insertions(+), 5 deletions(-)
 create mode 100644 classes/container-nonroot-user.bbclass
 rename {recipes-demo => recipes-containers}/images/app-container-curl.bb (58%)
 create mode 100644 recipes-containers/images/app-container-mosquitto.bb
 create mode 100644 recipes-containers/images/app-container-nginx.bb
 create mode 100644 recipes-containers/images/app-container-python.bb
 create mode 100644 recipes-containers/images/app-container-valkey.bb

-- 
2.54.0

[meta-virtualization][PATCH 1/7] classes: add container-nonroot-user.bbclass

[Tim Orling]

For secure and production environments, we want to run containers as a
non-root user. Some applications, such as Python, require a $HOME
directory with proper permissions. Because OCI_LAYERS :directories:
copies with 'cp -a --no-preserve=ownership', we need a fixup function
to create the proper permissions and ownership in a new raw layer.

The behavior here is inspired by dhi.io/python:3 (Docker Hardened Image)

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 classes/container-nonroot-user.bbclass | 68 ++++++++++++++++++++++++++
 1 file changed, 68 insertions(+)
 create mode 100644 classes/container-nonroot-user.bbclass

diff --git a/classes/container-nonroot-user.bbclass b/classes/container-nonroot-user.bbclass
new file mode 100644
index 00000000..5139ce8b
--- /dev/null
+++ b/classes/container-nonroot-user.bbclass
@@ -0,0 +1,68 @@
+# For secure and production environments, we want to run containers as a
+# non-root user. Some applications, such as Python, require a $HOME
+# directory with proper permissions. Because OCI_LAYERS :directories:
+# copies with 'cp -a --no-preserve=ownership', we need a fixup function
+# to create the proper permissions and ownership in a new raw layer.
+
+# The behavior here is inspired by dhi.io/python:3 (Docker Hardened Image)
+
+inherit extrausers
+
+NONROOT_USER ?= "nonroot"
+NONROOT_UID ?= "65532"
+NONROOT_GID ?= "65532"
+
+# ---------------------------------------------------------------------------
+# Create the unprivileged "nonroot" user (uid 65532, group 65532)
+# ---------------------------------------------------------------------------
+EXTRA_USERS_PARAMS = "\
+    groupadd -g ${NONROOT_GID} ${NONROOT_USER}; \
+    useradd -m -u ${NONROOT_UID} -g ${NONROOT_GID} -d /home/${NONROOT_USER}  ${NONROOT_USER}; \
+"
+
+# Allow a container to choose to run as 'root'
+OCI_IMAGE_RUNTIME_UID ?= "${NONROOT_UID}"
+OCI_IMAGE_ENV_VARS = "HOME=/home/${NONROOT_USER}"
+
+# Make sure we can write to e.g. /home/nonroot/.python_history
+# using :directories: in OCI_LAYERS does not preserve permissions
+fakeroot fix_oci_home_perms() {
+    cd ${IMGDEPLOYDIR}
+    image_name="${IMAGE_NAME}${IMAGE_NAME_SUFFIX}-oci"
+    layer_tar="${WORKDIR}/oci-home-fix-layer.tar"
+
+    rm -f "$layer_tar"
+
+    python3 - "$layer_tar" <<'PYEOF'
+import sys, tarfile, time
+
+layer_tar = sys.argv[1]
+mtime = int(time.time())
+
+# (path, mode, uid, gid)
+entries = [
+    ("home",         0o755, 0,     0),
+    ("home/${NONROOT_USER}", 0o700, ${NONROOT_UID}, ${NONROOT_GID}),
+]
+
+with tarfile.open(layer_tar, "w") as tar:
+    for name, mode, uid, gid in entries:
+        info = tarfile.TarInfo(name=name)
+        info.type  = tarfile.DIRTYPE
+        info.mode  = mode
+        info.uid   = uid
+        info.gid   = gid
+        info.uname = ""   # numeric-only; let umoci canonicalize
+        info.gname = ""
+        info.mtime = mtime
+        tar.addfile(info)
+PYEOF
+
+    umoci raw add-layer --image "$image_name:${OCI_IMAGE_TAG}" "$layer_tar"
+    rm -f "$layer_tar"
+
+    rm -f "$image_name.tar" "$image_name-dir.tar"
+    ( cd "$image_name" && tar -cf "../$image_name.tar" "." )
+    tar -cf "$image_name-dir.tar" "$image_name"
+}
+do_image_oci[postfuncs] += "fix_oci_home_perms"
-- 
2.54.0

[meta-virtualization][PATCH 2/7] recipes-containers/images: add app-container-python

[Tim Orling]

Add OCI container image recipe for Python to use as a base for
other Python app containers. The image uses multi-layer mode with
separate base, terminal and python layers.

Add ncurses-terminfo-base to a "terminal" layer to avoid warnings in the
REPL:
  "Cannot read termcap database;
  using dumb terminal settings."

Add coreutils to "python" layer to provide /usr/bin/env needed by
python3-idle in python3-modules.

Inherit container-nonroot-user and run a `nonroot` user by default.
Set PACKAGECONFIG:pn-app-container-python = "dev" in local.conf or
distro/image config to run as 'root' and include 'pip'.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 .../images/app-container-python.bb            | 57 +++++++++++++++++++
 1 file changed, 57 insertions(+)
 create mode 100644 recipes-containers/images/app-container-python.bb

diff --git a/recipes-containers/images/app-container-python.bb b/recipes-containers/images/app-container-python.bb
new file mode 100644
index 00000000..a93f1b0f
--- /dev/null
+++ b/recipes-containers/images/app-container-python.bb
@@ -0,0 +1,57 @@
+SUMMARY = "Base python3 container image"
+DESCRIPTION = "OCI container image running Python with non-root user. \
+\
+In "dev" mode, can optionally run as 'root' and add 'pip' to allow \
+developers to simply run 'pip install' on top of this container (Not \
+advised for production/hardened use)."
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+# Multi-layer mode: create explicit layers instead of single rootfs layer
+OCI_LAYER_MODE = "multi"
+
+# Optional 'dev' mode:
+#   - adds python3-pip to the python layer (enables `pip install` at runtime)
+#   - runs the container as root (UID 0) so pip can write to site-packages
+# Enable with: PACKAGECONFIG:pn-app-container-python = "dev"
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[dev] = ""
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for multiple items
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    terminal:packages:ncurses-terminfo-base \
+    python:packages:python3+coreutils${@bb.utils.contains('PACKAGECONFIG', 'dev', '+python3-pip', '', d)} \
+"
+
+# In 'dev' mode, override the nonroot UID inherited from container-nonroot-user
+# so the container runs as root (required for `pip install`).
+OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG', 'dev', '0', '${NONROOT_UID}', d)}"
+
+# Use CMD so `docker run image /bin/sh` works as expected
+OCI_IMAGE_CMD = "python3"
+
+IMAGE_FSTYPES = "container oci"
+inherit image
+inherit image-oci
+inherit container-nonroot-user
+
+IMAGE_FEATURES = ""
+IMAGE_LINGUAS = ""
+NO_RECOMMENDATIONS = "1"
+
+# IMAGE_INSTALL triggers package builds via do_rootfs recrdeptask.
+# Even for multi-layer mode, list packages here to ensure they're built.
+# The PM will install them directly to layers from DEPLOY_DIR_IPK.
+# Note: IMAGE_ROOTFS is still created but ignored for packages layers.
+IMAGE_INSTALL = "base-files base-passwd netbase"
+IMAGE_INSTALL += "ncurses-terminfo-base"
+IMAGE_INSTALL += "python3 coreutils"
+IMAGE_INSTALL += "${@bb.utils.contains('PACKAGECONFIG', 'dev', 'python3-pip', '', d)}"
+
+# Allow build with or without a specific kernel
+IMAGE_CONTAINER_NO_DUMMY = "1"
+
+# Note: No ROOTFS_POSTPROCESS_COMMAND needed - IMAGE_ROOTFS is empty
+# and PM handles installation directly to OCI layers
-- 
2.54.0

[meta-virtualization][PATCH 3/7] recipes-containers/images: add app-container-mosquitto

[Tim Orling]

Add OCI container image recipe for the Eclipse Mosquitto MQTT broker.
The image uses multi-layer mode with separate base and mosquitto layers,
exposes standard MQTT (1883) and WebSocket (9001) ports, and launches
mosquitto with its default config file as the entrypoint.

Inherit container-nonroot-user to run as 'nonroot' with UID 65532.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 .../images/app-container-mosquitto.bb         | 46 +++++++++++++++++++
 1 file changed, 46 insertions(+)
 create mode 100644 recipes-containers/images/app-container-mosquitto.bb

diff --git a/recipes-containers/images/app-container-mosquitto.bb b/recipes-containers/images/app-container-mosquitto.bb
new file mode 100644
index 00000000..37e34a47
--- /dev/null
+++ b/recipes-containers/images/app-container-mosquitto.bb
@@ -0,0 +1,46 @@
+SUMMARY = "Mosquitto MQTT broker container image"
+DESCRIPTION = "OCI container running the Eclipse Mosquitto MQTT broker \
+with standard MQTT (1883) and WebSocket (9001) listeners enabled."
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+# Multi-layer mode: create explicit layers instead of single rootfs layer
+OCI_LAYER_MODE = "multi"
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for multiple items
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    mosquitto:packages:mosquitto \
+"
+
+IMAGE_FSTYPES = "container oci"
+inherit image
+inherit image-oci
+inherit container-nonroot-user
+
+IMAGE_FEATURES = ""
+IMAGE_LINGUAS = ""
+NO_RECOMMENDATIONS = "1"
+
+IMAGE_INSTALL = " \
+    base-files \
+    base-passwd \
+    netbase \
+    mosquitto \
+"
+
+# Allow build with or without a specific kernel
+IMAGE_CONTAINER_NO_DUMMY = "1"
+
+# Workaround /var/volatile for now
+ROOTFS_POSTPROCESS_COMMAND += "rootfs_fixup_var_volatile ; "
+rootfs_fixup_var_volatile () {
+    install -m 1777 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/tmp
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/log
+}
+
+OCI_IMAGE_ENTRYPOINT = "${sbindir}/mosquitto"
+OCI_IMAGE_ENTRYPOINT_ARGS = "-c '${sysconfdir}/mosquitto/mosquitto.conf'"
+OCI_IMAGE_PORTS = "1883/tcp 9001/tcp"
+OCI_IMAGE_TAG = "latest"
-- 
2.54.0

[meta-virtualization][PATCH 4/7] recipes-containers/images: add app-container-valkey

[Tim Orling]

Add OCI container image recipe for the Valkey in-memory key-value
datastore. The image uses multi-layer mode with separate base and
valkey layers, exposes the standard Valkey port (6379), and launches
valkey-server with its default config file as the entrypoint.

The stock valkey.conf shipped by meta-oe is tuned for a host install
(daemonize yes, syslog-enabled yes, bind 127.0.0.1). Override those at
launch so the server stays in the foreground as PID 1, logs to stdout,
and is reachable from outside the container.

Inherit container-nonroot-user to run as 'nonroot' with UID 65532 by
default. Can optionally set PACKAGECONFIG:pn-app-container-valkey = "dev"
in local.conf or a distro/image config to run as root.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 .../images/app-container-valkey.bb            | 61 +++++++++++++++++++
 1 file changed, 61 insertions(+)
 create mode 100644 recipes-containers/images/app-container-valkey.bb

diff --git a/recipes-containers/images/app-container-valkey.bb b/recipes-containers/images/app-container-valkey.bb
new file mode 100644
index 00000000..b3da2efd
--- /dev/null
+++ b/recipes-containers/images/app-container-valkey.bb
@@ -0,0 +1,61 @@
+SUMMARY = "Valkey key-value store container image"
+DESCRIPTION = "OCI container running the Valkey in-memory key-value \
+datastore, a flexible distributed datastore that supports both caching \
+and beyond caching workloads."
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+# Multi-layer mode: create explicit layers instead of single rootfs layer
+OCI_LAYER_MODE = "multi"
+
+# Optional 'dev' mode:
+#   - runs the container as root (UID 0)
+# Enable with: PACKAGECONFIG:pn-app-container-valkey = "dev"
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[dev] = ""
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for multiple items
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    valkey:packages:valkey \
+"
+
+IMAGE_FSTYPES = "container oci"
+inherit image
+inherit image-oci
+inherit container-nonroot-user
+
+IMAGE_FEATURES = ""
+IMAGE_LINGUAS = ""
+NO_RECOMMENDATIONS = "1"
+
+IMAGE_INSTALL = " \
+    base-files \
+    base-passwd \
+    netbase \
+    valkey \
+"
+
+# Allow build with or without a specific kernel
+IMAGE_CONTAINER_NO_DUMMY = "1"
+
+# Workaround /var/volatile for now
+ROOTFS_POSTPROCESS_COMMAND += "rootfs_fixup_var_volatile ; "
+rootfs_fixup_var_volatile () {
+    install -m 1777 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/tmp
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/log
+}
+
+OCI_IMAGE_ENTRYPOINT = "${bindir}/valkey-server"
+# The stock valkey.conf shipped by meta-oe is tuned for a host install
+# (daemonize yes, syslog-enabled yes, bind 127.0.0.1). Override those at
+# launch so the server stays in the foreground as PID 1, logs to stdout,
+# and is reachable from outside the container.
+OCI_IMAGE_ENTRYPOINT_ARGS = "'${sysconfdir}/valkey/valkey.conf' \
+    --daemonize no \
+    --syslog-enabled no \
+    --bind '0.0.0.0 -::*' \
+    --protected-mode no"
+OCI_IMAGE_PORTS = "6379/tcp"
+OCI_IMAGE_TAG = "latest"
-- 
2.54.0

[meta-virtualization][PATCH 5/7] recipes-containers/images: add app-container-nginx

[Tim Orling]

Add OCI container image recipe for the NGINX web server. The image
uses multi-layer mode with separate base, nginx packages, nginx
runtime directories, and nginx log file layers. Exposes the standard
HTTP port (80) and launches nginx with 'daemon off' so it stays in
the foreground as PID 1 and logs to stderr.

Add ROOTFS_POSTPROCESS fixups to create the runtime directories
nginx expects: /var/volatile/{tmp,log}, /var/log/nginx (resolved
explicitly to guarantee inclusion in the container layer regardless
of /var/log symlink ordering), and /run/nginx for nginx's compiled-in
temp paths (client_body_temp, proxy_temp, etc.) which are not created
by any package. Also create empty /var/log/nginx/{access,error}.log
to avoid do_image_oci warnings.

Inherit container-nonroot-user with NONROOT_USER = "nginx" to run with
UID 65532 by default. Set PACKAGECONFIG:pn-app-container-nginx = "dev"
in local.conf or distro/image config to run as 'root'.

Add SKIP_RECIPE and comment to layer.conf since nginx requires
meta-webserver.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 conf/layer.conf                               |  1 +
 .../images/app-container-nginx.bb             | 77 +++++++++++++++++++
 2 files changed, 78 insertions(+)
 create mode 100644 recipes-containers/images/app-container-nginx.bb

diff --git a/conf/layer.conf b/conf/layer.conf
index 2a4a4c91..6ea8ccf8 100644
--- a/conf/layer.conf
+++ b/conf/layer.conf
@@ -33,6 +33,7 @@ LAYERDEPENDS_virtualization-layer = " \
 # webserver:
 # - naigos requires apache2
 # - cockpit-machines requires cockpit
+# - app-container-nginx requires nginx
 LAYERRECOMMENDS_virtualization-layer = " \
     webserver \
     selinux \
diff --git a/recipes-containers/images/app-container-nginx.bb b/recipes-containers/images/app-container-nginx.bb
new file mode 100644
index 00000000..09844376
--- /dev/null
+++ b/recipes-containers/images/app-container-nginx.bb
@@ -0,0 +1,77 @@
+SUMMARY = "Base NGINX container image for development"
+DESCRIPTION = "OCI container with NGINX web server."
+LICENSE = "MIT"
+LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
+
+# Multi-layer mode: create explicit layers instead of single rootfs layer
+OCI_LAYER_MODE = "multi"
+
+# Optional 'dev' mode:
+#   - runs the container as root (UID 0)
+# Enable with: PACKAGECONFIG:pn-app-container-nginx = "dev"
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[dev] = ""
+NONROOT_USER = "nginx"
+
+OCI_IMAGE_APP_RECIPE = "nginx"
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for multiple items
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    nginx:packages:nginx \
+    nginx-dirs:directories:${localstatedir}/log/nginx+/run/nginx+${localstatedir}/volatile/tmp+${localstatedir}/volatile/log \
+    nginx-files:files:${localstatedir}/log/nginx/access.log+${localstatedir}/log/nginx/error.log \
+"
+# Use CMD so `docker run image /bin/sh` works as expected
+OCI_IMAGE_CMD = ""
+
+IMAGE_FSTYPES = "container oci"
+inherit image
+inherit image-oci
+inherit container-nonroot-user
+
+IMAGE_FEATURES = ""
+IMAGE_LINGUAS = ""
+NO_RECOMMENDATIONS = "1"
+
+IMAGE_INSTALL = " \
+       base-files \
+       base-passwd \
+       netbase \
+       nginx \
+"
+
+# Allow build with or without a specific kernel
+IMAGE_CONTAINER_NO_DUMMY = "1"
+
+# Workaround /var/volatile for now
+ROOTFS_POSTPROCESS_COMMAND:append = " rootfs_fixup_var_volatile ; "
+rootfs_fixup_var_volatile () {
+    install -m 1777 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/tmp
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/log
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/log/nginx
+
+    # Fix do_image_oci warnings
+    # OCI: File not found in IMAGE_ROOTFS: /var/log/nginx/access.log
+    touch ${IMAGE_ROOTFS}/${localstatedir}/volatile/log/nginx/access.log
+    touch ${IMAGE_ROOTFS}/${localstatedir}/volatile/log/nginx/error.log
+
+    # nginx opens the compiled-in error_log path before reading -c config.
+    # /var/log is typically a symlink to /var/volatile/log in a Yocto rootfs,
+    # so create the target path explicitly to guarantee the directory lands in
+    # the container layer regardless of symlink resolution order.
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/log
+    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/log/nginx
+
+    # nginx's compiled-in temp paths (client_body_temp, proxy_temp, etc.) all
+    # live under /run/nginx, which is not created by any package.
+    install -m 755 -d ${IMAGE_ROOTFS}/run/nginx
+}
+
+OCI_IMAGE_ENTRYPOINT = "/usr/sbin/nginx"
+OCI_IMAGE_ENTRYPOINT_ARGS = "-g 'daemon off; error_log stderr notice;'"
+OCI_IMAGE_PORTS = "80/tcp"
+OCI_IMAGE_TAG = "latest"
+
+SKIP_RECIPE[app-container-nginx] ?= "${@bb.utils.contains('BBFILE_COLLECTIONS', 'webserver', '', 'Depends on meta-webserver which is not included', d)}"
-- 
2.54.0

[meta-virtualization][PATCH 6/7] vcontainer-bbmask.inc: allow meta-webserver/recipes-httpd

[Tim Orling]

Allow us to build nginx, apache2, etc. multiarch containers.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 conf/distro/include/vcontainer-bbmask.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/conf/distro/include/vcontainer-bbmask.inc b/conf/distro/include/vcontainer-bbmask.inc
index ac74464f..077255cb 100644
--- a/conf/distro/include/vcontainer-bbmask.inc
+++ b/conf/distro/include/vcontainer-bbmask.inc
@@ -93,6 +93,7 @@ BBMASK += "meta-networking/recipes-(?!filter|support)"
 BBMASK += "meta-openstack/recipes-dbs/postgresql/"
 BBMASK += "meta-oe/dynamic-layers/networking-layer/recipes-core/"
 BBMASK += "meta-openstack/recipes-extended/libvirt/"
+BBMASK += "meta-webserver/recipes-(?!httpd)"
 
 # ---------------------------------------------------------------------------
 # Entire layers with 0 recipes in the container image dependency graph.
@@ -101,7 +102,6 @@ BBMASK += "meta-openstack/recipes-extended/libvirt/"
 # ---------------------------------------------------------------------------
 BBMASK += "meta-filesystems/"
 BBMASK += "meta-python/"
-BBMASK += "meta-webserver/"
 # Warning suppression for these fully-masked layers is in meta-virt-host.conf
 # (BBFILE_PATTERN_IGNORE_EMPTY) because BitBake checks the base datastore,
 # not per-multiconfig datastores.
-- 
2.54.0

[meta-virtualization][PATCH 7/7] app-container-curl: use multilayer mode; container-nonroot-user

[Tim Orling]

* Move to recipes-containers/images to show maintenance intent
* Switch to multilayer mode to more like the other "library"/"official"
  container recipes.
* Change OCI_IMAGE_TAG to "latest" for similar reasons.
* Change OCI_IMAGE_ENTRYPOINT_ARGS to "--help" to be more like upstream
  containers.
* Install ca-certificates to enable handling https:// sites
* Inherit container-nonroot-user to run as 'nonroot' with UID 65532 by
  default. Set PACKAGECONFIG:pn-app-container-curl = "dev" in local.conf
  or distro/image config to run as `root` and include a CONTAINER_SHELL.

Signed-off-by: Tim Orling <tim.orling@konsulko.com>
---
 .../images/app-container-curl.bb              | 29 ++++++++++++++++---
 1 file changed, 25 insertions(+), 4 deletions(-)
 rename {recipes-demo => recipes-containers}/images/app-container-curl.bb (58%)

diff --git a/recipes-demo/images/app-container-curl.bb b/recipes-containers/images/app-container-curl.bb
similarity index 58%
rename from recipes-demo/images/app-container-curl.bb
rename to recipes-containers/images/app-container-curl.bb
index ddeb3022..34204fb9 100644
--- a/recipes-demo/images/app-container-curl.bb
+++ b/recipes-containers/images/app-container-curl.bb
@@ -2,9 +2,31 @@ SUMMARY = "Curl Application container image"
 LICENSE = "MIT"
 LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
 
+# Multi-layer mode: create explicit layers instead of single rootfs layer
+OCI_LAYER_MODE = "multi"
+
+# Optional 'dev' mode:
+#   - runs the container as root (UID 0)
+# Enable with: PACKAGECONFIG:pn-app-container-curl = "dev"
+PACKAGECONFIG ??= ""
+PACKAGECONFIG[dev] = ""
+
+# Define layers: each layer contains specific packages
+# Format: "name:type:content" where content uses + as delimiter for multiple items
+OCI_LAYERS = "\
+    base:packages:base-files+base-passwd+netbase \
+    ${@bb.utils.contains('PACKAGECONFIG', 'dev', 'shell:packages:${CONTAINER_SHELL}', '', d)} \
+    curl:packages:curl+ca-certificates \
+"
+
+# In 'dev' mode, override the nonroot UID inherited from container-nonroot-user
+# so the container runs as root.
+OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG', 'dev', '0', '${NONROOT_UID}', d)}"
+
 IMAGE_FSTYPES = "container oci"
 inherit image
 inherit image-oci
+inherit container-nonroot-user
 
 IMAGE_FEATURES = ""
 IMAGE_LINGUAS = ""
@@ -39,8 +61,7 @@ rootfs_fixup_var_volatile () {
 }
 
 OCI_IMAGE_ENTRYPOINT = "curl"
-OCI_IMAGE_TAG = "easy"
-OCI_IMAGE_ENTRYPOINT_ARGS = "http://localhost:80"
-CONTAINER_SHELL = "busybox"
+OCI_IMAGE_TAG = "latest"
+OCI_IMAGE_ENTRYPOINT_ARGS = "--help"
 
-IMAGE_INSTALL:append = " curl"
+IMAGE_INSTALL:append = " curl ca-certificates"
-- 
2.54.0

Re: [meta-virtualization][PATCH 2/7] recipes-containers/images: add app-container-python

[Paul Barker]

[-- Attachment #1: Type: text/plain, Size: 4451 bytes --]

On Fri, 2026-05-29 at 18:31 -0700, Tim Orling via lists.yoctoproject.org
wrote:
> Add OCI container image recipe for Python to use as a base for
> other Python app containers. The image uses multi-layer mode with
> separate base, terminal and python layers.
> 
> Add ncurses-terminfo-base to a "terminal" layer to avoid warnings in the
> REPL:
>   "Cannot read termcap database;
>   using dumb terminal settings."
> 
> Add coreutils to "python" layer to provide /usr/bin/env needed by
> python3-idle in python3-modules.
> 
> Inherit container-nonroot-user and run a `nonroot` user by default.
> Set PACKAGECONFIG:pn-app-container-python = "dev" in local.conf or
> distro/image config to run as 'root' and include 'pip'.
> 
> Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> ---
>  .../images/app-container-python.bb            | 57 +++++++++++++++++++
>  1 file changed, 57 insertions(+)
>  create mode 100644 recipes-containers/images/app-container-python.bb
> 
> diff --git a/recipes-containers/images/app-container-python.bb b/recipes-containers/images/app-container-python.bb
> new file mode 100644
> index 00000000..a93f1b0f
> --- /dev/null
> +++ b/recipes-containers/images/app-container-python.bb
> @@ -0,0 +1,57 @@
> +SUMMARY = "Base python3 container image"
> +DESCRIPTION = "OCI container image running Python with non-root user. \
> +\
> +In "dev" mode, can optionally run as 'root' and add 'pip' to allow \
> +developers to simply run 'pip install' on top of this container (Not \
> +advised for production/hardened use)."
> +LICENSE = "MIT"
> +LIC_FILES_CHKSUM = "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
> +
> +# Multi-layer mode: create explicit layers instead of single rootfs layer
> +OCI_LAYER_MODE = "multi"
> +
> +# Optional 'dev' mode:
> +#   - adds python3-pip to the python layer (enables `pip install` at runtime)
> +#   - runs the container as root (UID 0) so pip can write to site-packages
> +# Enable with: PACKAGECONFIG:pn-app-container-python = "dev"
> +PACKAGECONFIG ??= ""
> +PACKAGECONFIG[dev] = ""
> +
> +# Define layers: each layer contains specific packages
> +# Format: "name:type:content" where content uses + as delimiter for multiple items
> +OCI_LAYERS = "\
> +    base:packages:base-files+base-passwd+netbase \

Hi Tim,

I wonder if we should define the base layer contents in image-oci to
ensure that it is consistent across recipes.

E.g. in image-oci.bbclass:

    OCI_BASE_LAYER = "base:packages:base-files+base-passwd+netbase"

Then in the recipe:

    OCI_LAYERS = "\
        ${OCI_BASE_LAYER} \
        ...
    "

That gives consistency without forcing all OCI images to use the base
layer definition if it isn't relevant to them.

> +    terminal:packages:ncurses-terminfo-base \
> +    python:packages:python3+coreutils${@bb.utils.contains('PACKAGECONFIG', 'dev', '+python3-pip', '', d)} \
> +"
> +
> +# In 'dev' mode, override the nonroot UID inherited from container-nonroot-user
> +# so the container runs as root (required for `pip install`).
> +OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG', 'dev', '0', '${NONROOT_UID}', d)}"
> +
> +# Use CMD so `docker run image /bin/sh` works as expected
> +OCI_IMAGE_CMD = "python3"
> +
> +IMAGE_FSTYPES = "container oci"
> +inherit image
> +inherit image-oci
> +inherit container-nonroot-user
> +
> +IMAGE_FEATURES = ""
> +IMAGE_LINGUAS = ""
> +NO_RECOMMENDATIONS = "1"
> +
> +# IMAGE_INSTALL triggers package builds via do_rootfs recrdeptask.
> +# Even for multi-layer mode, list packages here to ensure they're built.
> +# The PM will install them directly to layers from DEPLOY_DIR_IPK.
> +# Note: IMAGE_ROOTFS is still created but ignored for packages layers.
> +IMAGE_INSTALL = "base-files base-passwd netbase"

Maybe we also need an OCI_BASE_PACKAGES to go with OCI_BASE_LAYER.
Pretty much every image is going to need these three packages installed.

> +IMAGE_INSTALL += "ncurses-terminfo-base"
> +IMAGE_INSTALL += "python3 coreutils"
> +IMAGE_INSTALL += "${@bb.utils.contains('PACKAGECONFIG', 'dev', 'python3-pip', '', d)}"
> +
> +# Allow build with or without a specific kernel
> +IMAGE_CONTAINER_NO_DUMMY = "1"
> +
> +# Note: No ROOTFS_POSTPROCESS_COMMAND needed - IMAGE_ROOTFS is empty
> +# and PM handles installation directly to OCI layers

Best regards,

-- 
Paul Barker


[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: [meta-virtualization][PATCH 2/7] recipes-containers/images: add app-container-python

[Bruce Ashfield]

[-- Attachment #1: Type: text/plain, Size: 5778 bytes --]

On Tue, Jun 2, 2026 at 6:02 AM Paul Barker via lists.yoctoproject.org <paul=
pbarker.dev@lists.yoctoproject.org> wrote:

> On Fri, 2026-05-29 at 18:31 -0700, Tim Orling via lists.yoctoproject.org
> wrote:
> > Add OCI container image recipe for Python to use as a base for
> > other Python app containers. The image uses multi-layer mode with
> > separate base, terminal and python layers.
> >
> > Add ncurses-terminfo-base to a "terminal" layer to avoid warnings in the
> > REPL:
> >   "Cannot read termcap database;
> >   using dumb terminal settings."
> >
> > Add coreutils to "python" layer to provide /usr/bin/env needed by
> > python3-idle in python3-modules.
> >
> > Inherit container-nonroot-user and run a `nonroot` user by default.
> > Set PACKAGECONFIG:pn-app-container-python = "dev" in local.conf or
> > distro/image config to run as 'root' and include 'pip'.
> >
> > Signed-off-by: Tim Orling <tim.orling@konsulko.com>
> > ---
> >  .../images/app-container-python.bb            | 57 +++++++++++++++++++
> >  1 file changed, 57 insertions(+)
> >  create mode 100644 recipes-containers/images/app-container-python.bb
> >
> > diff --git a/recipes-containers/images/app-container-python.bb
> b/recipes-containers/images/app-container-python.bb
> > new file mode 100644
> > index 00000000..a93f1b0f
> > --- /dev/null
> > +++ b/recipes-containers/images/app-container-python.bb
> > @@ -0,0 +1,57 @@
> > +SUMMARY = "Base python3 container image"
> > +DESCRIPTION = "OCI container image running Python with non-root user. \
> > +\
> > +In "dev" mode, can optionally run as 'root' and add 'pip' to allow \
> > +developers to simply run 'pip install' on top of this container (Not \
> > +advised for production/hardened use)."
> > +LICENSE = "MIT"
> > +LIC_FILES_CHKSUM =
> "file://${COREBASE}/meta/COPYING.MIT;md5=3da9cfbcb788c80a0384361b4de20420"
> > +
> > +# Multi-layer mode: create explicit layers instead of single rootfs
> layer
> > +OCI_LAYER_MODE = "multi"
> > +
> > +# Optional 'dev' mode:
> > +#   - adds python3-pip to the python layer (enables `pip install` at
> runtime)
> > +#   - runs the container as root (UID 0) so pip can write to
> site-packages
> > +# Enable with: PACKAGECONFIG:pn-app-container-python = "dev"
> > +PACKAGECONFIG ??= ""
> > +PACKAGECONFIG[dev] = ""
> > +
> > +# Define layers: each layer contains specific packages
> > +# Format: "name:type:content" where content uses + as delimiter for
> multiple items
> > +OCI_LAYERS = "\
> > +    base:packages:base-files+base-passwd+netbase \
>
> Hi Tim,
>
> I wonder if we should define the base layer contents in image-oci to
> ensure that it is consistent across recipes.
>
> E.g. in image-oci.bbclass:
>
>     OCI_BASE_LAYER = "base:packages:base-files+base-passwd+netbase"
>
> Then in the recipe:
>
>     OCI_LAYERS = "\
>         ${OCI_BASE_LAYER} \
>         ...
>     "
>
> That gives consistency without forcing all OCI images to use the base
> layer definition if it isn't relevant to them.
>

We can't (and shouldn't) enforce using packages at the base like that,
since specifying layers as Tim is using is simply one way of doing it.

It could just as easily be another image, or one of the other techniques.


>
> > +    terminal:packages:ncurses-terminfo-base \
> > +
> python:packages:python3+coreutils${@bb.utils.contains('PACKAGECONFIG',
> 'dev', '+python3-pip', '', d)} \
> > +"
> > +
> > +# In 'dev' mode, override the nonroot UID inherited from
> container-nonroot-user
> > +# so the container runs as root (required for `pip install`).
> > +OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG', 'dev',
> '0', '${NONROOT_UID}', d)}"
> > +
> > +# Use CMD so `docker run image /bin/sh` works as expected
> > +OCI_IMAGE_CMD = "python3"
> > +
> > +IMAGE_FSTYPES = "container oci"
> > +inherit image
> > +inherit image-oci
> > +inherit container-nonroot-user
> > +
> > +IMAGE_FEATURES = ""
> > +IMAGE_LINGUAS = ""
> > +NO_RECOMMENDATIONS = "1"
> > +
> > +# IMAGE_INSTALL triggers package builds via do_rootfs recrdeptask.
> > +# Even for multi-layer mode, list packages here to ensure they're built.
> > +# The PM will install them directly to layers from DEPLOY_DIR_IPK.
> > +# Note: IMAGE_ROOTFS is still created but ignored for packages layers.
> > +IMAGE_INSTALL = "base-files base-passwd netbase"
>
> Maybe we also need an OCI_BASE_PACKAGES to go with OCI_BASE_LAYER.
> Pretty much every image is going to need these three packages installed.
>

Again, we shouldn't be overly prescriptive here.

Factoring things out at this point is premature.

Bruce



>
> > +IMAGE_INSTALL += "ncurses-terminfo-base"
> > +IMAGE_INSTALL += "python3 coreutils"
> > +IMAGE_INSTALL += "${@bb.utils.contains('PACKAGECONFIG', 'dev',
> 'python3-pip', '', d)}"
> > +
> > +# Allow build with or without a specific kernel
> > +IMAGE_CONTAINER_NO_DUMMY = "1"
> > +
> > +# Note: No ROOTFS_POSTPROCESS_COMMAND needed - IMAGE_ROOTFS is empty
> > +# and PM handles installation directly to OCI layers
>
> Best regards,
>
> --
> Paul Barker
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#9842):
> https://lists.yoctoproject.org/g/meta-virtualization/message/9842
> Mute This Topic: https://lists.yoctoproject.org/mt/119557386/1050810
> Group Owner: meta-virtualization+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/unsub [
> bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>

-- 
- Thou shalt not follow the NULL pointer, for chaos and madness await thee
at its end
- "Use the force Harry" - Gandalf, Star Trek II

[-- Attachment #2: Type: text/html, Size: 9182 bytes --]

Re: [meta-virtualization][PATCH 0/7] Container improvements

[Bruce Ashfield]

Just ack'ing the series.

I'm working through issues with the recipe bumps right now, but
will get to this shortly and have a closer look.

Bruce

In message: [meta-virtualization][PATCH 0/7] Container improvements
on 29/05/2026 Tim Orling via lists.yoctoproject.org wrote:

> This series:
>   * Adds a class to create/run containers with a non-root user
>   * Adds new containers:
>     - app-container-python
>     - app-containter-mosquitto
>     - app-container-valkey
>     - app-container-nginx
>   * Modifies app-container-curl to be more like the upstream
>     experience (and more like the above containers)
>   * Allows meta-webserver/recipes-http to be parsed for
>     vcontainer distro so we can build multiarch containers
>     for app-container-nginx, etc.
> 
> Each of these containers was built in a MACHINE=qemuarm64
> environment as well as mc:container-amd64+mc:container-arm64
> multiarch environment.
> 
> The resulting containers were tested with simple command line
> usage compared to Docker provided equivalents to ensure the
> same expected behavior.
> 
> Tim Orling (7):
>   classes: add container-nonroot-user.bbclass
>   recipes-containers/images: add app-container-python
>   recipes-containers/images: add app-container-mosquitto
>   recipes-containers/images: add app-container-valkey
>   recipes-containers/images: add app-container-nginx
>   vcontainer-bbmask.inc: allow meta-webserver/recipes-httpd
>   app-container-curl: use multilayer mode; container-nonroot-user
> 
>  classes/container-nonroot-user.bbclass        | 68 ++++++++++++++++
>  conf/distro/include/vcontainer-bbmask.inc     |  2 +-
>  conf/layer.conf                               |  1 +
>  .../images/app-container-curl.bb              | 29 ++++++-
>  .../images/app-container-mosquitto.bb         | 46 +++++++++++
>  .../images/app-container-nginx.bb             | 77 +++++++++++++++++++
>  .../images/app-container-python.bb            | 57 ++++++++++++++
>  .../images/app-container-valkey.bb            | 61 +++++++++++++++
>  8 files changed, 336 insertions(+), 5 deletions(-)
>  create mode 100644 classes/container-nonroot-user.bbclass
>  rename {recipes-demo => recipes-containers}/images/app-container-curl.bb (58%)
>  create mode 100644 recipes-containers/images/app-container-mosquitto.bb
>  create mode 100644 recipes-containers/images/app-container-nginx.bb
>  create mode 100644 recipes-containers/images/app-container-python.bb
>  create mode 100644 recipes-containers/images/app-container-valkey.bb
> 
> -- 
> 2.54.0
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#9826): https://lists.yoctoproject.org/g/meta-virtualization/message/9826
> Mute This Topic: https://lists.yoctoproject.org/mt/119557384/1050810
> Group Owner: meta-virtualization+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/unsub [bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [meta-virtualization][PATCH 1/7] classes: add container-nonroot-user.bbclass

[Bruce Ashfield]

Hi Tim,

A few things on this one — none of them blocking the intent, but worth a
small follow-up either as a fixup in this series or a v2.

On Fri, May 29, 2026 at 18:31 -0700, Tim Orling wrote:
> For secure and production environments, we want to run containers as a
> non-root user. Some applications, such as Python, require a $HOME
> directory with proper permissions. Because OCI_LAYERS :directories:
> copies with 'cp -a --no-preserve=ownership', we need a fixup function
> to create the proper permissions and ownership in a new raw layer.
>
> The behavior here is inspired by dhi.io/python:3 (Docker Hardened Image)
>
> Signed-off-by: Tim Orling <tim.orling@konsulko.com>

The non-root + DHI-style image is a real gap in what we ship today;
glad to see it filled.

> +inherit extrausers
> +
> +NONROOT_USER ?= "nonroot"
> +NONROOT_UID ?= "65532"
> +NONROOT_GID ?= "65532"
> +
> +# ---------------------------------------------------------------------------
> +# Create the unprivileged "nonroot" user (uid 65532, group 65532)
> +# ---------------------------------------------------------------------------
> +EXTRA_USERS_PARAMS = "\
> +    groupadd -g ${NONROOT_GID} ${NONROOT_USER}; \
> +    useradd -m -u ${NONROOT_UID} -g ${NONROOT_GID} -d /home/${NONROOT_USER}  ${NONROOT_USER}; \
> +"

This hard-assigns EXTRA_USERS_PARAMS rather than appending. Any image
that inherits container-nonroot-user AND wants its own extrausers (e.g.
a recipe that needs an additional service account) loses whatever else
was set, and the order of inheritance silently determines who wins.

Suggest:

  EXTRA_USERS_PARAMS += " \
      groupadd -g ${NONROOT_GID} ${NONROOT_USER}; \
      useradd -m -u ${NONROOT_UID} -g ${NONROOT_GID} \
              -d /home/${NONROOT_USER} ${NONROOT_USER}; \
  "

That way the class composes cleanly with anyone else who sets
EXTRA_USERS_PARAMS earlier.

> +# Make sure we can write to e.g. /home/nonroot/.python_history
> +# using :directories: in OCI_LAYERS does not preserve permissions
> +fakeroot fix_oci_home_perms() {
> +    cd ${IMGDEPLOYDIR}
> +    image_name="${IMAGE_NAME}${IMAGE_NAME_SUFFIX}-oci"
> +    layer_tar="${WORKDIR}/oci-home-fix-layer.tar"
> +
> +    rm -f "$layer_tar"
> +
> +    python3 - "$layer_tar" <<'PYEOF'
> +import sys, tarfile, time
> +
> +layer_tar = sys.argv[1]
> +mtime = int(time.time())
> +
> +# (path, mode, uid, gid)
> +entries = [
> +    ("home",         0o755, 0,     0),
> +    ("home/${NONROOT_USER}", 0o700, ${NONROOT_UID}, ${NONROOT_GID}),
> +]

Two things about the Python heredoc:

1. The 'PYEOF' delimiter uses single quotes, which would normally suppress
   shell expansion inside the heredoc — but bitbake expands ${NONROOT_USER}
   etc. at parse time *before* the shell ever sees the body, so the
   expansion does happen, just via a different path than the reader expects.
   Worth a one-line comment so the next person doesn't try to "fix" it.

2. While the values are simple here (alphanumeric user, integer uid/gid)
   it's fine, but if someone ever sets NONROOT_USER to something with a
   quote or backslash, the Python source is no longer well-formed. Since
   this is a meta-virt-internal class, low risk — but a comment
   ("NONROOT_USER must be a bare identifier") would protect it.

> +    umoci raw add-layer --image "$image_name:${OCI_IMAGE_TAG}" "$layer_tar"
> +    rm -f "$layer_tar"
> +
> +    rm -f "$image_name.tar" "$image_name-dir.tar"
> +    ( cd "$image_name" && tar -cf "../$image_name.tar" "." )
> +    tar -cf "$image_name-dir.tar" "$image_name"
> +}
> +do_image_oci[postfuncs] += "fix_oci_home_perms"

The repackaging step at the bottom assumes do_image_oci's output layout
(image_name.tar / image_name-dir.tar) is stable. If image-oci.bbclass
ever changes its packaging naming, this silently produces a broken
tarball — there's no error checking against the assumption.

A defensive option: stat the expected files before rm -f / tar -cf, and
bbwarn if either is missing rather than recreating from a possibly-empty
directory. Or, if there's a helper in image-oci.bbclass that already
does the repackaging, call that instead of rebuilding by hand.

Otherwise the series looks good — I'll keep going through 2/7..7/7 and
respond per-patch as I have specific notes.

Bruce

Re: [meta-virtualization][PATCH 2/7] recipes-containers/images: add app-container-python

[Bruce Ashfield]

Hi Tim,

Just a couple of comments ...

On Fri, May 29, 2026 at 18:31 -0700, Tim Orling wrote:
> Add OCI container image recipe for Python to use as a base for
> other Python app containers. The image uses multi-layer mode with
> separate base, terminal and python layers.
[...]
> +OCI_LAYERS = "\
> +    base:packages:base-files+base-passwd+netbase \
> +    terminal:packages:ncurses-terminfo-base \
> +    python:packages:python3+coreutils${@bb.utils.contains('PACKAGECONFIG', 'dev', '+python3-pip', '', d)} \
> +"

I like this conditional. Putting the bb.utils.contains() directly inside
the layer's package list (instead of duplicating the OCI_LAYERS
declaration in two PACKAGECONFIG branches).

Worth calling out in our multi-layer mode docs as the recommended way
to do conditional packages-per-layer. I'll do a patch for that.

> +# IMAGE_INSTALL triggers package builds via do_rootfs recrdeptask.
> +# Even for multi-layer mode, list packages here to ensure they're built.
> +# The PM will install them directly to layers from DEPLOY_DIR_IPK.
> +# Note: IMAGE_ROOTFS is still created but ignored for packages layers.
> +IMAGE_INSTALL = "base-files base-passwd netbase"
> +IMAGE_INSTALL += "ncurses-terminfo-base"
> +IMAGE_INSTALL += "python3 coreutils"
> +IMAGE_INSTALL += "${@bb.utils.contains('PACKAGECONFIG', 'dev', 'python3-pip', '', d)}"

This is correct today but it doubles the source of truth. The packages
listed in OCI_LAYERS:packages: and the packages listed in IMAGE_INSTALL
have to be kept in sync.

If they drift (change OCI_LAYERS but forget IMAGE_INSTALL or vice
versa), the build silently breaks at layer assembly time when the
missing package isn't in DEPLOY_DIR_IPK.

Two paths I'd consider, either as part of this series or as a follow-up:

  a) Add a "# KEEP IN SYNC WITH OCI_LAYERS" comment so the next
     maintainer knows.

  b) Better: derive IMAGE_INSTALL from OCI_LAYERS:packages: layers
     inside image-oci.bbclass, so the recipe only sets it once. That
     fixes the problem for every multi-layer recipe, not just python.

I've staged b) on master-next (shortly), if you can do a) .. or maybe
it isn't needed at all now.

We can do the same for all the other similar recipes in the series.

One more, also non-blocking — the other 4 new image recipes in this
series (mosquitto, valkey, nginx, curl) all carry the
rootfs_fixup_var_volatile postprocess to create /var/volatile/{tmp,log}.

This one doesn't. Intentional because python doesn't touch /var/log? Or
worth adding for consistency in case someone runs as 'dev' and pip
wants a writable spot?

Bruce

Re: [meta-virtualization][PATCH 3/7] recipes-containers/images: add app-container-mosquitto

[Bruce Ashfield]

Hi Tim,

A few comments — most are series-level patterns that show up here for
the first time. I won't repeat them later to not waste our time

On Fri, May 29, 2026 at 18:31 -0700, Tim Orling wrote:
> Add OCI container image recipe for the Eclipse Mosquitto MQTT broker.
> The image uses multi-layer mode with separate base and mosquitto layers,
> exposes standard MQTT (1883) and WebSocket (9001) ports, and launches
> mosquitto with its default config file as the entrypoint.
>
> Inherit container-nonroot-user to run as 'nonroot' with UID 65532.

> +OCI_LAYERS = "\
> +    base:packages:base-files+base-passwd+netbase \
> +    mosquitto:packages:mosquitto \
> +"
[...]
> +IMAGE_INSTALL = " \
> +    base-files \
> +    base-passwd \
> +    netbase \
> +    mosquitto \
> +"

Same point as 2/7 — image-oci.bbclass now folds packages: layers into
IMAGE_INSTALL automatically (I pushed to master-next). When you respin,
this block can go.

> +# Workaround /var/volatile for now
> +ROOTFS_POSTPROCESS_COMMAND += "rootfs_fixup_var_volatile ; "
> +rootfs_fixup_var_volatile () {
> +    install -m 1777 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/tmp
> +    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/volatile/log
> +}

This same function appears in 3/7 (mosquitto), 4/7 (valkey), 5/7
(nginx), and 7/7 (curl) — four near-identical copies. Worth factoring
into a small helper bbclass (e.g. container-volatile-fixup.bbclass) that
each recipe can inherit, or rolling it into container-nonroot-user.bbclass
since every recipe that uses the fixup also inherits the nonroot class.
Not blocking — but it's a smell that's easy to silence in v2.

> +OCI_IMAGE_ENTRYPOINT = "${sbindir}/mosquitto"
> +OCI_IMAGE_ENTRYPOINT_ARGS = "-c '${sysconfdir}/mosquitto/mosquitto.conf'"

Two questions about running mosquitto as our nonroot (uid 65532):

1. The mosquitto package usually creates a 'mosquitto' system user
   (low uid, varies by build) and ships a default config that
   references paths under /var/lib/mosquitto/ and /var/log/mosquitto/
   owned by that user. As nonroot we won't be the package's expected
   user. Does mosquitto -c on the stock config actually start cleanly
   for you, or did you need to tweak the conf?

   I assume it runs fie, since you've been testing it for a while

2. If persistence is enabled in the stock mosquitto.conf (persistence
   true; persistence_location /var/lib/mosquitto/), we need a
   writable /var/lib/mosquitto for our nonroot user — same flavour of
   gap the rootfs_fixup_var_volatile workaround addresses, just for
   a different path. If you've already validated that persistence is
   off in the stock conf (or that mosquitto degrades gracefully when
   the persistence dir isn't writable), a one-line comment in the
   recipe explaining the trade-off would save the next person from
   re-deriving it.

Bruce

> +OCI_IMAGE_PORTS = "1883/tcp 9001/tcp"
> +OCI_IMAGE_TAG = "latest"

Re: [meta-virtualization][PATCH 4/7] recipes-containers/images: add app-container-valkey

[Bruce Ashfield]

Hi Tim,

Two valkey-specific comments — the series-level points from 2/7 and 3/7
apply here too but I won't re-raise them.

On Fri, May 29, 2026 at 18:31 -0700, Tim Orling wrote:
> Add OCI container image recipe for the Valkey in-memory key-value
> datastore. The image uses multi-layer mode with separate base and
> valkey layers, exposes the standard Valkey port (6379), and launches
> valkey-server with its default config file as the entrypoint.
[...]
> +OCI_IMAGE_ENTRYPOINT = "${bindir}/valkey-server"
> +# The stock valkey.conf shipped by meta-oe is tuned for a host install
> +# (daemonize yes, syslog-enabled yes, bind 127.0.0.1). Override those at
> +# launch so the server stays in the foreground as PID 1, logs to stdout,
> +# and is reachable from outside the container.
> +OCI_IMAGE_ENTRYPOINT_ARGS = "'${sysconfdir}/valkey/valkey.conf' \
> +    --daemonize no \
> +    --syslog-enabled no \
> +    --bind '0.0.0.0 -::*' \
> +    --protected-mode no"

The override-the-conf-at-launch trick is nice, I like it, and it's
documented in the comments.

What about a description of `--protected-mode no` ?

I quickly searched and found with protected-mode off and bind on all
interfaces, anyone who can reach the container gets unauthenticated
access — that's the right default for a base image people will pull
and customise, but a deployer who slaps this into production without
adding AUTH or namespace isolation gets a surprise. Worth a single
sentence in DESCRIPTION saying so explicitly:

Maybe this ?

  DESCRIPTION = "OCI container running the Valkey in-memory key-value \
  datastore, a flexible distributed datastore that supports both caching \
  and beyond caching workloads. This image runs with protected-mode \
  disabled and bound to all interfaces, intended as a base for \
  customisation — production deployments should enable requirepass / \
  ACLs or restrict the network namespace."

The second is the same persistence question I asked on mosquitto. The
stock meta-oe valkey.conf may or may not enable AOF / RDB snapshotting
(`appendonly yes` / `save <seconds> <writes>`). If either is on,
valkey-server tries to write to its `dir` (usually /var/lib/valkey).
Running as our nonroot uid 65532 against a /var/lib/valkey owned by the
valkey package user will fail-to-write or worse, silently lose data.

Did you confirm during testing whether the stock conf has persistence
on? If on, we want a /var/lib/valkey fixup similar to the
/var/volatile one. If off, a one-line comment saying so would save the
next person from re-deriving it.

Bruce

Re: [meta-virtualization][PATCH 5/7] recipes-containers/images: add app-container-nginx

[Bruce Ashfield]

Hi Tim,

Most complex recipe in the series, and I like the the runtime-dir
handling

A few nginx-specific things, that came up when I was searching up
the runtime parts.

On Fri, May 29, 2026 at 18:31 -0700, Tim Orling wrote:
> Add OCI container image recipe for the NGINX web server. The image
> uses multi-layer mode with separate base, nginx packages, nginx
> runtime directories, and nginx log file layers.
[...]
> +NONROOT_USER = "nginx"

The biggest open question here. The nginx recipe in meta-webserver
creates its own 'nginx' user (via useradd in inherit useradd or a
postinst), with a uid the package picks. Now we have two paths trying
to create user 'nginx':

  a) container-nonroot-user.bbclass adds it via extrausers as uid
     65532 / gid 65532 (whatever NONROOT_UID/GID are set to).

  b) The nginx package's own user-creation step adds it with the
     package's chosen uid.

Last one to run wins in /etc/passwd, but RPM/dpkg may also refuse a
duplicate-uid useradd outright. Did you see any "user already exists"
or uid-mismatch noise in do_rootfs? If not, the order happens to be
fine in your build but it's fragile.

Are these options for v2 ?

  1) Override NONROOT_UID to whatever the nginx package picks for the
     nginx user. Looks up the package's uid, set NONROOT_UID to match.

  2) Use NONROOT_USER = "nginxapp" or similar — a name that doesn't
     collide with the package's own — and tell nginx to run as that
     user via the conf file (`user nginxapp;`) instead of relying on
     the implicit uid match.

Not blocking but worth a note in the recipe about which approach you
chose and why, so we won't get cut and paste propagation without
a reason.

> +OCI_LAYERS = "\
> +    base:packages:base-files+base-passwd+netbase \
> +    nginx:packages:nginx \
> +    nginx-dirs:directories:${localstatedir}/log/nginx+/run/nginx+${localstatedir}/volatile/tmp+${localstatedir}/volatile/log \
> +    nginx-files:files:${localstatedir}/log/nginx/access.log+${localstatedir}/log/nginx/error.log \
> +"

Nice mix of packages + directories + files layer types in a single
recipe — this is exactly the scenario the multi-layer support was
built for, glad to see it landing in a real recipe.

> +    # nginx opens the compiled-in error_log path before reading -c config.
> +    # /var/log is typically a symlink to /var/volatile/log in a Yocto rootfs,
> +    # so create the target path explicitly to guarantee the directory lands in
> +    # the container layer regardless of symlink resolution order.
> +    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/log
> +    install -m 755 -d ${IMAGE_ROOTFS}/${localstatedir}/log/nginx
> +
> +    # nginx's compiled-in temp paths (client_body_temp, proxy_temp, etc.) all
> +    # live under /run/nginx, which is not created by any package.
> +    install -m 755 -d ${IMAGE_ROOTFS}/run/nginx

The /run/nginx catch is great — that's the kind of thing that bites
you at first-request time and you can't reproduce without the right
URL hitting the right module. Good comment too.

> +OCI_IMAGE_APP_RECIPE = "nginx"

Glad to see this in use. It's been sitting in image-oci.bbclass as
documentation-only / hook-point, and this is the first recipe that
actually sets it. Once we wire it up to auto-extract SRCREV/branch
for OCI labels (the "future versions may auto-extract" hook comment
in the bbclass), recipes that set it correctly will get the
provenance labels for free.

Bruce

Re: [meta-virtualization][PATCH 6/7] vcontainer-bbmask.inc: allow meta-webserver/recipes-httpd

[Bruce Ashfield]

Hi Tim,

Small focused change, intent is clear. Two questions, one cleanup.

On Fri, May 29, 2026 at 18:31 -0700, Tim Orling wrote:
> Allow us to build nginx, apache2, etc. multiarch containers.
[...]
> +BBMASK += "meta-webserver/recipes-(?!httpd)"
[...]
>  BBMASK += "meta-filesystems/"
>  BBMASK += "meta-python/"
> -BBMASK += "meta-webserver/"

The mask is right and the regex correctly leaves recipes-httpd
parseable while keeping everything else under meta-webserver out.

A couple of things worth a follow-up:

  1. recipes-httpd contains both nginx and apache2 (and a couple of
     smaller recipes — hiawatha, lighttpd, monkey iirc). With this
     change apache2 also becomes parseable for the vcontainer distro,
     even though only nginx is used by 5/7. Is the apache2 inclusion
     intentional (to enable an app-container-apache2 follow-up later),
     or accidental?

     If accidental, tightening the regex to recipes-httpd/nginx- might
     be safer — fewer recipes pulled into parse keeps the dep graph
     smaller and avoids surprise cascades from apache2's deps.

  2. The companion entry in meta-virt-host.conf:

         BBFILE_PATTERN_IGNORE_EMPTY_<meta-webserver> = "1"
         (or similar variable name — the exact form is what
         meta-virt-host.conf uses for the other fully-masked layers)

     ...was added because meta-webserver used to be a fully-masked
     layer producing the "no recipes in this layer" warning. With this
     change, meta-webserver does parse at least one recipe (nginx),
     so the IGNORE_EMPTY entry is no longer needed and should be
     removed in this same commit. Otherwise it sits as dead config.

  3. Did you run `bitbake -g <container-image>` after this change to
     confirm the dep graph didn't pick up anything heavy you didn't
     expect? recipes-httpd's deps tend to be fairly contained, but
     it's worth a one-time check so we know the BBMASK fence is doing
     what we think it is.

The change itself is fine to apply as-is; (2) is the only thing I'd
ask be folded in before merge.

Bruce

Re: [meta-virtualization][PATCH 7/7] app-container-curl: use multilayer mode; container-nonroot-user

[Bruce Ashfield]

Hi Tim,

Wrap-up patch, mostly aligns curl with the new conventions. One real
question, one observation.

On Fri, May 29, 2026 at 18:31 -0700, Tim Orling wrote:
> * Move to recipes-containers/images to show maintenance intent
> * Switch to multilayer mode to more like the other "library"/"official"
>   container recipes.
> * Change OCI_IMAGE_TAG to "latest" for similar reasons.
> * Change OCI_IMAGE_ENTRYPOINT_ARGS to "--help" to be more like upstream
>   containers.

The directory move right. no issues. recipes-demo was always
"here's an example", recipes-containers/images is "this is supported".
Same for "easy" → "latest"; matches Docker convention.

The "--help" default is a small but a win ... running the image with
no args now self-documents instead of trying to hit a probably-absent
localhost:80.

> +OCI_LAYERS = "\
> +    base:packages:base-files+base-passwd+netbase \
> +    ${@bb.utils.contains('PACKAGECONFIG', 'dev', 'shell:packages:${CONTAINER_SHELL}', '', d)} \
> +    curl:packages:curl+ca-certificates \
> +"
[...]
> -CONTAINER_SHELL = "busybox"

This worries me. The old recipe had CONTAINER_SHELL = "busybox" at
the top level, so the variable always had a value. The new recipe
removes that line and only references ${CONTAINER_SHELL} inside the
conditional shell layer.

In non-dev mode that's fine because the whole bb.utils.contains
expression resolves to the empty string and the layer doesn't appear.

In dev mode the OCI_LAYERS line expands to:

    shell:packages:

…because nothing in the recipe (or in container-nonroot-user.bbclass)
sets CONTAINER_SHELL. image-oci.bbclass's validator parses this as a
three-field entry with empty content, so it doesn't bb.fatal — but no
packages get added to the auto-derived IMAGE_INSTALL, and the
resulting layer is empty. The container is silently shipped without
a shell, which is exactly the thing dev mode is supposed to provide.

Two ways to fix:

  a) Add a default in the recipe near the PACKAGECONFIG declaration:

         CONTAINER_SHELL ??= "busybox"

     Then anyone enabling dev gets busybox by default and the variable
     remains overridable.

  b) Push the default into container-nonroot-user.bbclass so every
     recipe that inherits it gets a consistent shell value.

I'd lean (a) — the class shouldn't carry a shell concept; that's a
per-recipe choice — but either works.

> +OCI_IMAGE_RUNTIME_UID = "${@bb.utils.contains('PACKAGECONFIG', 'dev', '0', '${NONROOT_UID}', d)}"

This dev-mode-overrides-uid pattern also appears in 2/7 (python). It's
clean here, and identical there — another candidate for the helper
class refactor whenever the rootfs_fixup_var_volatile factor-out
happens, since it's the same shape of "PACKAGECONFIG dev means root".

> +IMAGE_INSTALL:append = " curl ca-certificates"

Same series-level point as the earlier recipes — image-oci.bbclass
now auto-derives this from OCI_LAYERS in v2 / master-next, so the
line can go.

Bruce