[meta-virtualization][PATCH 0/2] image-oci: two do_image_oci bugfixes (missing pm.update(), libgcc/libgcc-initial sysroot collision)

[meta-virtualization][PATCH 0/2] image-oci: two do_image_oci bugfixes (missing pm.update(), libgcc/libgcc-initial sysroot collision)

[Koen Kooi]

I've been working on a prototype for building a rootfs with multiple multi-layer OCI images images, all built from scratch, in one bitbake command. I ran into some issues along the way:

* image-oci never runs "opkg update" or its rpm equivalents, so nothing gets installed
* both libgcc-initial and libgcc want to be in the sysroot

The first issue is straightforward, for the second I leaned heavily on Claude to write a proper python version and explain it better. I'm not fond of the wall of text that ends up with, but this is already the 3rd iteration of trimming it down. 

Anyway, This is the first wave, I'm working on other fixes, like mtime influencing base-layer sharing, but that needs more testing and review. I do need to mention how surprisingly small the resulting image recipes are, a big thanks to everyone who has been working on this!

Below is what Claude suggested as coverletter, it is still verbose, but doesn't suffer from the "Is Koen being sarcastic or not?" issue :)

1/2 - image-oci: call pm.update() before pm.install() for multi-layer
      package layers

  Multi-layer OCI images (OCI_LAYER_MODE = "multi") with a "packages:"
  layer silently produce an empty layer: do_image_oci reports success,
  but the layer's opkg/rpm install never actually installs anything,
  because the per-layer package manager's local index/cache was never
  refreshed after write_index() (do_rootfs itself always calls
  pm.update() between the two; image-oci's per-layer installer
  skipped it). The failure is only visible as a bb.warn(), so the
  build looks green while shipping an empty (or partial) layer.

  Reproduced with a minimal recipe (base-files+coreutils in a
  "packages:" layer): before the fix, the resulting OCI layer blob
  contains 0 regular files; after the fix, the same layer contains
  272 files and the compressed blob grows from 299 bytes to 7.5 MB.
  Full before/after evidence in the patch commit message and in
  TEST-CASES.md if you'd like to reproduce it independently.

2/2 - image-oci: drop *-initial sysroot deps in do_image_oci to avoid
      libgcc collision

  do_image_oci can abort with:
    ERROR: <recipe> do_image_oci: The file .../crtbegin.o is installed
    by both libgcc and libgcc-initial, aborting

  Root cause is in extend_recipe_sysroot's dependency walk: the
  SSTATE_EXCLUDEDEPS_SYSROOT exclusion that normally keeps libgcc and
  libgcc-initial from landing in the same sysroot only applies when
  the *consuming* task is itself do_populate_sysroot. do_image_oci
  reaches extend_recipe_sysroot through the generic "any task whose
  [depends] contains populate_sysroot gets this prefunc" path
  (staging_taskhandler), so that exclusion never fires for it.

  This one is sstate-state-dependent -- whether it fires depends on
  whether extend_recipe_sysroot's walk actually needs to populate (vs.
  skip via sstate/setscene shortcuts) both libgcc and libgcc-initial
  for a given recipe/build. A minimal single-package OCI recipe did
  not reach libgcc in our testing, but a multi-layer "packages:" recipe
  installing a from-source-compiled library with a moderately large
  DEPENDS (boost) did: instrumenting the fix confirms it stripping two
  "*-initial" do_populate_sysroot nodes out of BB_TASKDEPDATA for that
  recipe, i.e. libgcc-initial is a real, reachable node in
  do_image_oci's dependency walk for realistic recipes. We were not
  able to force the raw abort deterministically within our test
  window (a downstream build did hit it deterministically across 6+
  clean-sstate attempts on a wider dependency set) -- see the patch
  commit message and TEST-CASES.md for exactly what was and wasn't
  reproduced, and why the fix is justified regardless.


Signed-off-by: Koen Kooi <koen.kooi@oss.qualcomm.com>

[PATCH 1/2] image-oci: call pm.update() before pm.install() in multi-layer package layers

[Koen Kooi]

For multi-layer packages, the class doesn't call pm.update() like the
OE-core image classes, making the package managers lack package lists.
And without lists, you can't install packages.

This is being hidden by the class treating opkg errors with bb.warn(),
leading to every tool shouting "Success!!!", but delivering empty
layers.

Applies to both opkg and rpm paths.

Reproducible with a minimal recipe:

  inherit image image-oci
  IMAGE_INSTALL = "base-files"
  OCI_LAYER_MODE = "multi"
  OCI_LAYERS = "base:packages:base-files+coreutils"

do_image_oci's log shows the real failure, demoted to a warning:

  WARNING: ... do_image_oci: Unable to install packages. Command
  '[...opkg... install base-files coreutils]' returned 255:
  error: opkg_prepare_url_for_install: Couldn't find anything to
  satisfy 'base-files'.

Signed-off-by: Koen Kooi <koen.kooi@oss.qualcomm.com>
---
 classes/image-oci.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/classes/image-oci.bbclass b/classes/image-oci.bbclass
index 9914411b..ed8926f6 100644
--- a/classes/image-oci.bbclass
+++ b/classes/image-oci.bbclass
@@ -484,6 +484,7 @@ def oci_install_layer_packages(d, layer_rootfs, layer_packages, layer_name):

         # Generate/update repo indexes
         pm.write_index()
+        pm.update()

         # Install packages
         # Use attempt_only=True to allow unresolved deps (resolved in later layers)
@@ -509,6 +510,7 @@ def oci_install_layer_packages(d, layer_rootfs, layer_packages, layer_name):

         # Write indexes
         pm.write_index()
+        pm.update()

         # Install packages
         try:
--
2.34.1

Re: [meta-virtualization] [PATCH 1/2] image-oci: call pm.update() before pm.install() in multi-layer package layers

[Bruce Ashfield]

Taken to master-next with two things worth flagging.

1) I reworded the commit message to capture historical context that
   only makes sense from the maintainer side: our own multi-layer
   test recipes use OCI_LAYER_MODE=multi with packages: layers and
   they build fine, so it wasn't obvious at first read why this bug
   didn't already burn us. Root cause turned out to be that
   image-oci.bbclass has three separate call sites resolving
   IMAGE_PKGTYPE with hardcoded defaults that disagree with each
   other (rpm/ipk/rpm), so our unpinned builds silently take the
   rpm path -- and dnf install refreshes repodata on demand, so the
   missing pm.update() was a no-op for us. Reproducing your ipk
   trace required someone to actually be using opkg, which we
   weren't. The reworded body walks through all of that so the next
   reader doesn't have to. Your Signed-off-by is preserved.

2) Two follow-up commits on top:

   * "image-oci: unify IMAGE_PKGTYPE default via oci_pkg_type()
     helper" -- collapses the three disagreeing defaults into one
     shared helper, defaulting to ipk to match oe-core's
     PACKAGE_CLASSES default. Nothing changes for anyone who was
     already pinning IMAGE_PKGTYPE.

   * "tests: assert multi-layer OCI blobs have real content, not
     just count" -- our existing multilayer test only checked
     len(layers) == 3, which happily passes on a fully-empty build.
     Adds a per-layer blob-size floor so a silent pm.install()
     failure fails the test the way it should have all along.

Thanks for finding this one. The 2/2 libgcc/libgcc-initial patch
I'm still working through; I'd like to reproduce it locally and
see whether the fix can be simpler than a full BB_TASKDEPDATA
mutation. I'll follow up on that thread separately.

Bruce

[PATCH 2/2] image-oci: drop *-initial sysroot deps in do_image_oci to avoid libgcc collision

[Koen Kooi]

do_image_oci can abort with:

  ERROR: <recipe> do_image_oci: The file .../crtbegin.o is installed
  by both libgcc and libgcc-initial, aborting

This was seen deterministically (6+ clean-sstate attempts) on an
OCI_LAYER_MODE = "multi" image recipe whose "packages:" content is a
from-source-compiled library with a moderately large DEPENDS. It is
sstate-state-dependent: it only fires once do_image_oci's dependency
walk actually has to populate (rather than skip via sstate/setscene
shortcuts) both libgcc and libgcc-initial into the same recipe
sysroot, so a given recipe/build combination can build clean one run
and hit the collision on the next, depending on what is already
present in sstate. Confirmed in our own testing: instrumenting the
fix below shows it stripping two "*-initial" do_populate_sysroot
nodes out of BB_TASKDEPDATA for a boost-based multi-layer "packages:"
recipe -- i.e. libgcc-initial is a real, reachable node in
do_image_oci's dependency walk for realistic recipes, even though a
minimal single-package recipe does not reach it.

Root cause: do_image_oci[depends] contains several
"...:do_populate_sysroot" entries (umoci-native, jq-native, and for
multi-layer package installs, opkg-native/rsync-native/dnf-native).
Any task whose [depends] flag contains the substring
"populate_sysroot" gets oe-core's extend_recipe_sysroot
(staging.bbclass) auto-attached as a prefunc, which walks the full
transitive do_populate_sysroot closure reachable from those deps and
copies it into the recipe's private sysroot.

oe-core already protects against exposing both libgcc and
libgcc-initial into the same sysroot: setscene_depvalid() excludes
"*-initial" recipes via SSTATE_EXCLUDEDEPS_SYSROOT
(".*->.*-initial.*", meta/conf/layer.conf, "nothing needs to depend
on libc-initial"), and staging_populate_sysroot_dir has its own
"skip libgcc-initial due to file overlap" special case. But the
SSTATE_EXCLUDEDEPS_SYSROOT check inside setscene_depvalid only
applies when the *dependent* task is itself do_populate_sysroot
("taskdependees[task][1] == 'do_populate_sysroot'"). For
do_image_oci, the dependent task is do_image_oci, not
do_populate_sysroot, so that exclusion is skipped entirely and both
libgcc and libgcc-initial fall through to the default "keep it"
path, landing in the same sysroot and colliding on their shared
crtbegin.o.

Fix: add a RecipeTaskPreProcess handler that, for do_image_oci
specifically, prepends a prefunc which strips "*-initial"
do_populate_sysroot nodes out of BB_TASKDEPDATA before
extend_recipe_sysroot runs (extend_recipe_sysroot re-reads
BB_TASKDEPDATA fresh each time, so this is sufficient). This mirrors
the existing "skip *-initial" intent from
staging_populate_sysroot_dir, scoped narrowly to do_image_oci so it
doesn't change behavior for any other task. The *-initial recipes
are bootstrap-only (used to build glibc) and are fully superseded by
their real counterparts by image-assembly time, so nothing an image
needs is lost by excluding them here.

Signed-off-by: Koen Kooi <koen.kooi@oss.qualcomm.com>
---
 classes/image-oci.bbclass | 77 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 77 insertions(+)

diff --git a/classes/image-oci.bbclass b/classes/image-oci.bbclass
index ed8926f6..c397e2f9 100644
--- a/classes/image-oci.bbclass
+++ b/classes/image-oci.bbclass
@@ -526,6 +526,83 @@ def oci_install_layer_packages(d, layer_rootfs, layer_packages, layer_name):

     bb.note(f"OCI: Package installation complete for layer '{layer_name}'")

+# =============================================================================
+# Work around libgcc vs libgcc-initial sysroot collision in do_image_oci
+# =============================================================================
+#
+# do_image_oci[depends] contains "...:do_populate_sysroot" entries (for the
+# native tools it needs: umoci/jq/opkg/rsync-native). oe-core's
+# staging_taskhandler therefore auto-attaches extend_recipe_sysroot as a
+# prefunc of do_image_oci (any task whose [depends] contains the substring
+# "populate_sysroot" gets it). extend_recipe_sysroot then walks the whole
+# reachable populate_sysroot dependency tree and exposes it into this recipe's
+# private sysroot.
+#
+# For a *do_populate_sysroot* consumer, oe-core's setscene_depvalid() excludes
+# *-initial recipes (SSTATE_EXCLUDEDEPS_SYSROOT ".*->.*-initial.*"), so a normal
+# recipe never sees both libgcc and libgcc-initial. But that exclusion is gated
+# behind "taskdependees[task][1] == 'do_populate_sysroot'", which is false here
+# (the consuming task is do_image_oci), so both libgcc and libgcc-initial get
+# exposed and their shared crtbegin.o collides:
+#   ERROR: ... The file .../crtbegin.o is installed by both libgcc and
+#          libgcc-initial, aborting
+#
+# oe-core already skips *-initial on its other sysroot-population codepaths
+# (staging_populate_sysroot_dir: "skip libgcc-initial due to file overlap", and
+# the do_sdk_depends/do_populate_sdk_ext skip inside extend_recipe_sysroot
+# itself). We apply the same intent here, narrowly and only for image-oci
+# recipes: before extend_recipe_sysroot runs, drop *-initial do_populate_sysroot
+# nodes from this task's copy of BB_TASKDEPDATA so they never enter the tree.
+# extend_recipe_sysroot re-reads BB_TASKDEPDATA fresh, so this is sufficient and
+# touches nothing else. The *-initial recipes are bootstrap-only (used to build
+# glibc) and are fully superseded by their real counterparts by image-assembly
+# time, so nothing an image needs is lost.
+python oci_strip_initial_from_taskdepdata () {
+    import copy
+
+    taskdepdata = d.getVar("BB_TASKDEPDATA", False)
+    if not taskdepdata:
+        return
+
+    filtered = copy.deepcopy(taskdepdata)
+    drop = set()
+    for k, v in filtered.items():
+        # v[0] = PN, v[1] = taskname
+        if v[1] == "do_populate_sysroot" and v[0].endswith("-initial"):
+            drop.add(k)
+
+    if not drop:
+        return
+
+    for k in drop:
+        del filtered[k]
+    # Remove inbound edges to the dropped nodes. Each entry is a bb.TaskData
+    # namedtuple (immutable), but its dep set (index 3) is a mutable set, so
+    # mutate it in place rather than reassigning the field.
+    for v in filtered.values():
+        v[3].difference_update(drop)
+
+    bb.note("OCI: dropped %d *-initial do_populate_sysroot node(s) from "
+            "BB_TASKDEPDATA to avoid libgcc/libgcc-initial sysroot collision"
+            % len(drop))
+    d.setVar("BB_TASKDEPDATA", filtered)
+}
+oci_strip_initial_from_taskdepdata[vardepsexclude] += "BB_TASKDEPDATA"
+
+# Ensure our prefunc runs *before* extend_recipe_sysroot. staging_taskhandler
+# (a global class) prepends extend_recipe_sysroot on the RecipeTaskPreProcess
+# event; this handler is registered later (image-oci is inherited after the
+# global classes) so its prepend lands first in the prefunc list.
+python oci_taskhandler() {
+    task = "do_image_oci"
+    if task in e.tasklist:
+        deps = d.getVarFlag(task, "depends")
+        if deps and "populate_sysroot" in deps:
+            d.prependVarFlag(task, "prefuncs", "oci_strip_initial_from_taskdepdata ")
+}
+oci_taskhandler[eventmask] = "bb.event.RecipeTaskPreProcess"
+addhandler oci_taskhandler
+
 # the IMAGE_CMD:oci comes from the .inc
 OCI_IMAGE_BACKEND_INC ?= "${@"image-oci-" + "${OCI_IMAGE_BACKEND}" + ".inc"}"
 include ${OCI_IMAGE_BACKEND_INC}
--
2.34.1

Re: [meta-virtualization] [PATCH 2/2] image-oci: drop *-initial sysroot deps in do_image_oci to avoid libgcc collision

[Bruce Ashfield]

Reviewed but holding this one for a bit. Two things:

First, on the fix mechanism: the analysis in the commit message is
convincing and matches what oe-core already does for do_sdk_depends
and do_populate_sdk_ext at staging.bbclass:457 (same "*-initial" skip,
same reason).

If that list of tasks was extendible, we could add do_image_oci to it,
but Richard wouldn't think it was a good idea if I tried to bury a
meta-virt task into that list :)

BitBake's python-function :prepend/:append doesn't help either -- the
mid-loop condition operates on setscenedeps built inside the function,
not on anything we could pre-filter via a prepend body. Prefunc +
BB_TASKDEPDATA mutation looks like the only path that won't need to
be updated as bitbake/oe-core change.

I'm testing a smaller-footprint variant of the same approach -- same
idea, same event-handler attachment, roughly 15 lines without the
deep-copy (setVar with a fresh dict is enough since
extend_recipe_sysroot re-reads BB_TASKDEPDATA on each entry).
Body/mechanism identical to yours; the trim is stylistic.

Second, and this is what's actually blocking: I haven't managed
to reproduce the raw abort locally. From your cover letter, it
doesn't sound like you are there yet either ? Without a repro that
fails on stock and passes with the fix, all we've got is "the code
path runs and does what the comment says it does" -- which is true
but doesn't prove the payoff. Eventually that might be enough, but
I'd like to try a bit more.

Two questions to unblock:

1) Can you share the specific recipe (or the DEPENDS shape) from
   the downstream build where you hit this deterministically?
   Something along the lines of a multilayer packages: layer
   pulling a from-source-compiled library with a boost-ish
   DEPENDS, if I'm reading your cover right, but I'd rather work
   from your actual trigger than guess.

2) Was there anything else in that build environment worth
   knowing -- IMAGE_PKGTYPE pin, non-default MACHINE, a specific
   PACKAGECONFIG that pulled in one of the *-initial-adjacent
   recipes? Anything that narrows the search.

I'll try a boost-multilayer-packages recipe over several clean-sstate
cycles in the meantime and report back either way.  Once we have that,
merging this (in whichever variant) is easy.

Below is the shortened variant I mentioned -- meant to fully replace
the two python blocks + the comment header in your 2/2, drop-in on the
same insertion point in image-oci.bbclass. Same mechanism as yours:
event handler on RecipeTaskPreProcess attaches a prefunc to
do_image_oci that filters *-initial do_populate_sysroot nodes out of
BB_TASKDEPDATA before extend_recipe_sysroot walks it. Differences are
stylistic -- no deep-copy (dict comprehension builds a fresh dict
directly), set comprehension for the drop set, comment made smaller.

Not requesting a v2 on this; posting so you can try it against your
downstream trigger and see if it holds up the same way your original
does.

# =============================================================================
# Work around libgcc vs libgcc-initial sysroot collision in do_image_oci
# =============================================================================
#
# do_image_oci[depends] contains "...:do_populate_sysroot" entries, so
# oe-core's staging_taskhandler auto-attaches extend_recipe_sysroot as a
# prefunc. That walk exposes the full transitive populate_sysroot closure
# into this recipe's private sysroot -- including both libgcc and
# libgcc-initial, which collide on crtbegin.o.
#
# oe-core's SSTATE_EXCLUDEDEPS_SYSROOT (".*->.*-initial.*") is gated on
# the consuming task being do_populate_sysroot (see setscene_depvalid()
# in sstate.bbclass); for do_image_oci that gate is False and the
# exclusion is skipped. extend_recipe_sysroot itself has a hardcoded
# "*-initial" skip but only for do_sdk_depends / do_populate_sdk_ext
# (staging.bbclass:457) -- an oe-core task allow-list we can't cleanly
# extend from a downstream layer.
#
# Filter *-initial do_populate_sysroot nodes out of BB_TASKDEPDATA before
# extend_recipe_sysroot walks it. extend_recipe_sysroot re-reads
# BB_TASKDEPDATA on each entry, so a setVar with the filtered dict is
# sufficient.
python oci_strip_initial_taskdepdata () {
    tdd = d.getVar("BB_TASKDEPDATA", False)
    if not tdd:
        return
    drop = {k for k, v in tdd.items()
            if v[1] == "do_populate_sysroot" and v[0].endswith("-initial")}
    if not drop:
        return
    filtered = {k: v for k, v in tdd.items() if k not in drop}
    for v in filtered.values():
        v[3].difference_update(drop)
    d.setVar("BB_TASKDEPDATA", filtered)
    bb.note("OCI: dropped %d *-initial node(s) to avoid libgcc collision" % len(drop))
}
oci_strip_initial_taskdepdata[vardepsexclude] += "BB_TASKDEPDATA"

python oci_attach_strip_prefunc () {
    task = "do_image_oci"
    if task in e.tasklist:
        deps = d.getVarFlag(task, "depends")
        if deps and "populate_sysroot" in deps:
            d.prependVarFlag(task, "prefuncs", "oci_strip_initial_taskdepdata ")
}
oci_attach_strip_prefunc[eventmask] = "bb.event.RecipeTaskPreProcess"
addhandler oci_attach_strip_prefunc

Bruce

Re: [meta-virtualization] [PATCH 2/2] image-oci: drop *-initial sysroot deps in do_image_oci to avoid libgcc collision

[Koen Kooi]

> Op 29 jul 2026, om 05:17 heeft Bruce Ashfield via lists.yoctoproject.org <bruce.ashfield=gmail.com@lists.yoctoproject.org> het volgende geschreven:
> 
> [...]
> 
> Two questions to unblock:
> 
> 1) Can you share the specific recipe (or the DEPENDS shape) from
>  the downstream build where you hit this deterministically?
>  Something along the lines of a multilayer packages: layer
>  pulling a from-source-compiled library with a boost-ish
>  DEPENDS, if I'm reading your cover right, but I'd rather work
>  from your actual trigger than guess.

The incredibly messy WIP branch is here: https://github.com/koenkooi/meta-ai/tree/full-integration-test, you can build it via kas-container, but that will apply these (and more) patches to meta-virtualization automatically.

The important bits:

 meta-ai-tensorflow-lite-oci
   OCI_LAYER_MODE = "multi"
   OCI_LAYERS = "tensorflow-lite:packages:tensorflow-lite"
   OCI_BASE_IMAGE = "meta-ai-container-base-oci"

 meta-ai-container-base-oci
   OCI_LAYERS = "base:packages:base-files+base-passwd+netbase+glibc+libgcc+<abseil libs>"

DEPENDS in app image: flatbuffers-tflite, jpeg, libeigen, protobuf, plus gcc-runtime/abseil pulled through the base layer.

> 2) Was there anything else in that build environment worth
>  knowing -- IMAGE_PKGTYPE pin, non-default MACHINE, a specific
>  PACKAGECONFIG that pulled in one of the *-initial-adjacent
>  recipes? Anything that narrows the search.

I reproduced it with qemuarm64 as well now, instead of meta-qcom machines:

MACHINE=qemuarm64
  * meta-virtualization master-next
  * IMAGE_PKGTYPE unset -> default ipk/opkg
  * No PACKAGECONFIG

The reproducer itself, clean sstate, first try, unpatched:

 ERROR: meta-ai-tensorflow-lite-oci-1.0-r0 do_image_oci: The file /usr/lib/aarch64-oe-linux/16.1.0/crtendS.o is installed by both libgcc and libgcc-initial, aborting

Now the bad news: the patch fixes do_image_oci, but not do_image_complete, that hits the same issue on the same file:

 ERROR: meta-ai-tensorflow-lite-oci-1.0-r0 do_image_complete: Error executing a python function in exec_func_python() autogenerated:
 Exception: FileExistsError: [Errno 17] File exists: '/build/oe/repro/bugtest-qemux86/build/tmp/sysroots-components/cortexa57/libgcc-initial/usr/lib/aarch64-oe-linux/16.1.0/crtendS.o' -> '/build/oe/repro/bugtest-qemux86/build/tmp/work/qemuarm64-oe-linux/meta-ai-tensorflow-lite-oci/1.0/recipe-sysroot/usr/lib/aarch64-oe-linux/16.1.0/crtendS.o'

So patch 2/2 isn't enough to fix this issue :(

regards,

Koen

> 
> I'll try a boost-multilayer-packages recipe over several clean-sstate
> cycles in the meantime and report back either way.  Once we have that,
> merging this (in whichever variant) is easy.
> 
> Below is the shortened variant I mentioned -- meant to fully replace
> the two python blocks + the comment header in your 2/2, drop-in on the
> same insertion point in image-oci.bbclass. Same mechanism as yours:
> event handler on RecipeTaskPreProcess attaches a prefunc to
> do_image_oci that filters *-initial do_populate_sysroot nodes out of
> BB_TASKDEPDATA before extend_recipe_sysroot walks it. Differences are
> stylistic -- no deep-copy (dict comprehension builds a fresh dict
> directly), set comprehension for the drop set, comment made smaller.
> 
> Not requesting a v2 on this; posting so you can try it against your
> downstream trigger and see if it holds up the same way your original
> does.
> 
> # =============================================================================
> # Work around libgcc vs libgcc-initial sysroot collision in do_image_oci
> # =============================================================================
> #
> # do_image_oci[depends] contains "...:do_populate_sysroot" entries, so
> # oe-core's staging_taskhandler auto-attaches extend_recipe_sysroot as a
> # prefunc. That walk exposes the full transitive populate_sysroot closure
> # into this recipe's private sysroot -- including both libgcc and
> # libgcc-initial, which collide on crtbegin.o.
> #
> # oe-core's SSTATE_EXCLUDEDEPS_SYSROOT (".*->.*-initial.*") is gated on
> # the consuming task being do_populate_sysroot (see setscene_depvalid()
> # in sstate.bbclass); for do_image_oci that gate is False and the
> # exclusion is skipped. extend_recipe_sysroot itself has a hardcoded
> # "*-initial" skip but only for do_sdk_depends / do_populate_sdk_ext
> # (staging.bbclass:457) -- an oe-core task allow-list we can't cleanly
> # extend from a downstream layer.
> #
> # Filter *-initial do_populate_sysroot nodes out of BB_TASKDEPDATA before
> # extend_recipe_sysroot walks it. extend_recipe_sysroot re-reads
> # BB_TASKDEPDATA on each entry, so a setVar with the filtered dict is
> # sufficient.
> python oci_strip_initial_taskdepdata () {
>   tdd = d.getVar("BB_TASKDEPDATA", False)
>   if not tdd:
>       return
>   drop = {k for k, v in tdd.items()
>           if v[1] == "do_populate_sysroot" and v[0].endswith("-initial")}
>   if not drop:
>       return
>   filtered = {k: v for k, v in tdd.items() if k not in drop}
>   for v in filtered.values():
>       v[3].difference_update(drop)
>   d.setVar("BB_TASKDEPDATA", filtered)
>   bb.note("OCI: dropped %d *-initial node(s) to avoid libgcc collision" % len(drop))
> }
> oci_strip_initial_taskdepdata[vardepsexclude] += "BB_TASKDEPDATA"
> 
> python oci_attach_strip_prefunc () {
>   task = "do_image_oci"
>   if task in e.tasklist:
>       deps = d.getVarFlag(task, "depends")
>       if deps and "populate_sysroot" in deps:
>           d.prependVarFlag(task, "prefuncs", "oci_strip_initial_taskdepdata ")
> }
> oci_attach_strip_prefunc[eventmask] = "bb.event.RecipeTaskPreProcess"
> addhandler oci_attach_strip_prefunc
> 
> Bruce
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#10014): https://lists.yoctoproject.org/g/meta-virtualization/message/10014
> Mute This Topic: https://lists.yoctoproject.org/mt/120482434/9418801
> Group Owner: meta-virtualization+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/unsub [koen.kooi@oss.qualcomm.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [meta-virtualization] [PATCH 2/2] image-oci: drop *-initial sysroot deps in do_image_oci to avoid libgcc collision

[Bruce Ashfield]

Aha .. I missed some of the details your cover-letter reply until just
now.  Rereading it now, along with today's do_image_complete report,
changes what I think we should do here.

Disregard the shortened variant I posted.  It inherits the same scope
limitation as 2/2. If the reachability path is actually

  do_image_oci -> do_image -> do_create_rootfs_spdx
      -> libgcc-initial.do_create_spdx
      -> libgcc-initial.do_populate_sysroot

then libgcc-initial is coming in through do_rootfs's own SPDX chain,
not through do_image_oci[depends] at all. Which means:

  * Anything that inherits image has the same reachable node,
    which matches exactly what you're seeing on do_image_complete
    hitting the same crtendS.o collision.
  * Patching only do_image_oci's prefunc is whack-a-mole. There
    are more image tasks in that chain and each one hits the
    same wall for the same reason.
  * I agree with your original thought in the cover letter --
    "blacklisting -initial feels a bit too specific and narrow
    to me" .. but I still don't have a solid idea either.

So: not merging 2/2, and dropping the shortened version I sent as
an alternative. Neither addresses the actual scope.

On next steps.  I'd like to try your reproducer myself so we're both
looking at the same failure. My boost-in-a-packages-layer recipe ran 8
clean-sstate cycles without triggering it; whatever crosses the
"sstate has to populate both libgcc variants" line was clearly missing
in my effort, so tensorflow-lite as the actual C++-heavy trigger might
actually trigger it for me.

Once I can hit it locally, I want to look at your first option --
fixing SSTATE_EXCLUDEDEPS_SYSROOT's gating in oe-core to key off the
dependency being *-initial instead of requiring the consuming task to
be do_populate_sysroot. That's the root-cause fix, and if it we fix it
in oe-core it covers do_image_oci, do_image_complete, and whatever the
next image task in the SPDX chain is that we haven't hit yet, all at
once.

The current gating is could arguably a bug of its own -- the exclusion
regex is named SSTATE_EXCLUDEDEPS_SYSROOT, not
SSTATE_EXCLUDEDEPS_DO_POPULATE_SYSROOT_ONLY, and the gate narrows it
to only one consumer for reasons that don't seem to be documented
anywhere I could find.

The other two options you named: keeping do_image_oci off the generic
prefunc path, or giving do_image_oci its own sysroot .. I think they
would still leave do_image_complete and friends broken by the same
root cause .. so we'd just be chasing similar things popping up.

I'll try and get set up to see the problem here, that way either of
us could potentially take a run at the oe-core fix.

If that fails, we'll come up with something that can live in meta-virt
since this does need to be fixed.

Bruce

Re: [meta-virtualization][PATCH 0/2] image-oci: two do_image_oci bugfixes (missing pm.update(), libgcc/libgcc-initial sysroot collision)

[Bruce Ashfield]

On Tue, Jul 28, 2026 at 7:56 AM Koen Kooi via lists.yoctoproject.org
<koen.kooi=oss.qualcomm.com@lists.yoctoproject.org> wrote:
>
> I've been working on a prototype for building a rootfs with multiple multi-layer OCI images images, all built from scratch, in one bitbake command. I ran into some issues along the way:
>
> * image-oci never runs "opkg update" or its rpm equivalents, so nothing gets installed
> * both libgcc-initial and libgcc want to be in the sysroot
>
> The first issue is straightforward, for the second I leaned heavily on Claude to write a proper python version and explain it better. I'm not fond of the wall of text that ends up with, but this is already the 3rd iteration of trimming it down.
>
> Anyway, This is the first wave, I'm working on other fixes, like mtime influencing base-layer sharing, but that needs more testing and review. I do need to mention how surprisingly small the resulting image recipes are, a big thanks to everyone who has been working on this!
>
> Below is what Claude suggested as coverletter, it is still verbose, but doesn't suffer from the "Is Koen being sarcastic or not?" issue :)
>
> 1/2 - image-oci: call pm.update() before pm.install() for multi-layer
>       package layers
>
>   Multi-layer OCI images (OCI_LAYER_MODE = "multi") with a "packages:"
>   layer silently produce an empty layer: do_image_oci reports success,
>   but the layer's opkg/rpm install never actually installs anything,
>   because the per-layer package manager's local index/cache was never
>   refreshed after write_index() (do_rootfs itself always calls
>   pm.update() between the two; image-oci's per-layer installer
>   skipped it). The failure is only visible as a bb.warn(), so the
>   build looks green while shipping an empty (or partial) layer.

Aha. Since I've obviously tested the package installs and they work,
but multi layer is something that I don't recall testing in that combination.

>
>   Reproduced with a minimal recipe (base-files+coreutils in a
>   "packages:" layer): before the fix, the resulting OCI layer blob
>   contains 0 regular files; after the fix, the same layer contains
>   272 files and the compressed blob grows from 299 bytes to 7.5 MB.
>   Full before/after evidence in the patch commit message and in
>   TEST-CASES.md if you'd like to reproduce it independently.
>

I'd definitely want to see a reproducer, both so I can confirm how
I didn't see this before, and so I can get it into the layer tests.

Does this same image reproduce the second issue as well ?
I'd also like to reproduce that, to hunt around and see if there's
any other options to resolve the problem.

Bruce

> 2/2 - image-oci: drop *-initial sysroot deps in do_image_oci to avoid
>       libgcc collision
>
>   do_image_oci can abort with:
>     ERROR: <recipe> do_image_oci: The file .../crtbegin.o is installed
>     by both libgcc and libgcc-initial, aborting
>
>   Root cause is in extend_recipe_sysroot's dependency walk: the
>   SSTATE_EXCLUDEDEPS_SYSROOT exclusion that normally keeps libgcc and
>   libgcc-initial from landing in the same sysroot only applies when
>   the *consuming* task is itself do_populate_sysroot. do_image_oci
>   reaches extend_recipe_sysroot through the generic "any task whose
>   [depends] contains populate_sysroot gets this prefunc" path
>   (staging_taskhandler), so that exclusion never fires for it.
>
>   This one is sstate-state-dependent -- whether it fires depends on
>   whether extend_recipe_sysroot's walk actually needs to populate (vs.
>   skip via sstate/setscene shortcuts) both libgcc and libgcc-initial
>   for a given recipe/build. A minimal single-package OCI recipe did
>   not reach libgcc in our testing, but a multi-layer "packages:" recipe
>   installing a from-source-compiled library with a moderately large
>   DEPENDS (boost) did: instrumenting the fix confirms it stripping two
>   "*-initial" do_populate_sysroot nodes out of BB_TASKDEPDATA for that
>   recipe, i.e. libgcc-initial is a real, reachable node in
>   do_image_oci's dependency walk for realistic recipes. We were not
>   able to force the raw abort deterministically within our test
>   window (a downstream build did hit it deterministically across 6+
>   clean-sstate attempts on a wider dependency set) -- see the patch
>   commit message and TEST-CASES.md for exactly what was and wasn't
>   reproduced, and why the fix is justified regardless.
>
>
> Signed-off-by: Koen Kooi <koen.kooi@oss.qualcomm.com>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#9990): https://lists.yoctoproject.org/g/meta-virtualization/message/9990
> Mute This Topic: https://lists.yoctoproject.org/mt/120482430/1050810
> Group Owner: meta-virtualization+owner@lists.yoctoproject.org
> Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/unsub [bruce.ashfield@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>


--
- Thou shalt not follow the NULL pointer, for chaos and madness await
thee at its end
- "Use the force Harry" - Gandalf, Star Trek II

Re: [meta-virtualization][PATCH 0/2] image-oci: two do_image_oci bugfixes (missing pm.update(), libgcc/libgcc-initial sysroot collision)

[Koen Kooi]

[-- Attachment #1: Type: text/plain, Size: 6391 bytes --]



> Op 28 jul 2026, om 14:45 heeft Bruce Ashfield <bruce.ashfield@gmail.com> het volgende geschreven:
> 
> On Tue, Jul 28, 2026 at 7:56 AM Koen Kooi via lists.yoctoproject.org
> <koen.kooi=oss.qualcomm.com@lists.yoctoproject.org> wrote:
>> 
>> I've been working on a prototype for building a rootfs with multiple multi-layer OCI images images, all built from scratch, in one bitbake command. I ran into some issues along the way:
>> 
>> * image-oci never runs "opkg update" or its rpm equivalents, so nothing gets installed
>> * both libgcc-initial and libgcc want to be in the sysroot
>> 
>> The first issue is straightforward, for the second I leaned heavily on Claude to write a proper python version and explain it better. I'm not fond of the wall of text that ends up with, but this is already the 3rd iteration of trimming it down.
>> 
>> Anyway, This is the first wave, I'm working on other fixes, like mtime influencing base-layer sharing, but that needs more testing and review. I do need to mention how surprisingly small the resulting image recipes are, a big thanks to everyone who has been working on this!
>> 
>> Below is what Claude suggested as coverletter, it is still verbose, but doesn't suffer from the "Is Koen being sarcastic or not?" issue :)
>> 
>> 1/2 - image-oci: call pm.update() before pm.install() for multi-layer
>>      package layers
>> 
>>  Multi-layer OCI images (OCI_LAYER_MODE = "multi") with a "packages:"
>>  layer silently produce an empty layer: do_image_oci reports success,
>>  but the layer's opkg/rpm install never actually installs anything,
>>  because the per-layer package manager's local index/cache was never
>>  refreshed after write_index() (do_rootfs itself always calls
>>  pm.update() between the two; image-oci's per-layer installer
>>  skipped it). The failure is only visible as a bb.warn(), so the
>>  build looks green while shipping an empty (or partial) layer.
> 
> Aha. Since I've obviously tested the package installs and they work,
> but multi layer is something that I don't recall testing in that combination.

It's a first for me as well :)

> 
>> 
>>  Reproduced with a minimal recipe (base-files+coreutils in a
>>  "packages:" layer): before the fix, the resulting OCI layer blob
>>  contains 0 regular files; after the fix, the same layer contains
>>  272 files and the compressed blob grows from 299 bytes to 7.5 MB.
>>  Full before/after evidence in the patch commit message and in
>>  TEST-CASES.md if you'd like to reproduce it independently.
>> 
> 
> I'd definitely want to see a reproducer, both so I can confirm how
> I didn't see this before, and so I can get it into the layer tests.
> 
> Does this same image reproduce the second issue as well ?

It does not, I've attached a tarball with what I think is the minimal reproducer from scratch. Tried it on a different, clean build machine and after suffering through llvm/clang/boost builds, it does reproduce! My robot buddy summed it up as:

Both real builds ran to completion, first-hand:
    • Bug 1 (pm.update): reproduced live — 0-file layer unpatched, confirmed by extracting the actual OCI tar myself.
    • Bug 2 (libgcc-initial): the raw abort didn't fire in two attempts (plain build + cleansstate-forced rebuild), matching the earlier investigation. But I got independent proof of the mechanism that doesn't need Koen's patch at all — a bitbake -g + task-depends.dot BFS showing libgcc-initial.do_populate_sysroot is genuinely reachable from do_image_oci via the rootfs/SPDX chain. Bonus: the boost recipe also hits bug 1, so it's not coreutils-specific.

> I'd also like to reproduce that, to hunt around and see if there's
> any other options to resolve the problem.

The blacklisting of -initial feels a bit too specific and narrow to me. But I've never really looked into this part of OE, so I don't trust my feelings on that.

regards,

Koen

Robot generated text:

Reproducer attached (image-oci-repro.tar.gz): standalone kas tree, oe-core + meta-openembedded + meta-virtualization, all master, qemux86-64, package_ipk. README has the exact commands; short version below.

  kas checkout kas-bugtest.yml
  . openembedded-core/oe-init-build-env build
  rm -rf oci-layer-cache && bitbake test-oci-pmupdate

do_image_oci warns "Unable to install packages... Couldn't find anything to satisfy 'base-files'" but reports Succeeded. Pulled the shipped OCI tar apart: layer blob is 299B compressed/7680B uncompressed, 0 regular files, just the bin/lib/sbin/usr/var skeleton. With 1/2 applied + cache cleared + cleansstate: 272 files, 7.5MB, all the expected coreutils/base-files content.

Same image doesn't reach bug 2 - coreutils never pulls libgcc into do_image_oci's walk. Swapped it for boost (test-oci-initial-collision, also attached): same multi+packages shape, so it hits bug 1 too (opkg install of base-files+boost fails the same way - not just a coreutils quirk). For bug 2 specifically: `bitbake -g` + a BFS over task-depends.dot from do_image_oci finds do_image_oci -> do_image -> do_create_rootfs_spdx -> libgcc-initial.do_create_spdx -> libgcc-initial.do_populate_sysroot - so libgcc-initial is reachable, via do_rootfs's own SPDX chain, not via do_image_oci's declared [depends] (umoci/jq/opkg/rsync-native, none of which reach it). That's the mechanism confirmed live, no patch needed to see it.

Couldn't force the raw crtbegin.o abort itself, unpatched, in two tries - plain build, and cleansstate on libgcc+libgcc-initial+umoci-native+jq-native+recipe. Both built clean. Consistent with what we saw downstream: depends on whether extend_recipe_sysroot actually has to populate both libgcc variants vs. setscene-shortcut them, which cleansstate on a handful of recipes doesn't control. Downstream hit it deterministically over 6+ clean-sstate attempts on a wider DEPENDS set; not reproduced here.

Options I looked at for 2/2 besides the BB_TASKDEPDATA strip: fixing SSTATE_EXCLUDEDEPS_SYSROOT in oe-core to key off the dependency instead of the consuming task being do_populate_sysroot (root cause, wide blast radius), keeping do_image_oci off staging_taskhandler's generic prefunc path, giving do_image_oci its own sysroot. Happy to take a run at any of those if you'd rather not carry the strip as a meta-virtualization-local workaround.


[-- Attachment #2: imageocirepro.tar.gz --]
[-- Type: application/x-gzip, Size: 4271 bytes --]

[-- Attachment #3: Type: text/plain, Size: 2518 bytes --]


> 
> Bruce
> 
>> 2/2 - image-oci: drop *-initial sysroot deps in do_image_oci to avoid
>>      libgcc collision
>> 
>>  do_image_oci can abort with:
>>    ERROR: <recipe> do_image_oci: The file .../crtbegin.o is installed
>>    by both libgcc and libgcc-initial, aborting
>> 
>>  Root cause is in extend_recipe_sysroot's dependency walk: the
>>  SSTATE_EXCLUDEDEPS_SYSROOT exclusion that normally keeps libgcc and
>>  libgcc-initial from landing in the same sysroot only applies when
>>  the *consuming* task is itself do_populate_sysroot. do_image_oci
>>  reaches extend_recipe_sysroot through the generic "any task whose
>>  [depends] contains populate_sysroot gets this prefunc" path
>>  (staging_taskhandler), so that exclusion never fires for it.
>> 
>>  This one is sstate-state-dependent -- whether it fires depends on
>>  whether extend_recipe_sysroot's walk actually needs to populate (vs.
>>  skip via sstate/setscene shortcuts) both libgcc and libgcc-initial
>>  for a given recipe/build. A minimal single-package OCI recipe did
>>  not reach libgcc in our testing, but a multi-layer "packages:" recipe
>>  installing a from-source-compiled library with a moderately large
>>  DEPENDS (boost) did: instrumenting the fix confirms it stripping two
>>  "*-initial" do_populate_sysroot nodes out of BB_TASKDEPDATA for that
>>  recipe, i.e. libgcc-initial is a real, reachable node in
>>  do_image_oci's dependency walk for realistic recipes. We were not
>>  able to force the raw abort deterministically within our test
>>  window (a downstream build did hit it deterministically across 6+
>>  clean-sstate attempts on a wider dependency set) -- see the patch
>>  commit message and TEST-CASES.md for exactly what was and wasn't
>>  reproduced, and why the fix is justified regardless.
>> 
>> 
>> Signed-off-by: Koen Kooi <koen.kooi@oss.qualcomm.com>
>> 
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#9990): https://lists.yoctoproject.org/g/meta-virtualization/message/9990
>> Mute This Topic: https://lists.yoctoproject.org/mt/120482430/1050810
>> Group Owner: meta-virtualization+owner@lists.yoctoproject.org
>> Unsubscribe: https://lists.yoctoproject.org/g/meta-virtualization/unsub [bruce.ashfield@gmail.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>> 
> 
> 
> --
> - Thou shalt not follow the NULL pointer, for chaos and madness await
> thee at its end
> - "Use the force Harry" - Gandalf, Star Trek II