Proposal: Utilizing Container Registry for Shared BitBake Sstate-Cache

Proposal: Utilizing Container Registry for Shared BitBake Sstate-Cache

[Yash Patel]

Hello Team,

My name is Yash Patel and I'm a summer intern at IBM working with
Andrew Geissler. My goal is to start storing the sstate for our
bitbake builds into containers and upload them to a container registry
that the openbmc CI process can utilize as well as other openbmc
developers. This will make it much easier to bring new build nodes
online and to reset bad ones. It will also allow openbmc developers to
be able to quickly spin up a container and do bitbakes quickly for a
target machine with the sstate already pre-loaded.

There would be a container per machine type. The default machines
supported would be what we run CI for up at
https://jenkins.openbmc.org/job/ci-openbmc/. Supporting only a single
machine per container will keep the size of the container down and
most use cases are just building a single machine. Also, a lot of the
free opensource container registries have size limits on the
containers.

We are thinking that the
https://jenkins.openbmc.org/job/latest-master/ job will be what
generates and uploads the containers. Currently this job runs once a
day and builds whatever is in master at the time (we could tweak this
schedule if needed). We would then update
https://github.com/openbmc/openbmc-build-scripts/blob/master/build-setup.sh
(script used by openbmc CI) to look for an available container and use
it if available, otherwise just default to the standard flow.

We would generate containers for both x86 and arm as they will have
different sstates.

We've done some research and it appears that github provides a free
container registry for open source projects so this is the direction
we're thinking.

Any thoughts or comments appreciated!
Yash

Re: Proposal: Utilizing Container Registry for Shared BitBake Sstate-Cache

[Patrick Williams]

[-- Attachment #1: Type: text/plain, Size: 4550 bytes --]

On Thu, Jun 04, 2026 at 09:15:42AM -0500, Yash Patel wrote:
> Hello Team,
> 
> My name is Yash Patel and I'm a summer intern at IBM working with
> Andrew Geissler. 

Welcome Yash.

> My goal is to start storing the sstate for our
> bitbake builds into containers and upload them to a container registry
> that the openbmc CI process can utilize as well as other openbmc
> developers. This will make it much easier to bring new build nodes
> online and to reset bad ones. It will also allow openbmc developers to
> be able to quickly spin up a container and do bitbakes quickly for a
> target machine with the sstate already pre-loaded.

Sounds like a good idea.

One of the issues we have right now is that we do not have a good way to
clean up the sstate/downloads directories out of each Jenkins node.
Eventually they run out of space.  Managing it through containers will
hopefully make it easier to prune old state and keep the nodes from
running out of space.

> There would be a container per machine type. The default machines
> supported would be what we run CI for up at
> https://jenkins.openbmc.org/job/ci-openbmc/. Supporting only a single
> machine per container will keep the size of the container down and
> most use cases are just building a single machine. Also, a lot of the
> free opensource container registries have size limits on the
> containers.

In addition to per-platform containers, you'll also need the branch
included there.  We should make sure to build this for Wrynose since
that is the Yocto LTS branch and we've committed to supporting that for
a few years.

> 
> We are thinking that the
> https://jenkins.openbmc.org/job/latest-master/ job will be what
> generates and uploads the containers. Currently this job runs once a
> day and builds whatever is in master at the time (we could tweak this
> schedule if needed). 

I would suggest a separate job for two reasons:
    - Ideally we need to trigger this more often than once per day if
      you want other Jenkins nodes to use that for a starting point of
      their sstate / downloads cache.

    - You are going to want to create sub-jobs per machine / branch (and
      like I mentioned, we should do this at least for the Wrynose
      branch in addition to master).

> We would then update
> https://github.com/openbmc/openbmc-build-scripts/blob/master/build-setup.sh
> (script used by openbmc CI) to look for an available container and use
> it if available, otherwise just default to the standard flow.

When you create these containers, ideally you'd use the
most-recent-previous container's sstate and downloads directory as a
mirror for bitbake.  This will let you do incremental rebuilds of these
sstate containers much faster.  If you try to build it fresh all the
time, you're going to consume hours of CI time per platform we're
trying to build (plus the x86/arm duplication).

Make sure when you do this that you don't use 'FROM: <old container>'
because that will just create a chain of container subsets that will
grow monsterrously large over time.  You have to create a build
container and a final container.  The build container uses the 'FROM' to
pull the previous container, but then the final container just takes the
resulting final sstate/downloads as a directory.

One tricky thing in the `build-setup` is that you're going to have to
figure out what the "latest best container" is to use.  You're going to
have to traverse the git history of what you're trying to build looking
for a docker tag that exists somewhere in the history as your starting
point.  That could be the previous commit or it could be dozens of
commits old.

> We would generate containers for both x86 and arm as they will have
> different sstates.
> 
> We've done some research and it appears that github provides a free
> container registry for open source projects so this is the direction
> we're thinking.

Have you done any experimentation on if the GHCR will rate limit us in a
way to make this non-useful?  Do we have enough space on the Jenkins
server to run a container repository there?  There are a few smaller
ones that are self-contained in containers like how we run
Jenkins/Gerrit.

We might want a Jenkins job on each node that is continuously pulling
the latest images from the container repository so that each node is
already "up to date" when a CI job kicks off.

> Any thoughts or comments appreciated!
> Yash
> 

-- 
Patrick Williams

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 870 bytes --]