Hosting first-party kernel.org container images

Hosting first-party kernel.org container images

[Guillaume Tucker]

Hi Konstantin, Arnd et al,

This is a follow-up from the series about adding a scripts/container
tool [1] to run kernel builds in containers.  As per the discussion
at Plumbers last year and the summary I put in a blog post [2], it
would be great to have container images with kernel.org toolchains
hosted upstream.  This can mean several things, so let's break it
down into a set of potential options to choose from:


* Containerfiles Git repository

There is currently a PoC repository on GitLab with a Makefile and a
number of Containerfiles to build a set of images:

     https://gitlab.com/gtucker/korg-containers

It can be improved in many ways since this is an early PoC.  The key
decision to make here, if we do want to have container images
supported upstream, is how to manage these files or a derived
implementation.

One option is to add it to the kernel tree itself under e.g.
tools/container.

Another option is to add a separate repository on git.kernel.org,
which I believe would be a better approach as there aren't any direct
dependencies on the kernel tree itself.

A third option might be to keep it alongside any recipes used to
produce the existing kernel.org toolchain tarballs although I'm not
entirely sure how that's managed - something for Arnd to judge I
guess.

A last option would be to keep it on GitLab or move it to GitHub
which would provide some CI/CD tools for building the images but I
doubt this is something viable for the kernel community as it would
create some vendor lock-in.


* Container image registry

This is where things get a bit more complicated.  As far as I'm
aware, there aren't any container registries hosted in the kernel.org
infrastructure at the moment.  A classic option would be to push the
images to an established one e.g. Docker Hub (docker.io) or the
Google Artifact Registry.  GitLab and GitHub also provide theirs of
course.  I believe there is still a free plan for community projects
to host images on docker.io and that would be the easiest from a user
point of view e.g. "docker pull kernel.org/gcc".  It comes with some
maintenance burden of course, and Docker Hub has a history of
changing its policies quite unexpectedly so it's not entirely
future-proof.

A classic alternative would be to host a dedicated service
e.g. registry.kernel.org and have the images managed there.  This
would obviously involve higher sysadmin efforts and add scalability
issues but would decouple it from external providers.

Then a third option would be to host the container images as OCI
tarball dumps alongside the toolchain tarballs.  They can then be
downloaded and imported with "docker image load" or any other
container runtime.  The only infrastructure resources needed would be
storage space.  This is of course suboptimal as all the layers get
bundled together and users would have to manage these images
themselves, but it's very effective from a kernel.org sysadmin point
of view.


There are undoubtedly other ways to look at this, I'm curious to know
what people think.  The benefits of having readily-available
container images upstream appear to be pretty clear, several
maintainers have expressed their support already.  It's all down to
how much these benefits can outweigh the upstream maintenance costs.
Maybe this can be done in two steps, first with just the
Containerfiles and later on a full solution to host the actual
images.

Best wishes,
Guillaume

[1] https://lore.kernel.org/all/cover.1766061692.git.gtucker@gtucker.io/
[2] https://gtucker.io/posts/2024-09-30-korg-containers/

Re: Hosting first-party kernel.org container images

[Ben Copeland]

Hello Guillaume,

On Fri, 19 Dec 2025 at 13:37, Guillaume Tucker <gtucker@gtucker.io> wrote:
>
> Hi Konstantin, Arnd et al,
>
> This is a follow-up from the series about adding a scripts/container
> tool [1] to run kernel builds in containers.  As per the discussion
> at Plumbers last year and the summary I put in a blog post [2], it
> would be great to have container images with kernel.org toolchains
> hosted upstream.  This can mean several things, so let's break it
> down into a set of potential options to choose from:
>
>
> * Containerfiles Git repository
>
> There is currently a PoC repository on GitLab with a Makefile and a
> number of Containerfiles to build a set of images:
>
>      https://gitlab.com/gtucker/korg-containers

TuxMake already provides container images with kernel.org toolchains
(korg-gcc 8-15, korg-clang 11-22). The Dockerfiles are maintained at
[1]

Since TuxMake is now part of KernelCI and already referenced in the
kernel documentation you pushed [2], it seems like a natural home for
this rather than starting fresh, or having two places for images.

>
> It can be improved in many ways since this is an early PoC.  The key
> decision to make here, if we do want to have container images
> supported upstream, is how to manage these files or a derived
> implementation.
>
> One option is to add it to the kernel tree itself under e.g.
> tools/container.
>
> Another option is to add a separate repository on git.kernel.org,
> which I believe would be a better approach as there aren't any direct
> dependencies on the kernel tree itself.
>
> A third option might be to keep it alongside any recipes used to
> produce the existing kernel.org toolchain tarballs although I'm not
> entirely sure how that's managed - something for Arnd to judge I
> guess.
>
> A last option would be to keep it on GitLab or move it to GitHub
> which would provide some CI/CD tools for building the images but I
> doubt this is something viable for the kernel community as it would
> create some vendor lock-in.
>
>
> * Container image registry
>
> This is where things get a bit more complicated.  As far as I'm
> aware, there aren't any container registries hosted in the kernel.org
> infrastructure at the moment.  A classic option would be to push the
> images to an established one e.g. Docker Hub (docker.io) or the
> Google Artifact Registry.  GitLab and GitHub also provide theirs of
> course.  I believe there is still a free plan for community projects
> to host images on docker.io and that would be the easiest from a user
> point of view e.g. "docker pull kernel.org/gcc".  It comes with some
> maintenance burden of course, and Docker Hub has a history of
> changing its policies quite unexpectedly so it's not entirely
> future-proof.

The images are hosted on Docker Hub (https://hub.docker.com/u/tuxmake)
with ECR Public as fallback:

      docker pull tuxmake/arm64_korg-gcc-14
      docker pull tuxmake/x86_64_korg-clang-22

Happy to discuss how we can align efforts here. But to me, it makes
sense for KernelCI to be the place for these images (or for us to have
a single place and a single set of images).

[1] https://github.com/kernelci/tuxmake/tree/master/support/docker
[2] https://www.kernel.org/doc/html/next/dev-tools/container.html

Regards,

Ben

>
> A classic alternative would be to host a dedicated service
> e.g. registry.kernel.org and have the images managed there.  This
> would obviously involve higher sysadmin efforts and add scalability
> issues but would decouple it from external providers.
>
> Then a third option would be to host the container images as OCI
> tarball dumps alongside the toolchain tarballs.  They can then be
> downloaded and imported with "docker image load" or any other
> container runtime.  The only infrastructure resources needed would be
> storage space.  This is of course suboptimal as all the layers get
> bundled together and users would have to manage these images
> themselves, but it's very effective from a kernel.org sysadmin point
> of view.
>
>
> There are undoubtedly other ways to look at this, I'm curious to know
> what people think.  The benefits of having readily-available
> container images upstream appear to be pretty clear, several
> maintainers have expressed their support already.  It's all down to
> how much these benefits can outweigh the upstream maintenance costs.
> Maybe this can be done in two steps, first with just the
> Containerfiles and later on a full solution to host the actual
> images.
>
> Best wishes,
> Guillaume
>
> [1] https://lore.kernel.org/all/cover.1766061692.git.gtucker@gtucker.io/
> [2] https://gtucker.io/posts/2024-09-30-korg-containers/

Re: Hosting first-party kernel.org container images

[Guillaume Tucker]

+kernelci

On 02/02/2026 13:07, Ben Copeland wrote:
> Hello Guillaume,
> 
> On Fri, 19 Dec 2025 at 13:37, Guillaume Tucker <gtucker@gtucker.io> wrote:
>>
>> Hi Konstantin, Arnd et al,
>>
>> This is a follow-up from the series about adding a scripts/container
>> tool [1] to run kernel builds in containers.  As per the discussion

The scripts/container tool has now been merged so it's on its way to
be part of v7.0:

    https://docs.kernel.org/next/dev-tools/container.html

As such, the topic of maintaining first-party container images is
becoming even more relevant.

>> at Plumbers last year and the summary I put in a blog post [2], it
>> would be great to have container images with kernel.org toolchains
>> hosted upstream.  This can mean several things, so let's break it
>> down into a set of potential options to choose from:
>>
>>
>> * Containerfiles Git repository
>>
>> There is currently a PoC repository on GitLab with a Makefile and a
>> number of Containerfiles to build a set of images:
>>
>>      https://gitlab.com/gtucker/korg-containers
> 
> TuxMake already provides container images with kernel.org toolchains
> (korg-gcc 8-15, korg-clang 11-22). The Dockerfiles are maintained at
> [1]
> 
> Since TuxMake is now part of KernelCI and already referenced in the
> kernel documentation you pushed [2], it seems like a natural home for
> this rather than starting fresh, or having two places for images.

Thanks Ben for the input here, these sound like very good points and
the situation has improved since the idea of having upstream
containers was first initiated.  The TuxMake images are mentioned in
the documentation as they're already available so it made sense in
practice.

>> It can be improved in many ways since this is an early PoC.  The key
>> decision to make here, if we do want to have container images
>> supported upstream, is how to manage these files or a derived
>> implementation.
>>
>> One option is to add it to the kernel tree itself under e.g.
>> tools/container.
>>
>> Another option is to add a separate repository on git.kernel.org,
>> which I believe would be a better approach as there aren't any direct
>> dependencies on the kernel tree itself.
>>
>> A third option might be to keep it alongside any recipes used to
>> produce the existing kernel.org toolchain tarballs although I'm not
>> entirely sure how that's managed - something for Arnd to judge I
>> guess.
>>
>> A last option would be to keep it on GitLab or move it to GitHub
>> which would provide some CI/CD tools for building the images but I
>> doubt this is something viable for the kernel community as it would
>> create some vendor lock-in.
>>
>>
>> * Container image registry
>>
>> This is where things get a bit more complicated.  As far as I'm
>> aware, there aren't any container registries hosted in the kernel.org
>> infrastructure at the moment.  A classic option would be to push the
>> images to an established one e.g. Docker Hub (docker.io) or the
>> Google Artifact Registry.  GitLab and GitHub also provide theirs of
>> course.  I believe there is still a free plan for community projects
>> to host images on docker.io and that would be the easiest from a user
>> point of view e.g. "docker pull kernel.org/gcc".  It comes with some
>> maintenance burden of course, and Docker Hub has a history of
>> changing its policies quite unexpectedly so it's not entirely
>> future-proof.
> 
> The images are hosted on Docker Hub (https://hub.docker.com/u/tuxmake)
> with ECR Public as fallback:
> 
>       docker pull tuxmake/arm64_korg-gcc-14
>       docker pull tuxmake/x86_64_korg-clang-22
> 
> Happy to discuss how we can align efforts here. But to me, it makes
> sense for KernelCI to be the place for these images (or for us to have
> a single place and a single set of images).

Yes, I see the value in consolidating things rather than duplicating
efforts.  However, I believe there are still a few good reasons worth
considering for keeping a separate set of images:

* images related to a tool e.g. TuxMake or KernelCI are less likely
  to be adopted by developers

CI bots and other services are a subset of the use cases for these
container images, so if they're branded as being tool-specific then
developers will see them as specialised images.

An ideal approach, if there were no images already published
anywhere, would be to have Containerfiles and images coming directly
from kernel.org and then specialised use cases would base their own
tooling on top.  I know the current TuxMake images with kernel.org
toolchains don't have TuxMake-specific packages installed in
principle, but they're still branded as TuxMake and it's hard to tell
how they might evolve.

* external projects can have their own agendas

Projects such as TuxMake and KernelCI may decide to host images on
docker.io and code on GitHub for now, which may not be the preferred
choices for the kernel developers' community.  Then if the projects
are forced to move hosting because of pricing or limitations, it's
also likely to be for reasons beyond or against the community's best
interests.


Creating those container images isn't particularly challenging in
itself, the main point is to align use cases with a standard set of
images to facilitate reproducible builds across the board.  And the
main part of the maintenance work is to keep them up to date,
functional and available.  As such, any pre-existing implementation
or hosting solution doesn't seem like a big factor for deciding what
to host and where.  Rather, the interests and resources associated
with them are critical.  In other words, it's much easier to create a
new repository from scratch on git.kernel.org and implement a new way
of building these images there than having to deal with the ongoing
risks associated with third parties.

This boils things down to a few practical options:

1. treating TuxMake images with kernel.org toolchains as the de facto
   stardard,

2. creating a repository from scratch on git.kernel.org with
   independent hosting for base images,

3. some middle ground to be defined which would remove the risks
   associated with third parties without duplicating efforts.

I feel it would be good to have more maintainers' feedback though.

Nathan, Nicolas, Miguel - what are your thoughts on this?

Thanks,
Guillaume

> [1] https://github.com/kernelci/tuxmake/tree/master/support/docker
> [2] https://www.kernel.org/doc/html/next/dev-tools/container.html
> 
> Regards,
> 
> Ben
> 
>>
>> A classic alternative would be to host a dedicated service
>> e.g. registry.kernel.org and have the images managed there.  This
>> would obviously involve higher sysadmin efforts and add scalability
>> issues but would decouple it from external providers.
>>
>> Then a third option would be to host the container images as OCI
>> tarball dumps alongside the toolchain tarballs.  They can then be
>> downloaded and imported with "docker image load" or any other
>> container runtime.  The only infrastructure resources needed would be
>> storage space.  This is of course suboptimal as all the layers get
>> bundled together and users would have to manage these images
>> themselves, but it's very effective from a kernel.org sysadmin point
>> of view.
>>
>>
>> There are undoubtedly other ways to look at this, I'm curious to know
>> what people think.  The benefits of having readily-available
>> container images upstream appear to be pretty clear, several
>> maintainers have expressed their support already.  It's all down to
>> how much these benefits can outweigh the upstream maintenance costs.
>> Maybe this can be done in two steps, first with just the
>> Containerfiles and later on a full solution to host the actual
>> images.
>>
>> Best wishes,
>> Guillaume
>>
>> [1] https://lore.kernel.org/all/cover.1766061692.git.gtucker@gtucker.io/
>> [2] https://gtucker.io/posts/2024-09-30-korg-containers/

Re: Hosting first-party kernel.org container images

[Nathan Chancellor]

Hi Guillaume,

On Wed, Feb 25, 2026 at 03:44:13PM +0100, Guillaume Tucker wrote:
> This boils things down to a few practical options:
> 
> 1. treating TuxMake images with kernel.org toolchains as the de facto
>    stardard,
> 
> 2. creating a repository from scratch on git.kernel.org with
>    independent hosting for base images,
> 
> 3. some middle ground to be defined which would remove the risks
>    associated with third parties without duplicating efforts.
> 
> I feel it would be good to have more maintainers' feedback though.
> 
> Nathan, Nicolas, Miguel - what are your thoughts on this?

To be entirely honest, I do not really have a strong opinion here. I
generally agree with your thoughts around branded container images,
although I do think that TuxMake's images tend to be fairly lean, so
those easily could become the recommended images without many downsides
aside from maybe where they are hosted and how they are maintained.
Having a clean set of Containerfiles on git.kernel.org does not sound
like a bad idea, especially if they would be structured in such a way
that other entities could customize them for their use case further. I
guess its usefulness really depends on the other stakeholders like
KernelCI. It seems like there has to be some sort of buy in from the
kernel.org administrators around hosting built container images
somewhere on kernel.org though, as I don't think the regular developer
is going to want to build images locally. Not really sure what that
looks like.

Cheers,
Nathan

Re: Hosting first-party kernel.org container images

[Guillaume Tucker]

Hi Nathan et al,

On 10/03/2026 1:48 am, Nathan Chancellor wrote:
> Hi Guillaume,
> 
> On Wed, Feb 25, 2026 at 03:44:13PM +0100, Guillaume Tucker wrote:
>> This boils things down to a few practical options:
>>
>> 1. treating TuxMake images with kernel.org toolchains as the de facto
>>     stardard,
>>
>> 2. creating a repository from scratch on git.kernel.org with
>>     independent hosting for base images,
>>
>> 3. some middle ground to be defined which would remove the risks
>>     associated with third parties without duplicating efforts.
>>
>> I feel it would be good to have more maintainers' feedback though.
>>
>> Nathan, Nicolas, Miguel - what are your thoughts on this?
> 
> To be entirely honest, I do not really have a strong opinion here. I
> generally agree with your thoughts around branded container images,
> although I do think that TuxMake's images tend to be fairly lean, so
> those easily could become the recommended images without many downsides
> aside from maybe where they are hosted and how they are maintained.
> Having a clean set of Containerfiles on git.kernel.org does not sound
> like a bad idea, especially if they would be structured in such a way
> that other entities could customize them for their use case further. I
> guess its usefulness really depends on the other stakeholders like
> KernelCI. It seems like there has to be some sort of buy in from the
> kernel.org administrators around hosting built container images
> somewhere on kernel.org though, as I don't think the regular developer
> is going to want to build images locally. Not really sure what that
> looks like.

Thank you for the feedback, it's very valuable even without a strong
opinion regarding the 3 options in my previous email.  It also tends
to confirm that the risks I mentioned aren't too critical to get
started, having some developer adoption is more important for now.

With all this in mind, I believe we can follow a step-by-step
approach to start simple and deploy more tailored solutions as and
when the need arises.  This can be seen as a bit of a Beta testing
phase until there is enough momentum to look into the longer term.

So here's what I would like to propose for the short term:

1. review current TuxMake kernel.org toolchain images[1] and
    determine what needs to be added or changed there (e.g. I don't
    see any GCC ones right now)

2. update the scripts/container tool to rely on tuxmake korg images
    with default entries in a reference user settings file

3. encourage kernel developers to go and try it out and provide
    feedback, maybe even run a small community survey

4. review the situation after a while e.g. once v7.1-rc2 is out and
    decide what steps to take next based on community priorities

I believe there aren't any blockers here so we can probably just give
it a try as an experiment and see how it goes, unless someone has
anything else to suggest.

Things like moving the Containerfiles to git.kernel.org or hosting
the images in a dedicated registry etc. should eventually emerge by
themselves if there is a real need for them.

Best wishes,
Guillaume

[1] https://hub.docker.com/u/tuxmake?search=korg

Re: Hosting first-party kernel.org container images

[Miguel Ojeda]

On Wed, Feb 25, 2026 at 3:44 PM Guillaume Tucker <gtucker@gtucker.io> wrote:
>
> Nathan, Nicolas, Miguel - what are your thoughts on this?

I agree with Nathan that `Containerfile`s on git.kernel.org sounds good.

While many kernel developers may not use them (mostly because everyone
has their own setup already), if we had a set of "base" images that
are maintained by someone trusted (including publishing them in the
usual registries for easy use), then I think a bunch of us would
actually use them for CI and possibly other tasks; just like some of
us already use the kernel.org toolchains "manually" anyway etc.

It could serve as an entry point to discover the different toolchains
and software offered by kernel.org, i.e. not just the GCC and LLVM and
LLVM+Rust toolchains, but also perhaps `b4` pre-installed and so on.
Even things like `tc-build` or the latest and greatest set of AI
context text files could be considered too.

It could also perhaps simplify a tiny bit giving quick reproducers
compared to giving the instructions for a distro base image + extra
things to install. It could also be a good thing to give newcomers
(i.e. how to set up all the development environment is a common
question) and LLMs.

I mean, all this is nothing new of course, but sometimes trust and
convenience go a long way.

I hope that helps!

Cheers,
Miguel

Re: Hosting first-party kernel.org container images

[Theodore Tso]

On Thu, Mar 19, 2026 at 12:10:19PM +0100, Miguel Ojeda wrote:
> 
> While many kernel developers may not use them (mostly because everyone
> has their own setup already), if we had a set of "base" images that
> are maintained by someone trusted (including publishing them in the
> usual registries for easy use), then I think a bunch of us would
> actually use them for CI and possibly other tasks; just like some of
> us already use the kernel.org toolchains "manually" anyway etc.

I publish test appliance images suitable for use by kvm-xfstests and
android-xfstests on www.kernel.org[1], and have for years.  A
description of how they are generated, and what I do in order to keep
the GPL complaint can be described in this slide deck[2].  In
particular, note how I am very careful to explicitly specify where the
corresponding source can be found in a README file[3].  This is
required by the GPL, so that someone can recreate the binary image
using the exact sources that were used to generate them.  I've noted
that not all container / docker images seem to follow the letter of
the law, and fortunately various groups such as the SFLC hasn't gone
after people who publish these images.... but I think it's better to
be careful about such things.

[1] https://www.kernel.org/pub/linux/kernel/people/tytso/kvm-xfstests/
[2] https://thunk.org/gce-xfstests
[3] https://www.kernel.org/pub/linux/kernel/people/tytso/kvm-xfstests/README

My kvm-xfstests images are relatively small, and I don't update them
all that often, so I haven't had any complaints from the folks who
manage kernel.org about.  So aside from the GPL compliance issues, if
you think that you might be making a large number of images, which
might be potentially large, you might want to check with them first.

Cheers,

					- Ted