[RFC PATCH 0/1] kbuild: introduce containerized builds

[RFC PATCH 0/1] kbuild: introduce containerized builds

[Guillaume Tucker]

This proposal emerged from an email discussion and a talk at Plumbers
last year:

    https://lore.kernel.org/all/affb7aff-dc9b-4263-bbd4-a7965c19ac4e@gtucker.io/

The aim is to facilitate reproducing builds for CI bots as well as
developers using containers.  It's achieved by providing a wrapper
around `make` in a separate Makefile.  Here's an illustrative example
with a kernel.org toolchain in a Docker image from tuxmake:

    $ make -f scripts/Makefile.container CONTAINER=tuxmake/korg-clang-21 LLVM=1 defconfig
    Running in docker tuxmake/korg-clang-21
      HOSTCC  scripts/basic/fixdep
      HOSTCC  scripts/kconfig/conf.o
    [...]
      HOSTCC  scripts/kconfig/util.o
      HOSTLD  scripts/kconfig/conf
    *** Default configuration is based on 'x86_64_defconfig'
    #
    # configuration written to .config
    #

    $ make -f scripts/Makefile.container CONTAINER=tuxmake/korg-clang-21 LLVM=1 -j8
    Running in docker tuxmake/korg-clang-21
    make: warning: -j8 forced in submake: resetting jobserver mode.
      SYNC    include/config/auto.conf
      GEN     arch/x86/include/generated/asm/orc_hash.h
      WRAP    arch/x86/include/generated/uapi/asm/bpf_perf_event.h
      WRAP    arch/x86/include/generated/uapi/asm/errno.h
    [...]
      LD      arch/x86/boot/setup.elf
      OBJCOPY arch/x86/boot/setup.bin
      BUILD   arch/x86/boot/bzImage
    Kernel: arch/x86/boot/bzImage is ready  (#1)

The initial idea was to add an if-else block for when $(CONTAINER)
was defined directly in the top-level Makefile but this seemed too
intrusive, hence the approach here with a separate file.  It's easy
enough to create an alias for development purposes if needed.

While the example above uses a tuxmake image, I've also started
preparing reference container images with kernel.org toolchains and
no third-party dependencies other than the base Debian distro:

    https://gitlab.com/gtucker/korg-containers

These are convenient for exercising this feature but any arbitrary
image may be used of course.

Guillaume Tucker (1):
  kbuild: add Makefile.container with CONTAINER option

 scripts/Makefile.container | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
 create mode 100644 scripts/Makefile.container

-- 
2.47.3

[RFC PATCH 1/1] kbuild: add Makefile.container with CONTAINER option

[Guillaume Tucker]

Add scripts/Makefile.container to wrap the make command in a container
using the CONTAINER= variable to specify the image name.  For example:

    make -f scripts/Makefile.container CONTAINER=korg-gcc defconfig

The container image name is entirely arbitrary and the container tool
may be Docker, Podman or any other compatible alternative specified by
the CONTAINER_COMMAND variable.  The default is set to docker for now.

Cc: Nathan Chancellor <nathan@kernel.org>
Cc: Miguel Ojeda <ojeda@kernel.org>
Link: https://lore.kernel.org/all/affb7aff-dc9b-4263-bbd4-a7965c19ac4e@gtucker.io/
Signed-off-by: Guillaume Tucker <gtucker@gtucker.io>
---
 scripts/Makefile.container | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
 create mode 100644 scripts/Makefile.container

diff --git a/scripts/Makefile.container b/scripts/Makefile.container
new file mode 100644
index 000000000000..711cf9188016
--- /dev/null
+++ b/scripts/Makefile.container
@@ -0,0 +1,15 @@
+# SPDX-License-Identifier: GPL-2.0
+
+ifeq ($(CONTAINER),)
+$(error "Usage: make CONTAINER=<container-image-name> [...]")
+endif
+CONTAINER_COMMAND ?= docker
+PHONY := __all
+
+__all:
+%:
+	@echo Running in $(CONTAINER_COMMAND) $(CONTAINER)
+	@$(CONTAINER_COMMAND) run -it -v $(PWD):/src -w /src \
+	$(CONTAINER) $(MAKE) \
+	$(subst CONTAINER=$(CONTAINER),,$(MAKEFLAGS)) \
+	$(GNUMAKEFLAGS) $(MAKECMDGOALS)
-- 
2.47.3

Re: [RFC PATCH 1/1] kbuild: add Makefile.container with CONTAINER option

[Miguel Ojeda]

On Tue, Oct 14, 2025 at 11:45 AM Guillaume Tucker <gtucker@gtucker.io> wrote:
>
> Add scripts/Makefile.container to wrap the make command in a container
> using the CONTAINER= variable to specify the image name.  For example:
>
>     make -f scripts/Makefile.container CONTAINER=korg-gcc defconfig
>
> The container image name is entirely arbitrary and the container tool
> may be Docker, Podman or any other compatible alternative specified by
> the CONTAINER_COMMAND variable.  The default is set to docker for now.

IIUC, this wraps reruns `make` inside the container, but it means
hardcoding a particular tool and path, right? (unless one sets even
more variables)

The cover letter says one can create an alias for this, but one could
also do that for the underlying call anyway, unless I am missing
something. And if we do this, then I would prefer one doesn't need to
type `-f ...`.

Put another way, for a user, what is the benefit of having this extra
way of running in a container? For instance, I could see the benefit
if different tools had different flags or it was a complicated
procedure, but I think at least `podman` shares the flags used here.

Should this instead be a document inside `Documentation/` somewhere
that explains how to do this, pitfalls, advanced options, etc. and
give example command lines for different tools?

If we do end up with `CONTAINER=`, then I think it should make it work
without having to pass `-f ...`, to make it easier. Or, even better,
like the KUnit script, we could have a script that does the right
thing and reads a config from the user, so that one can just type
something like, picking whatever tooling the user configured (e.g.
Docker vs. Podman, default image, etc.):

    scripts/container.py defconfig

Thanks!

Cheers,
Miguel

Re: [RFC PATCH 1/1] kbuild: add Makefile.container with CONTAINER option

[Onur Özkan]

On Tue, 14 Oct 2025 13:58:10 +0200
Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> wrote:

> On Tue, Oct 14, 2025 at 11:45 AM Guillaume Tucker
> <gtucker@gtucker.io> wrote:
> >
> > Add scripts/Makefile.container to wrap the make command in a
> > container using the CONTAINER= variable to specify the image name.
> > For example:
> >
> >     make -f scripts/Makefile.container CONTAINER=korg-gcc defconfig
> >
> > The container image name is entirely arbitrary and the container
> > tool may be Docker, Podman or any other compatible alternative
> > specified by the CONTAINER_COMMAND variable.  The default is set to
> > docker for now.
> 
> IIUC, this wraps reruns `make` inside the container, but it means
> hardcoding a particular tool and path, right? (unless one sets even
> more variables)
> 
> The cover letter says one can create an alias for this, but one could
> also do that for the underlying call anyway, unless I am missing
> something. And if we do this, then I would prefer one doesn't need to
> type `-f ...`.
> 
> Put another way, for a user, what is the benefit of having this extra
> way of running in a container? For instance, I could see the benefit
> if different tools had different flags or it was a complicated
> procedure, but I think at least `podman` shares the flags used here.
> 
> Should this instead be a document inside `Documentation/` somewhere
> that explains how to do this, pitfalls, advanced options, etc. and
> give example command lines for different tools?
> 
> If we do end up with `CONTAINER=`, then I think it should make it work
> without having to pass `-f ...`, to make it easier. Or, even better,
> like the KUnit script, we could have a script that does the right
> thing and reads a config from the user, so that one can just type
> something like, picking whatever tooling the user configured (e.g.
> Docker vs. Podman, default image, etc.):
> 
>     scripts/container.py defconfig
> 

I think this functionality would be better implemented as a script
(like you mentioned) rather than a Makefile. The current approach is
likely to run into several practical issues (e.g. file permission
mismatches between host and container, the need to manually remove
containers with `docker rm`, etc.) and addressing all of these
reliably in Makefile can become quite messy. Writing a python (or even
perl) script would make it much easier to maintain. Also, it can be
self-documented quite nicely with `scripts/container.py --help` command.

Regards,
Onur

> Thanks!
> 
> Cheers,
> Miguel

Re: [RFC PATCH 1/1] kbuild: add Makefile.container with CONTAINER option

[Guillaume Tucker]

Hi Miguel,

On 14/10/2025 1:58 pm, Miguel Ojeda wrote:
> On Tue, Oct 14, 2025 at 11:45 AM Guillaume Tucker <gtucker@gtucker.io> wrote:
>>
>> Add scripts/Makefile.container to wrap the make command in a container
>> using the CONTAINER= variable to specify the image name.  For example:
>>
>>      make -f scripts/Makefile.container CONTAINER=korg-gcc defconfig
>>
>> The container image name is entirely arbitrary and the container tool
>> may be Docker, Podman or any other compatible alternative specified by
>> the CONTAINER_COMMAND variable.  The default is set to docker for now.
> 
> IIUC, this wraps reruns `make` inside the container, but it means
> hardcoding a particular tool and path, right? (unless one sets even
> more variables)

This is what the CONTAINER_COMMAND variable is for.  You can easily
set it in the environment e.g. `export CONTAINER_COMMAND=podman` or
anything and be done with it.  The default is set to `docker`.  As
far as I know, there's no need for anything else than the container
command and image name.  So it's not more hard-coded than say, the
default $(CC) executable.

> The cover letter says one can create an alias for this, but one could
> also do that for the underlying call anyway, unless I am missing
> something. And if we do this, then I would prefer one doesn't need to
> type `-f ...`.

Yes having to specify the path to the Makefile is an extra hurdle.  As 
the cover letter mentions, the first patch I made was in the top-level 
Makefile[1].

The issue here is that it means that all regular `make` commands
would be going through the `else` branch within a make sub-process
and I wasn't too comfortable with this.  There's every chance that
someone, somewhere is using a particular make version or environment
that will cause subtle differences, or mess things up for someone
keeping an eye on the steps taken by a build.  I couldn't figure out
a way of doing this with zero impact except than having a separate
Makefile and let people decide to opt-in when they want containers.

Then I was only suggesting using a minimalist alias e.g.:

     alias kmake='make -f scripts/Makefile.container'

which would appear to work just like in the original patch.

Defining an alias to run a build inside a container is a lot more
fragile, here it's using standard `make` variables such as $(MAKE),
$(MAKEFLAGS) etc. to keep the commands entirely compatible.

> Put another way, for a user, what is the benefit of having this extra
> way of running in a container? For instance, I could see the benefit
> if different tools had different flags or it was a complicated
> procedure, but I think at least `podman` shares the flags used here.

The advantage here is that it's easier to define your CONTAINER
variable and run make as usual than start an interactive container by
hand with a volume and then run make inside.  It reduces the scope
for differences and makes builds more reproducible, for example you
can just share your command line and others will be using the exact
same toolchain and build environment as you.  This is also to enable
developers to easily reproduce builds from CI bots.  It's been one of
the driving principles behind tuxmake except I'm looking at it from a
neutral point of view here.  In other words, it's a step towards
increasing quality control upstream.

A related topic which was covered at Plumbers is to have first-party
container images, with Containerfiles maintained upstream to
facilitate using the kernel.org toolchains.  It's not a requirement
for this patch but both ideas enhance each other.  An upstream CI bot
using these container images would just have to share i.e. `make
CONTAINER=korg-clang:21`.

> Should this instead be a document inside `Documentation/` somewhere
> that explains how to do this, pitfalls, advanced options, etc. and
> give example command lines for different tools?

Yes, I did think of writing a documentation page alongside this patch
but eventually made the RFC with a cover letter instead to keep it
more as an open discussion.  Any solution to run containerized builds
would need to be documented, even if they're trivial to use.  I think
the Makefile approach is the most elegant one but if others aren't
convinced by it then starting with just some documentation might help
getting to the bottom of this and decide what to do next.

> If we do end up with `CONTAINER=`, then I think it should make it work
> without having to pass `-f ...`, to make it easier. Or, even better,
> like the KUnit script, we could have a script that does the right
> thing and reads a config from the user, so that one can just type
> something like, picking whatever tooling the user configured (e.g.
> Docker vs. Podman, default image, etc.):
> 
>      scripts/container.py defconfig

Right but then I think we would have to deal with the variables
handled by `make` which can be passed either via the environment or
on the command line, so that's similar to the issues with an alias.

We could do something like with Android builds (build/envsetup.sh)
with a file to source:

     . scripts/containerize
     m CONTAINER=korg-clang:21 defconfig

where `m` is just an arbitrary alias name obviously :)

Thank you for your feedback.  I can spend some time investigating
alternative approaches if they seem worthwhile.  I'd be interested to
know what others think of this too.

Thanks,
Guillaume


[1] Initial patch:

diff --git a/Makefile b/Makefile
index c6f549f6a4ae..4cbd8e040db7 100644
--- a/Makefile
+++ b/Makefile
@@ -1,4 +1,15 @@
  # SPDX-License-Identifier: GPL-2.0
+ifneq ($(CONTAINER),)
+CONTAINER_COMMAND ?= docker
+PHONY := __all
+__all:
+%:
+       @echo Running in $(CONTAINER_COMMAND) $(CONTAINER)
+       @$(CONTAINER_COMMAND) run -it -v $(PWD):/src -w /src \
+       $(CONTAINER) $(MAKE) \
+       $(subst CONTAINER=$(CONTAINER),,$(MAKEFLAGS)) \
+       $(GNUMAKEFLAGS) $(MAKECMDGOALS)
+else
  VERSION = 6
  PATCHLEVEL = 7
  SUBLEVEL = 0
@@ -2051,3 +2062,4 @@ FORCE:
  # Declare the contents of the PHONY variable as phony.  We keep that
  # information in a variable so we can use it in if_changed and friends.
  .PHONY: $(PHONY)
+endif # CONTAINER

Re: [RFC PATCH 1/1] kbuild: add Makefile.container with CONTAINER option

[Guillaume Tucker]

Hi Onur,

On 14/10/2025 4:08 pm, Onur Özkan wrote:
> On Tue, 14 Oct 2025 13:58:10 +0200
> Miguel Ojeda <miguel.ojeda.sandonis@gmail.com> wrote:
> 
>> On Tue, Oct 14, 2025 at 11:45 AM Guillaume Tucker
>> <gtucker@gtucker.io> wrote:
>>>
>>> Add scripts/Makefile.container to wrap the make command in a
>>> container using the CONTAINER= variable to specify the image name.
>>> For example:
>>>
>>>      make -f scripts/Makefile.container CONTAINER=korg-gcc defconfig
>>>
>>> The container image name is entirely arbitrary and the container
>>> tool may be Docker, Podman or any other compatible alternative
>>> specified by the CONTAINER_COMMAND variable.  The default is set to
>>> docker for now.
>>
>> IIUC, this wraps reruns `make` inside the container, but it means
>> hardcoding a particular tool and path, right? (unless one sets even
>> more variables)
>>
>> The cover letter says one can create an alias for this, but one could
>> also do that for the underlying call anyway, unless I am missing
>> something. And if we do this, then I would prefer one doesn't need to
>> type `-f ...`.
>>
>> Put another way, for a user, what is the benefit of having this extra
>> way of running in a container? For instance, I could see the benefit
>> if different tools had different flags or it was a complicated
>> procedure, but I think at least `podman` shares the flags used here.
>>
>> Should this instead be a document inside `Documentation/` somewhere
>> that explains how to do this, pitfalls, advanced options, etc. and
>> give example command lines for different tools?
>>
>> If we do end up with `CONTAINER=`, then I think it should make it work
>> without having to pass `-f ...`, to make it easier. Or, even better,
>> like the KUnit script, we could have a script that does the right
>> thing and reads a config from the user, so that one can just type
>> something like, picking whatever tooling the user configured (e.g.
>> Docker vs. Podman, default image, etc.):
>>
>>      scripts/container.py defconfig
>>
> 
> I think this functionality would be better implemented as a script
> (like you mentioned) rather than a Makefile. The current approach is
> likely to run into several practical issues (e.g. file permission
> mismatches between host and container, the need to manually remove
> containers with `docker rm`, etc.) and addressing all of these
> reliably in Makefile can become quite messy. Writing a python (or even
> perl) script would make it much easier to maintain. Also, it can be
> self-documented quite nicely with `scripts/container.py --help` command.

Our emails have crossed - OK I'll take a look into this.  I fear a
script is actually going to be more difficult to maintain and will
require additional dependencies on the host i.e. Python.  But like I
wrote in my previous email, I'm happy to consider some alternatives
and see if we can find a consensus.

The issue with file ownership can be addressed with user id mapping
in principle.  Garbage collecting containers is not something I've
looked into as it's not a new problem compared to starting containers
explicitly.  I'll keep these things in mind too when comparing
solutions.

Thanks,
Guillaume

Re: [RFC PATCH 1/1] kbuild: add Makefile.container with CONTAINER option

[Miguel Ojeda]

On Tue, Oct 14, 2025 at 4:31 PM Guillaume Tucker <gtucker@gtucker.io> wrote:
>
> This is what the CONTAINER_COMMAND variable is for.  You can easily
> set it in the environment e.g. `export CONTAINER_COMMAND=podman` or
> anything and be done with it.  The default is set to `docker`.

Yeah, that is what I mean, i.e. you have to more variables to make it
do what you want.

At that point, a user could also equally set up an alias or a quick
one liner script to call the real command with whatever setup they
need. That would be more flexible too with essentially the same
complexity on their side.

> As
> far as I know, there's no need for anything else than the container
> command and image name.  So it's not more hard-coded than say, the
> default $(CC) executable.

There is the `src` path there, no? e.g. I use upstream distro images
to test some builds.

> Then I was only suggesting using a minimalist alias e.g.:
>
>      alias kmake='make -f scripts/Makefile.container'

Yeah, what I was trying to say is that, at that point, you can just
have an alias (or a one liner script etc.) for the real command.

The user still needs to know how to use Docker/Podman/... anyway, i.e.
I would see the benefit if somehow this was the only command they
would need or if this handled some extra logic.

> hand with a volume and then run make inside.  It reduces the scope
> for differences and makes builds more reproducible, for example you
> can just share your command line and others will be using the exact
> same toolchain and build environment as you.  This is also to enable
> developers to easily reproduce builds from CI bots.  It's been one of
> the driving principles behind tuxmake except I'm looking at it from a
> neutral point of view here.  In other words, it's a step towards
> increasing quality control upstream.

Definitely, I am not saying containers are a bad idea (I use them
myself), i.e. I am only talking about what is the best way to provide
this (e.g. documentation, a script, this new file, directly in the
main `Makefile`...).

> A related topic which was covered at Plumbers is to have first-party
> container images, with Containerfiles maintained upstream to
> facilitate using the kernel.org toolchains.  It's not a requirement
> for this patch but both ideas enhance each other.

kernel.org images would be nice, indeed.

> Yes, I did think of writing a documentation page alongside this patch
> but eventually made the RFC with a cover letter instead to keep it
> more as an open discussion.  Any solution to run containerized builds
> would need to be documented, even if they're trivial to use.  I think
> the Makefile approach is the most elegant one but if others aren't
> convinced by it then starting with just some documentation might help
> getting to the bottom of this and decide what to do next.

Yeah, docs would be nice regardless of the way of wrapping it.

> Right but then I think we would have to deal with the variables
> handled by `make` which can be passed either via the environment or
> on the command line, so that's similar to the issues with an alias.

The script may be more involved, i.e. similar to the KUnit one, but
then you also gain the ability to provide more functionality/logic.

Say, for instance, the ability to test with a set of container images
that you define in the config file, to run certain things with the
built kernel (possibly in another container), and so on and so forth.

> We could do something like with Android builds (build/envsetup.sh)
> with a file to source:
>
>      . scripts/containerize
>      m CONTAINER=korg-clang:21 defconfig
>
> where `m` is just an arbitrary alias name obviously :)

Personally, for a new script, I would use a normal `--flags` like
interface, and leave the Make-like one if one wants to pass the actual
Make ones. I would also allow the user to define a default image too
in their config file too, and things like that.

In other words, the idea is that you can actually easily use it
without typing a lot, e.g.

    scripts/container b defconfig

(Possibly aliasing `scripts/container` itself to something shorter locally)

Or even more high-level operations like the "build with my set of
toolchain images", "build patch by patch this range of commits inside
the container", etc.

But I know this may be way out of scope :) After all, it gets into the
realm of essentially a lot of the custom tooling/scripts that
different subsystems build for themselves over time.

In fact, another possibility to think about is generalizing the
existing KUnit one to support containers.

> Thank you for your feedback.  I can spend some time investigating
> alternative approaches if they seem worthwhile.  I'd be interested to
> know what others think of this too.

You're welcome! I hope that helped at least.

Cc'ing David/Rae/Shuah as well for the KUnit bits.

Cheers,
Miguel