Re: [RFC] cells: a container as one kernel object

[Andrew Halaney <andrew@amutable.com>]

On Wed, Oct 21, 2026 at 01:43:23PM +0000, Josef Bacik wrote:
> On Tue, 2026-10-20 at 22:35 -0400, James Bottomley wrote:
> > On Mon, 2026-10-19 at 16:50 +0000, Josef Bacik wrote:
> > > A cell is a container as one object. A task is born in a cell and
> > > never leaves it, and nothing can enter one, so there is no setns()
> > > and no unshare().
> >
> > I can't agree with this.  If we're trying to fix existing bugs with a
> > new paradigm, it must apply to all of the common use cases and entering
> > an existing container is common: I use it for my build containers and
> > arch emulation containers, sidecars use it, debugging requires it and
> > it's part of the kubernetes acceptance testing, etc.
> 
> This one is on me, I worded it badly. There are two things we call
> entering a container, and I only want to get rid of one of them.
> 
> The first is an existing task moving in, which is setns(). That's what
> is gone. The second is starting a new task inside of a container that is
> already running, which is what docker exec and kubectl exec are for,
> they just happen to be built on the first one today. That one is there,
> it works and it has tests. Whoever holds the cellfd can spawn into the
> cell at any point while it's running, not just to get it going, and
> hands the new task whatever fds it should have, so a pty for a shell.
> Your build containers would keep the cellfd around and spawn the next
> command into it.
> 
> The reason I care about the difference is that a task that moves in
> brings the host with it: its mm, its fd table, its cwd, the binary it is
> running. That is CVE-2016-9962 (the container ptraces what runc exec
> brings in and gets at its fds), CVE-2019-5736 (the container overwrites
> the host's runc through /proc/self/exe) and CVE-2024-21626 (a leaked fd
> and a cwd on the host, again by way of runc exec). The runtimes have
> gotten very good at scrubbing themselves on the way in, but it is
> scrubbing. A task that is born in the cell starts with no mm and no
> files, so there is nothing to scrub. It also means the cell of a task
> never changes, which is what lets kill be complete and "dead" mean gone
> without any races to think about. And it answers "who may enter" without
> a uid or a capability, it's whoever holds the fd.
> 
> That said your list does poke at real holes in what I have, so to be
> straight about those:
> 
> - Spawning from the outside fails with EBUSY once the cell has enabled
>   controllers on its own cgroup, which systemd does right away. That's
>   question 7 in my mail, and it has to be fixed or the above is a hollow
>   answer for anything that runs systemd.
> 
> - If you aren't the one holding the cellfd you have no way to get one.
>   The holder can pass it over a unix socket, so a runtime can offer
>   that, but there is no "root walks up and nsenters by pid". I could see
>   a privileged lookup by id later, I would rather not start with it.
> 
> - The program you spawn has to be in the cell's tree, which is no good
>   for debugging an image without a shell in it. Letting the exec take an
>   fd would cover a static busybox or gdbserver from the host, and adding
>   a mount to a running cell from the outside would cover a whole tools
>   image. Neither is written.
> 
> - Port forwarding wants a socket in the network namespace. If the
>   runtime makes the network namespace and shares it with the cell it
>   still owns it and can still enter it. If the cell owns it then nobody
>   can once it is sealed, and that wants a helper spawned into the cell
>   or something like a socketat().
> 
> For debugging in general you don't need to be inside. The host sees the
> tasks of a cell by pid, and a capability on the host still lets you
> ptrace them, so it is a good deal nicer than a VM. If we ever close that
> off (question 3) it would have to come back as a right on the cellfd.
> 
> For sidecars and pods I think the answer is several cells and not one.
> The runtime makes the network namespace and puts it in the "net" slot of
> each cell, which works today, I added a test for it. Every container
> gets its own tree, ids and cgroup, they talk over localhost as they do
> now, and volumes can be shared. A sidecar or a debug container that
> shows up later is a new cell filled from the same fds, nothing has to
> enter anything. What you don't get is a shared pid or ipc namespace
> between them, and unix sockets between them carry bytes but not fds or
> credentials.

I'm not sure how big of a deal it really is, but I've used k8s's
shareProcessNamespace[0] in the past to do crummy things from a sidecar
against the main container... e.g. signalling it or poking at its files via
/proc/<pid>/root. Folks do that sort of thing often enough it seems.

As I read this, we're ruling out implementing support for that via:

  1. A slot to share one pidns between the cells of a pod. That's
     further complicated by the same cell rule about signals / ptrace
     iiuc.

  2. One cell for the pod that creates each container's namespaces
     itself, since CLONE_NEW* is EPERM in a cell. Spawning every
     container into one cell from the outside works, but then they all
     share the same root and cgroup.

But back to the bigger point, maybe that sort of thing is something
cells intentionally don't allow, since it opens a can of worms? Just
does present problems in adoption I guess for k8s world, but so does
idmap mounts in a lot of contexts (/me looks at various filesystems),
etc. Might be worth kind of explicitly calling out if the stance is
really strong on that front. Sort of thinking out loud here, I like
the ideas and am trying to brainstorm usecases that might need thinking
thru better.

[0]: https://kubernetes.io/docs/tasks/configure-pod-container/share-process-namespace/

Thanks,
Andrew