Re: [RFC] cells: a container as one kernel object

[Jeremy Bingham <jbingham@gmail.com>]

> Hello,
> 
> This is the thing Christian and I and a few others have been going back
> and forth on, a container as one kernel object instead of a pile of
> namespaces that userspace holds together. I have a barebones
> implementation of it. It's 62 patches on top of the 13 patch cred series
> [1], and I don't think anybody wants that in their inbox for what is at
> this point a design discussion, so the code is in a branch, and what's
> in this email is the pitch, the things I would like opinions on, and the
> design document from the branch in full at the bottom so you can quote
> it and yell at specific lines.
> 
>   https://github.com/josefbacik/linux/commits/cells/
>   git fetch https://github.com/josefbacik/linux.git cells
> 
> That's 6cbfd1952c30 as I write this, based on vfs.all. "Cell" is a
> placeholder name, I don't love it, suggestions welcome.

Coincidentally, I've been thinking a lot about some issues adjacent to
this, albeit not exactly the same. What I was thinking about what would
be required to get something more akin to z/Arch LPARs on amd64 or
arm64 machines vs. your standard Linux container, which then morphed
into wondering about what might be possible in the space between
standard containers and a fullblown VM under a hypervisor. There are the
microVM options out there, which look useful, but still weren't *quite*
what I was thinking about.

Obviously we can't get a direct analog of a z/Arch LPAR on non-z/Machine
computers, but this "cell" idea might fill the kind of role I was
envisioning. I'll be grabbing that tree and taking a look. I'll be
leaving some comments below.

In regards to the name for these objects, how about "stylite"[1]. A
stylite was one of those monks who would spend decades sitting on top of
pillars, isolated from the world. The only ways to communicate with them
were to get a ladder and climb close enough to talk, or send things up
and down in a bucket tied to a rope. I think that describes what we're
discussing here pretty well.

[1] https://en.wikipedia.org/wiki/Stylite

> The pitch
> ---------
> 
> A container isn't a thing in the kernel. It's eight kinds of namespace,
> a cgroup, a seccomp filter and a pile of conventions that the runtime
> holds together. The kernel can't name it, can't kill it in one go, and
> can't tell you what's in it. Privilege inside of it hangs off of uids
> and user namespaces, and a task can be half way between two sets of
> namespaces at any given moment, which is where a depressing number of
> our bugs come from.
> 
> A cell is a container as one object. A task is born in a cell and never
> leaves it, and nothing can enter one, so there is no setns() and no
> unshare(). You build it from the outside by filling named slots from
> file descriptors you already hold, you seal it, and then you spawn tasks
> into it. Holding the cellfd is the authority to do all of that, there is
> no uid or capability check on it, and making one takes no privilege. The
> cell lives exactly as long as somebody holds the cellfd, when the last
> one goes the cell is killed.

I find the idea of a single object container very appealing. There are
a couple of things that I thought of that might make this easier to work
with, although they might be more appropriate for userspace tools:

* Channels analogous to golang channels one uses to communicate with
  goroutines. Obviously, these ultimately end up being filehandles, but
  providing a ready interface with read, write, and read/write
  channels to interact with the cell might be clearer to users than
  "grab some filehandles and toss them at the cell".
* Provide a convenient way to attach a network interface. I would think
  that like standard containers, many cells would have services that
  communicate over the network running inside.

> Spawning is the other half. clone3(CLONE_EMBRYO) gives you a pidfd on a
> task that exists but has never run anything, you set it up with
> task_configure() the way you would set up a mount with fsconfig(), and
> then you start it and it execs. It starts with no mm and no files, so
> what it has is what you gave it. I think this is useful without cells at
> all and would like to send it on its own eventually.
> 
> A few rules fall out of wanting the boundary to actually mean something.
> The kernel allocates the kuids for a cell out of a reserved window, the
> builder never names one, and those ids can never be stored on a
> filesystem the cell doesn't own. A uid means nothing across cells, not
> an equal one and not 0. Files impart no privilege, every task has
> no_new_privs, and set-id bits get dropped on anything a cell creates,
> chmods or opens for writing. Unix sockets carry bytes between cells and
> nothing else, no SCM_RIGHTS and no peer credentials. The mount tree is
> built from scratch, nothing in it hides anything, and nothing propagates
> into it.
> 
> What I would like opinions on
> -----------------------------
> 
> 1. The shape. One object, born in it and never leaves, nothing enters,
>    built from the outside out of fds, authority is the fd. If you think
>    this is the wrong shape then the rest doesn't matter, so start here.

The basic idea for the shape is sound, I think.

> 2. The id window. Blocks of 65536 kuids come out of the upper half of
>    the id space, and "nobody else holds an id of this block" is what
>    replaces the privilege check for mapping a range, so it has to be
>    enforced. With CONFIG_CELLS on the host can't chown a file to an id
>    of 2^31 or above anymore, because the check has to go by value. This
>    is the one I expect to get yelled at for, and I'm open to better
>    ideas.

Unfortunately, I think this could easily become a problem. Maybe instead
of slicing up the id space with its upper half, you allocate a separate
id space?

> 3. Capabilities. A capability held in an ancestor user namespace still
>    applies to what a cell owns, so root on the host can still signal and
>    ptrace by pid. In the checks that let one task act on another I
>    closed the branch that goes by a uid, euid 0 included, and left the
>    capability branch alone. I would like the boundary to be hard in both
>    directions, you hold the cellfd or you have nothing, but that is a
>    much bigger conversation. How far do we want to go?
> 
> 4. MNT_LOCKED (Christian, this one is for you). The seal refuses a tree
>    with a locked mount in it, since a locked mount hides what it covers.
>    But for a builder that works in a user namespace of its own, the
>    kernel locks the tree itself onto the empty root of a namespace made
>    with FSMOUNT_NAMESPACE or OPEN_TREE_NAMESPACE, so an unprivileged
>    builder could never seal anything. I exempted exactly that mount, on
>    the grounds that there is nothing under it to hide. Is that right, or
>    should it not be locked there to begin with?
> 
> 5. seccomp. An embryo inherits the filters of whoever made it and they
>    apply to everything the new program does, but they are not consulted
>    for what TASK_CMD_START does on the way there, because the exec isn't
>    a system call of the embryo. So a filter that denies execve() and
>    execveat() to keep a task from running another program has to learn
>    about task_configure() as well, same as it once had to learn about
>    execveat(). This is only about tasks of the host, a task of a cell
>    can't spawn at all. Documented for now. Good enough, or do we run the
>    filters at the start?
> 
> 6. Lifetime. A cell dies with its last cellfd, there is no opt out and
>    no way to look one up by id. An earlier version had this as a flag,
>    and without it you ended up with a running cell that nobody held, so
>    nothing could spawn into it or wait for it, and killing it meant
>    going around it by pid or cgroup. You can pass the fd to a
>    supervisor. I would rather add the foot gun later if somebody really
>    needs it.
> 
> 7. The budget. Spawning from the outside fails with EBUSY once the cell
>    enables controllers on its own cgroup, which systemd does right away.
>    It needs a leaf below the budget for spawns to land in. Does the
>    kernel make that leaf, or is it a convention for the builder? Also a
>    cell with a block of ids can't mkdir in cgroupfs until cgroupfs can
>    be idmapped.
> 
> 8. procfs, sysfs, devtmpfs and cgroupfs. There are no per-cell instances
>    yet, the builder assembles those with what exists today and can get
>    it wrong the same way a runtime can today. Where I want to go is each
>    of the four taking the cell as a mount parameter and showing only
>    what the cell holds. That's a lot of work and I wanted this out
>    first, but I would like to know if anybody hates the direction.

I think that per-cell procfs/sysfs/etc. mounts is ultimately the way to
go. More revealing procfs/sysfs/etc. mounts would be acceptable in the
initial stages, but it ought to be dealt with before it's "done".

> 9. The API. Five system calls, cell_create(), cell_configure(),
>    cell_info(), cell_open() and task_configure(), no ioctls, with string
>    keys in the style of fsconfig(). The right to spawn is a slot as
>    well, reserved and always empty for now, which is why cells don't
>    nest. We do want unprivileged things to be able to build smaller
>    cells out of what they hold eventually, but it touches ids, cgroups
>    and mounts all at once and I left it out.
> 

I'm very interested in this concept, given that I've already been
thinking about ideas like this, so I'll pull these branches and
experiment with them as time permits. I'll be following this with
interest, and if I come up with anything worthwhile I'll report back.

-j