How to use cgroups within containers?

How to use cgroups within containers?

[Richard Weinberger]

Dear systemd and container folks,

at Plumbers the question raised how to provide cgroups to a systemd that lives
in a container (with user namespaces).
Due to the GDL train strikes I had to leave very soon and had no chance to
talk to you in person.

Was a solution proposed?
All I want to know is how to provide cgroups in a sane and secure way
to systemd. :-)

-- 
Thanks,
//richard

Re: [systemd-devel] How to use cgroups within containers?

[Cameron Norman]

On Fri, Oct 17, 2014 at 2:37 PM, Richard Weinberger
<richard.weinberger-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> ...fixing LXC devel mailinglist... :-\
>
> On Fri, Oct 17, 2014 at 11:35 PM, Richard Weinberger
> <richard.weinberger-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>> Dear systemd and container folks,
>>
>> at Plumbers the question raised how to provide cgroups to a systemd that lives
>> in a container (with user namespaces).
>> Due to the GDL train strikes I had to leave very soon and had no chance to
>> talk to you in person.
>>
>> Was a solution proposed?
>> All I want to know is how to provide cgroups in a sane and secure way
>> to systemd. :-)

I am not at all an expert on systemd's cgroups, however I do know the
basic design of cgmanager. cgmanager provides cgroups to containers
via a "cgproxy" service that relays messages to the main cgmanager
service. You can read more about it here:
https://github.com/cgmanager/cgmanager/blob/master/README. Perhaps
systemd can use a similar model.

Cheers,
--
Cameron

Re: How to use cgroups within containers?

[Serge E. Hallyn]

Quoting Richard Weinberger (richard.weinberger-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org):
> ...fixing LXC devel mailinglist... :-\
> 
> On Fri, Oct 17, 2014 at 11:35 PM, Richard Weinberger
> <richard.weinberger-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> > Dear systemd and container folks,
> >
> > at Plumbers the question raised how to provide cgroups to a systemd that lives
> > in a container (with user namespaces).
> > Due to the GDL train strikes I had to leave very soon and had no chance to
> > talk to you in person.
> >
> > Was a solution proposed?
> > All I want to know is how to provide cgroups in a sane and secure way
> > to systemd. :-)

My takeaway from the discussion was that there was general agreement that
the cgroup namespaces patches posted by Aditya Kali would solve the
problem.

-serge

Re: How to use cgroups within containers?

[Richard Weinberger]

...fixing LXC devel mailinglist... :-\

On Fri, Oct 17, 2014 at 11:35 PM, Richard Weinberger
<richard.weinberger-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Dear systemd and container folks,
>
> at Plumbers the question raised how to provide cgroups to a systemd that lives
> in a container (with user namespaces).
> Due to the GDL train strikes I had to leave very soon and had no chance to
> talk to you in person.
>
> Was a solution proposed?
> All I want to know is how to provide cgroups in a sane and secure way
> to systemd. :-)
>
> --
> Thanks,
> //richard



-- 
Thanks,
//richard

Re: [systemd-devel] How to use cgroups within containers?

[Lennart Poettering]

On Fri, 17.10.14 23:35, Richard Weinberger (richard.weinberger-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org) wrote:

> Dear systemd and container folks,
> 
> at Plumbers the question raised how to provide cgroups to a systemd that lives
> in a container (with user namespaces).
> Due to the GDL train strikes I had to leave very soon and had no chance to
> talk to you in person.
> 
> Was a solution proposed?
> All I want to know is how to provide cgroups in a sane and secure way
> to systemd. :-)

The cgroups setup systemd requires to be able to run cleanly without
changes in a container is documented here:

http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/

You have to mount the full cgroupfs hierarchies into the containers,
so that /proc/$PID/cgroup makes sense inside the containers (that file
lists absolute paths...). They can be mounted read-only up to the
container's root, but further down they need to be writable to the
container, so that systemd inside the container can do its job.

If a container manager does not mount them systemd will mount them on
its own.

Lennart

-- 
Lennart Poettering, Red Hat

Re: [systemd-devel] How to use cgroups within containers?

[Richard Weinberger]

Am 20.10.2014 um 18:24 schrieb Lennart Poettering:
> On Fri, 17.10.14 23:35, Richard Weinberger (richard.weinberger-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org) wrote:
> 
>> Dear systemd and container folks,
>>
>> at Plumbers the question raised how to provide cgroups to a systemd that lives
>> in a container (with user namespaces).
>> Due to the GDL train strikes I had to leave very soon and had no chance to
>> talk to you in person.
>>
>> Was a solution proposed?
>> All I want to know is how to provide cgroups in a sane and secure way
>> to systemd. :-)
> 
> The cgroups setup systemd requires to be able to run cleanly without
> changes in a container is documented here:
> 
> http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/
> 
> You have to mount the full cgroupfs hierarchies into the containers,
> so that /proc/$PID/cgroup makes sense inside the containers (that file
> lists absolute paths...). They can be mounted read-only up to the
> container's root, but further down they need to be writable to the
> container, so that systemd inside the container can do its job.

And what solution do you propose?
Will cgroup namespaces make systemd finally happy?

Thanks,
//richard

Re: [systemd-devel] How to use cgroups within containers?

[Lennart Poettering]

On Mon, 20.10.14 18:49, Richard Weinberger (richard-/L3Ra7n9ekc@public.gmane.org) wrote:

> Am 20.10.2014 um 18:24 schrieb Lennart Poettering:
> > On Fri, 17.10.14 23:35, Richard Weinberger (richard.weinberger-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org) wrote:
> > 
> >> Dear systemd and container folks,
> >>
> >> at Plumbers the question raised how to provide cgroups to a systemd that lives
> >> in a container (with user namespaces).
> >> Due to the GDL train strikes I had to leave very soon and had no chance to
> >> talk to you in person.
> >>
> >> Was a solution proposed?
> >> All I want to know is how to provide cgroups in a sane and secure way
> >> to systemd. :-)
> > 
> > The cgroups setup systemd requires to be able to run cleanly without
> > changes in a container is documented here:
> > 
> > http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/
> > 
> > You have to mount the full cgroupfs hierarchies into the containers,
> > so that /proc/$PID/cgroup makes sense inside the containers (that file
> > lists absolute paths...). They can be mounted read-only up to the
> > container's root, but further down they need to be writable to the
> > container, so that systemd inside the container can do its job.
> 
> And what solution do you propose?

Solution? For what problem precisely?

> Will cgroup namespaces make systemd finally happy?

I have no idea about cgroup namespaces and what they entail.

systemd is quite happy already, if you follow the guidelines for
container managers we put together...

Lennart

-- 
Lennart Poettering, Red Hat

Re: [systemd-devel] How to use cgroups within containers?

[Richard Weinberger]

Am 20.10.2014 um 18:51 schrieb Lennart Poettering:
> On Mon, 20.10.14 18:49, Richard Weinberger (richard-/L3Ra7n9ekc@public.gmane.org) wrote:
> 
>> Am 20.10.2014 um 18:24 schrieb Lennart Poettering:
>>> On Fri, 17.10.14 23:35, Richard Weinberger (richard.weinberger-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org) wrote:
>>>
>>>> Dear systemd and container folks,
>>>>
>>>> at Plumbers the question raised how to provide cgroups to a systemd that lives
>>>> in a container (with user namespaces).
>>>> Due to the GDL train strikes I had to leave very soon and had no chance to
>>>> talk to you in person.
>>>>
>>>> Was a solution proposed?
>>>> All I want to know is how to provide cgroups in a sane and secure way
>>>> to systemd. :-)
>>>
>>> The cgroups setup systemd requires to be able to run cleanly without
>>> changes in a container is documented here:
>>>
>>> http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/
>>>
>>> You have to mount the full cgroupfs hierarchies into the containers,
>>> so that /proc/$PID/cgroup makes sense inside the containers (that file
>>> lists absolute paths...). They can be mounted read-only up to the
>>> container's root, but further down they need to be writable to the
>>> container, so that systemd inside the container can do its job.
>>
>> And what solution do you propose?
> 
> Solution? For what problem precisely?

Running systemd inside Linux container (including user namespaces). :-)

>> Will cgroup namespaces make systemd finally happy?
> 
> I have no idea about cgroup namespaces and what they entail.
> 
> systemd is quite happy already, if you follow the guidelines for
> container managers we put together...

Have you ever used systemd inside a container?
Say, LXC or libvirt-lxc...

Thanks,
//richard

Re: [systemd-devel] How to use cgroups within containers?

[Lennart Poettering]

On Mon, 20.10.14 18:55, Richard Weinberger (richard-/L3Ra7n9ekc@public.gmane.org) wrote:

> Am 20.10.2014 um 18:51 schrieb Lennart Poettering:
> > On Mon, 20.10.14 18:49, Richard Weinberger (richard-/L3Ra7n9ekc@public.gmane.org) wrote:
> > 
> >> Am 20.10.2014 um 18:24 schrieb Lennart Poettering:
> >>> On Fri, 17.10.14 23:35, Richard Weinberger (richard.weinberger-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org) wrote:
> >>>
> >>>> Dear systemd and container folks,
> >>>>
> >>>> at Plumbers the question raised how to provide cgroups to a systemd that lives
> >>>> in a container (with user namespaces).
> >>>> Due to the GDL train strikes I had to leave very soon and had no chance to
> >>>> talk to you in person.
> >>>>
> >>>> Was a solution proposed?
> >>>> All I want to know is how to provide cgroups in a sane and secure way
> >>>> to systemd. :-)
> >>>
> >>> The cgroups setup systemd requires to be able to run cleanly without
> >>> changes in a container is documented here:
> >>>
> >>> http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/
> >>>
> >>> You have to mount the full cgroupfs hierarchies into the containers,
> >>> so that /proc/$PID/cgroup makes sense inside the containers (that file
> >>> lists absolute paths...). They can be mounted read-only up to the
> >>> container's root, but further down they need to be writable to the
> >>> container, so that systemd inside the container can do its job.
> >>
> >> And what solution do you propose?
> > 
> > Solution? For what problem precisely?
> 
> Running systemd inside Linux container (including user namespaces). :-)
> 
> >> Will cgroup namespaces make systemd finally happy?
> > 
> > I have no idea about cgroup namespaces and what they entail.
> > 
> > systemd is quite happy already, if you follow the guidelines for
> > container managers we put together...
> 
> Have you ever used systemd inside a container?
> Say, LXC or libvirt-lxc...

Have you read the link I posted?

Yes, I test systemd inside containers. Daily. Actually it's my primary
way of testing systemd, since it is extremely quick and allows me to
attach from the host with debugging tools...

As long as you follow the suggestions in the document I linked systemd
will work without modifications in container managers. At least
libvirt-lxc and nspawn follows these suggestions, not sure about the
other container managers.

Also read:

http://www.freedesktop.org/wiki/Software/systemd/writing-vm-managers/

We have documented this all so nicely, I can only recommend to
actually take the time to read this. Thanks!

Lennart

-- 
Lennart Poettering, Red Hat

Re: [systemd-devel] How to use cgroups within containers?

[Richard Weinberger]

Am 20.10.2014 um 19:04 schrieb Lennart Poettering:
> On Mon, 20.10.14 18:55, Richard Weinberger (richard-/L3Ra7n9ekc@public.gmane.org) wrote:
> 
>> Am 20.10.2014 um 18:51 schrieb Lennart Poettering:
>>> On Mon, 20.10.14 18:49, Richard Weinberger (richard-/L3Ra7n9ekc@public.gmane.org) wrote:
>>>
>>>> Am 20.10.2014 um 18:24 schrieb Lennart Poettering:
>>>>> On Fri, 17.10.14 23:35, Richard Weinberger (richard.weinberger@gmail.com) wrote:
>>>>>
>>>>>> Dear systemd and container folks,
>>>>>>
>>>>>> at Plumbers the question raised how to provide cgroups to a systemd that lives
>>>>>> in a container (with user namespaces).
>>>>>> Due to the GDL train strikes I had to leave very soon and had no chance to
>>>>>> talk to you in person.
>>>>>>
>>>>>> Was a solution proposed?
>>>>>> All I want to know is how to provide cgroups in a sane and secure way
>>>>>> to systemd. :-)
>>>>>
>>>>> The cgroups setup systemd requires to be able to run cleanly without
>>>>> changes in a container is documented here:
>>>>>
>>>>> http://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/
>>>>>
>>>>> You have to mount the full cgroupfs hierarchies into the containers,
>>>>> so that /proc/$PID/cgroup makes sense inside the containers (that file
>>>>> lists absolute paths...). They can be mounted read-only up to the
>>>>> container's root, but further down they need to be writable to the
>>>>> container, so that systemd inside the container can do its job.
>>>>
>>>> And what solution do you propose?
>>>
>>> Solution? For what problem precisely?
>>
>> Running systemd inside Linux container (including user namespaces). :-)
>>
>>>> Will cgroup namespaces make systemd finally happy?
>>>
>>> I have no idea about cgroup namespaces and what they entail.
>>>
>>> systemd is quite happy already, if you follow the guidelines for
>>> container managers we put together...
>>
>> Have you ever used systemd inside a container?
>> Say, LXC or libvirt-lxc...
> 
> Have you read the link I posted?

Sure, I've also been in the room in Düsseldorf while you've read it in front of us.

> Yes, I test systemd inside containers. Daily. Actually it's my primary
> way of testing systemd, since it is extremely quick and allows me to
> attach from the host with debugging tools...
> 
> As long as you follow the suggestions in the document I linked systemd
> will work without modifications in container managers. At least
> libvirt-lxc and nspawn follows these suggestions, not sure about the
> other container managers.

If I read the source of nspwan correctly, it does not use user namespaces.
libvirt-lxc is currently not sure how to support systemd. So far it
bind mounts only the machine specific part of cgroups into the container.
Which is not really nice but better than exposing the whole hierarchy into
the container.
This is why I was asking for cgroup namespaces...

> Also read:
> 
> http://www.freedesktop.org/wiki/Software/systemd/writing-vm-managers/
> 
> We have documented this all so nicely, I can only recommend to
> actually take the time to read this. Thanks!

Thanks a lot!
//richard

Re: [systemd-devel] How to use cgroups within containers?

[Lennart Poettering]

On Mon, 20.10.14 19:16, Richard Weinberger (richard-/L3Ra7n9ekc@public.gmane.org) wrote:

> > Have you read the link I posted?
> 
> Sure, I've also been in the room in Düsseldorf while you've read it
> in front of us.

Not that I changed it since then... ;-)

> > Yes, I test systemd inside containers. Daily. Actually it's my primary
> > way of testing systemd, since it is extremely quick and allows me to
> > attach from the host with debugging tools...
> > 
> > As long as you follow the suggestions in the document I linked systemd
> > will work without modifications in container managers. At least
> > libvirt-lxc and nspawn follows these suggestions, not sure about the
> > other container managers.
> 
> If I read the source of nspwan correctly, it does not use user
> namespaces.

Ah, this is about user namespaces? No I have not played around with
them so far. Sorry.

> libvirt-lxc is currently not sure how to support systemd. So far it
> bind mounts only the machine specific part of cgroups into the container.
> Which is not really nice but better than exposing the whole hierarchy into
> the container.

It really should also bind mount the upper parts, but possibly mark
them read-only (which nspawn currently doesn't do).

Thanks,

Lennart

-- 
Lennart Poettering, Red Hat

Re: [systemd-devel] How to use cgroups within containers?

[Richard Weinberger]

Am 20.10.2014 um 19:27 schrieb Lennart Poettering:
> On Mon, 20.10.14 19:16, Richard Weinberger (richard-/L3Ra7n9ekc@public.gmane.org) wrote:
> 
>>> Have you read the link I posted?
>>
>> Sure, I've also been in the room in Düsseldorf while you've read it
>> in front of us.
> 
> Not that I changed it since then... ;-)
> 
>>> Yes, I test systemd inside containers. Daily. Actually it's my primary
>>> way of testing systemd, since it is extremely quick and allows me to
>>> attach from the host with debugging tools...
>>>
>>> As long as you follow the suggestions in the document I linked systemd
>>> will work without modifications in container managers. At least
>>> libvirt-lxc and nspawn follows these suggestions, not sure about the
>>> other container managers.
>>
>> If I read the source of nspwan correctly, it does not use user
>> namespaces.
> 
> Ah, this is about user namespaces? No I have not played around with
> them so far. Sorry.

Yep. Please have a look at them. There are some pitfalls.

>> libvirt-lxc is currently not sure how to support systemd. So far it
>> bind mounts only the machine specific part of cgroups into the container.
>> Which is not really nice but better than exposing the whole hierarchy into
>> the container.
> 
> It really should also bind mount the upper parts, but possibly mark
> them read-only (which nspawn currently doesn't do).

Okay. Or maybe cgroup namespaces will help.
Let's find out. :)

Thanks,
//richard