[QUESTION] Cgroup namespace and cgroup v2

[QUESTION] Cgroup namespace and cgroup v2

[Tom Hromatka]

I am in the process of adding automated cgroup v2 testing to
libcgroup and ran into an unsatisfactory user experience when
using cgroup v2 and cgroup namespaces.

I used lxc to create a container on a cgroup v2 system.  lxc
uses cgroup namespaces to remap /sys/fs/cgroup/mycontainer/ on
the host to /sys/fs/cgroup/ within the container.  Within the
container, I then created a cgroup - /sys/fs/cgroup/MyCgroup/ and
placed a long-running process in it.  (Note that as part of the
container creation, there are other cgroups in /sys/fs/cgroup/.)

I then used cgdelete to recursively delete the MyCgroup/ folder.
In recursive mode, cgdelete will move processes from a child
cgroup to its parent.  But this fails on a cgroup v2 system in a
cgroup namespace because the root cgroup is a non-leaf cgroup.

A couple questions/thoughts:

* As outlined above, the behavior of the "root" cgroup in a cgroup
   namespace on a v2 system differs from the behavior of the
   unnamespaced root cgroup.  At best this is inconsistent; at worst,
   this may leak information to an unethical program.  Any ideas how
   we can make the behavior more consistent for the user and
   libcgroup?

* I will likely add a flag to cgdelete to simply kill processes in
   a cgroup rather than try and move them to the parent cgroup.
   Moving processes to the parent cgroup is somewhat challenging
   even in a cgroup v1 system due to permissions, etc.

tl;dr - I can't move a process to the root cgroup within a cgroup
         v2 cgroup namespace because its "root" is a non-leaf cgroup

Thanks.

Tom

Re: [QUESTION] Cgroup namespace and cgroup v2

[Michal Koutný]

[-- Attachment #1: Type: text/plain, Size: 1559 bytes --]

Hi Tom.

On Tue, Oct 20, 2020 at 03:12:28PM -0600, Tom Hromatka <tom.hromatka@oracle.com> wrote:
> But this fails on a cgroup v2 system in a
> cgroup namespace because the root cgroup is a non-leaf cgroup.
Yes, the no internal process constraint simplifies and enables many
things for v2 controllers.

> * As outlined above, the behavior of the "root" cgroup in a cgroup
>   namespace on a v2 system differs from the behavior of the
>   unnamespaced root cgroup.  At best this is inconsistent; at worst,
>   this may leak information to an unethical program.
What information does this leak? (That it's running in a cgroup namespace?)
Note that this isn't the only discrepancy between host root cgroup and
namespaced root cgroup. The host group is simply special.

>   Any ideas how   we can make the behavior more consistent for the
> user and   libcgroup?
You can disable the controllers (via parent's cgroup.subtree_control) to
allow migration into the parent. Of course that affects also siblings of
the removed cgroup.

> * I will likely add a flag to cgdelete to simply kill processes in
>   a cgroup rather than try and move them to the parent cgroup.
>   Moving processes to the parent cgroup is somewhat challenging
>   even in a cgroup v1 system due to permissions, etc.
In general, migrations with controlled v2 cgroup are not supported, so
moving processes up and (especially) down has less sense than in v1.
Hence, refusing the delete operation on a populated cgroup (with
controllers) is IMO justified.


Michal

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [QUESTION] Cgroup namespace and cgroup v2

[Tom Hromatka]

On 10/27/20 12:26 PM, Michal Koutný wrote:
> Hi Tom.
>
> On Tue, Oct 20, 2020 at 03:12:28PM -0600, Tom Hromatka <tom.hromatka-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
>> But this fails on a cgroup v2 system in a
>> cgroup namespace because the root cgroup is a non-leaf cgroup.
> Yes, the no internal process constraint simplifies and enables many
> things for v2 controllers.
>
>> * As outlined above, the behavior of the "root" cgroup in a cgroup
>>    namespace on a v2 system differs from the behavior of the
>>    unnamespaced root cgroup.  At best this is inconsistent; at worst,
>>    this may leak information to an unethical program.
> What information does this leak? (That it's running in a cgroup namespace?)
> Note that this isn't the only discrepancy between host root cgroup and
> namespaced root cgroup. The host group is simply special.

Oops.  My wording was overly dramatic.  My in-house customers
get nervous when things differ within a container vs. on a host.

You're right, it differs, but that's an acceptable side effect
of the improved design of cgroup v2.

Would you mind sharing some of the other discrepancies?  I
would like to be prepared when we run into them as well.

Thanks!

>
>>    Any ideas how   we can make the behavior more consistent for the
>> user and   libcgroup?
> You can disable the controllers (via parent's cgroup.subtree_control) to
> allow migration into the parent. Of course that affects also siblings of
> the removed cgroup.

Good call.  I didn't think of that.

>
>> * I will likely add a flag to cgdelete to simply kill processes in
>>    a cgroup rather than try and move them to the parent cgroup.
>>    Moving processes to the parent cgroup is somewhat challenging
>>    even in a cgroup v1 system due to permissions, etc.
> In general, migrations with controlled v2 cgroup are not supported, so
> moving processes up and (especially) down has less sense than in v1.
> Hence, refusing the delete operation on a populated cgroup (with
> controllers) is IMO justified.

That makes a lot of sense.  I think I will need to spend
time with the database team and others.  Cgroup v2 is simply
different enough that we'll need to rethink some of the
decisions that were made for cgroup v1.

Thanks so much for the help.

Tom

>
>
> Michal

Re: [QUESTION] Cgroup namespace and cgroup v2

[Michal Koutný]

[-- Attachment #1: Type: text/plain, Size: 452 bytes --]

On Fri, Oct 30, 2020 at 07:11:20AM -0600, Tom Hromatka <tom.hromatka@oracle.com> wrote:
> Would you mind sharing some of the other discrepancies?  I
> would like to be prepared when we run into them as well.
Search for CFTYPE_NOT_ON_ROOT flag (that was on my mind above). It
causes a visible difference between host and container (OTOH, you won't
be able to write into such files typically, so that's effectively equal
to the host).

Michal


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [QUESTION] Cgroup namespace and cgroup v2

[Tom Hromatka]

On 11/2/20 7:09 AM, Michal Koutný wrote:
> On Fri, Oct 30, 2020 at 07:11:20AM -0600, Tom Hromatka <tom.hromatka-QHcLZuEGTsvQT0dZR+AlfA@public.gmane.org> wrote:
>> Would you mind sharing some of the other discrepancies?  I
>> would like to be prepared when we run into them as well.
> Search for CFTYPE_NOT_ON_ROOT flag (that was on my mind above). It
> causes a visible difference between host and container (OTOH, you won't
> be able to write into such files typically, so that's effectively equal
> to the host).

Awesome!  Thanks.

Tom

>
> Michal
>