Cgroups v2 thread mode "domain invalid" state

Cgroups v2 thread mode "domain invalid" state

[Michael Kerrisk (man-pages)]

Hello Tejun,

At some point, I'd like to add some details about cgroups v2 "thread
mode" to the cgroup(7) manual page. I think I understand most of the
details by now, but there's one question I still can't work out an
answer to: why does the "domain invalid" cgroup type exist?

I did find the July 2017 mail thread
"[PATCH 5/6] cgroup: implement cgroup v2 thread support"
(https://marc.info/?t=150025735500001&r=1&w=2)
where this cgroup type was discussed, but I must admit that I still
don't get it. Instead of converting cgroups under a threaded subtree
to "domain invalid" type, why can't they just be turned straight into
"threaded" type?

I guess another way of asking this question is: what possibilities
does manually switching each "domain invalid" cgroup to "threaded"
allow that wouldn't be allowed by immediately making all of the
cgroups in a threaded subtree "threaded"?

Cheers,

Michael

-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: Cgroups v2 thread mode "domain invalid" state

[Tejun Heo]

Hello,

On Wed, Dec 20, 2017 at 09:32:04PM +0100, Michael Kerrisk (man-pages) wrote:
> I did find the July 2017 mail thread
> "[PATCH 5/6] cgroup: implement cgroup v2 thread support"
> (https://marc.info/?t=150025735500001&r=1&w=2)
> where this cgroup type was discussed, but I must admit that I still
> don't get it. Instead of converting cgroups under a threaded subtree
> to "domain invalid" type, why can't they just be turned straight into
> "threaded" type?
> 
> I guess another way of asking this question is: what possibilities
> does manually switching each "domain invalid" cgroup to "threaded"
> allow that wouldn't be allowed by immediately making all of the
> cgroups in a threaded subtree "threaded"?

IIRC, there were two reasons.  (Writing from memory, so I might not be
completely accurate.)

* Consistency w/ the cgroups right under the root cgroup.  Because
  they can be both domains and threadroots, we can't switch the
  children over to thread mode automatically.  Doing that for cgroups
  further down in the hierarchy would be really inconsistent.

* Possible extension to threadmode.  If we figure out how to do mixed
  mode further down in the hierarchy (and if there are actual use
  cases which require that), automatically switching would be really
  confusing.

The invalid state while not the most convenient is straight forward
(only the operations which are explicitly asked are performed) and
keeps the door open for future changes.

Thanks.

-- 
tejun

Re: Cgroups v2 thread mode "domain invalid" state

[Michael Kerrisk (man-pages)]

Hi Tejun,

On 22 December 2017 at 01:15, Tejun Heo <tj@kernel.org> wrote:
> Hello,
>
> On Wed, Dec 20, 2017 at 09:32:04PM +0100, Michael Kerrisk (man-pages) wrote:
>> I did find the July 2017 mail thread
>> "[PATCH 5/6] cgroup: implement cgroup v2 thread support"
>> (https://marc.info/?t=150025735500001&r=1&w=2)
>> where this cgroup type was discussed, but I must admit that I still
>> don't get it. Instead of converting cgroups under a threaded subtree
>> to "domain invalid" type, why can't they just be turned straight into
>> "threaded" type?
>>
>> I guess another way of asking this question is: what possibilities
>> does manually switching each "domain invalid" cgroup to "threaded"
>> allow that wouldn't be allowed by immediately making all of the
>> cgroups in a threaded subtree "threaded"?
>
> IIRC, there were two reasons.  (Writing from memory, so I might not be
> completely accurate.)
>
> * Consistency w/ the cgroups right under the root cgroup.  Because
>   they can be both domains and threadroots, we can't switch the
>   children over to thread mode automatically.  Doing that for cgroups
>   further down in the hierarchy would be really inconsistent.

I don't get the above. I mean, if we have the following hierarchy,
where R is the cgoup v2 mount point:

R---A(d)------B(d)
 \       \----C(d)
  \
   \
    X(d)------Y(d)

And we write "threaded" to A/cgroup.type, then we end up with:

R---A(t)------B(di)
 \       \----C(di)
  \
   \
    X(d)------Y(d)

(To be clear: "(d)" means "domain", "(di)" means "domain invalid", and
"(t)" means "threaded".)

But equally (in another world), that operation could have resulted in:

R---A(t)------B(t)
 \       \----C(t)
  \
   \
    X(d)------Y(d)

It's not clear to me how "Doing that for cgroups further down in the
hierarchy would be really inconsistent", since in the current
implementation, those same thread groups are converted to "domain
invalid" type.  What am I missing?

But, regarding the next point:

> * Possible extension to threadmode.  If we figure out how to do mixed
>   mode further down in the hierarchy (and if there are actual use
>   cases which require that), automatically switching would be really
>   confusing.
>
> The invalid state while not the most convenient is straight forward
> (only the operations which are explicitly asked are performed) and
> keeps the door open for future changes.

The "allowing for future" extensions idea makes some sense to me.

One other point that occurred to me after I wrote my email yesterday
was that if the threaded root reverts from being "domain threaded" to
"domain" (because it no longer has "threaded" children AND either it
has no member processes or it has no threaded controllers enabled),
then the "domain invalid" descendants revert to type "domain". I'm not
sure whether that detail also provides some rationale as to why all
descendants of the threaded root cgroup aren't automatically converted
to type "threaded". Any thoughts about that?

Thanks,

Michael

-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/

Re: Cgroups v2 thread mode "domain invalid" state

[Tejun Heo]

Hello, Michael.

On Fri, Dec 22, 2017 at 10:11:00AM +0100, Michael Kerrisk (man-pages) wrote:
> I don't get the above. I mean, if we have the following hierarchy,
> where R is the cgoup v2 mount point:

You're right, I'm probably confused with an earlier variant.  I might
still be forgetting something in this area.  I'll ping back if I can
recall something.

> > * Possible extension to threadmode.  If we figure out how to do mixed
> >   mode further down in the hierarchy (and if there are actual use
> >   cases which require that), automatically switching would be really
> >   confusing.
> >
> > The invalid state while not the most convenient is straight forward
> > (only the operations which are explicitly asked are performed) and
> > keeps the door open for future changes.
> 
> The "allowing for future" extensions idea makes some sense to me.
> 
> One other point that occurred to me after I wrote my email yesterday
> was that if the threaded root reverts from being "domain threaded" to
> "domain" (because it no longer has "threaded" children AND either it
> has no member processes or it has no threaded controllers enabled),
> then the "domain invalid" descendants revert to type "domain". I'm not
> sure whether that detail also provides some rationale as to why all
> descendants of the threaded root cgroup aren't automatically converted
> to type "threaded". Any thoughts about that?

Currently, thread mode implementation doesn't allow reverting back to
domain.  The problem there is that it's impossible to tell which
portions of the domain consumptions that are accounted to the threaded
domain (the parent of threaded subtree) belong to the cgroup which is
trying to revert to domain.

But assuming we in the future allow reverting back to domain,
scenarios similar to what you suggested can become problematic.  All
operations always behaving recursively will probably be the only sane
solution but that blocks some possibilities for future changes.

Thanks.

-- 
tejun

Re: Cgroups v2 thread mode "domain invalid" state

[Michael Kerrisk (man-pages)]

Hello Tejun,

On 01/08/2018 05:08 PM, Tejun Heo wrote:
> Hello, Michael.
> 
> On Fri, Dec 22, 2017 at 10:11:00AM +0100, Michael Kerrisk (man-pages) wrote:
>> I don't get the above. I mean, if we have the following hierarchy,
>> where R is the cgoup v2 mount point:
> 
> You're right, I'm probably confused with an earlier variant.  I might
> still be forgetting something in this area.  I'll ping back if I can
> recall something.

Okay -- if you do think of something, I'd be very happy to add it to
the man page.

>>> * Possible extension to threadmode.  If we figure out how to do mixed
>>>   mode further down in the hierarchy (and if there are actual use
>>>   cases which require that), automatically switching would be really
>>>   confusing.
>>>
>>> The invalid state while not the most convenient is straight forward
>>> (only the operations which are explicitly asked are performed) and
>>> keeps the door open for future changes.
>>
>> The "allowing for future" extensions idea makes some sense to me.
>>
>> One other point that occurred to me after I wrote my email yesterday
>> was that if the threaded root reverts from being "domain threaded" to
>> "domain" (because it no longer has "threaded" children AND either it
>> has no member processes or it has no threaded controllers enabled),
>> then the "domain invalid" descendants revert to type "domain". I'm not
>> sure whether that detail also provides some rationale as to why all
>> descendants of the threaded root cgroup aren't automatically converted
>> to type "threaded". Any thoughts about that?
> 
> Currently, thread mode implementation doesn't allow reverting back to
> domain.  The problem there is that it's impossible to tell which
> portions of the domain consumptions that are accounted to the threaded
> domain (the parent of threaded subtree) belong to the cgroup which is
> trying to revert to domain.
> 
> But assuming we in the future allow reverting back to domain,
> scenarios similar to what you suggested can become problematic.  All
> operations always behaving recursively will probably be the only sane
> solution but that blocks some possibilities for future changes.

I think I needed to be more explicit in my description of "reversion". 
I mean this situation, where we have written "threaded" to 
t1/t1-a/cgroup.type:

    t1 [dt]
        t1-a [t]
        t1-b [inv]
            t1-c [inv]

t1 is in the "domain threaded" state, and t1/t1-b and t1/t1-b/t1-c
are "domain invalid". If we now remove the t1/t1-a cgroup,
then the various other cgroups revert to type "domain":

    t1 [d]
        t1-b [d]
            t1-c [d]

And my point was that I wondered whether that had any relevance
in this discussion of why the "domain invalid" state exists.
Maybe it is irrelevant, but it just occurred to me that maybe it
is relevant.

Cheers,

Michael

-- 
Michael Kerrisk
Linux man-pages maintainer; http://www.kernel.org/doc/man-pages/
Linux/UNIX System Programming Training: http://man7.org/training/