Attaching a cgroup subsystem to multiple hierarchies

Attaching a cgroup subsystem to multiple hierarchies

[Glyn Normington]

Reading cgroups.txt and casting around the net leads me to believe that it is possible to attach a cgroup subsystem (e.g. memory) to multiple hierarchies, but this seems to result in “mirrored” hierarchies which are automatically kept in step with each other - essentially it looks like the same hierarchy at multiple file system paths.

Take the following interaction for example:

\begin{verbatim}
$ pwd   
/home/vagrant
$ mkdir mem1
$ mkdir mem2
$ sudo su
# mount -t cgroup -o memory none /home/vagrant/mem1
# mount -t cgroup -o memory none /home/vagrant/mem2
# cd mem1
# mkdir inst1  
# ls inst1 
cgroup.clone_children  memory.failcnt ...
# ls ../mem2
cgroup.clone_children  inst1 memory.limit_in_bytes ...
# cd inst1
# echo 1000000 > memory.limit_in_bytes 
# cat memory.limit_in_bytes 
1003520
# cat ../../mem2/inst1/memory.limit_in_bytes 
1003520
# echo $$ > tasks
# cat tasks
1365
1409
# cat ../../mem2/inst1/tasks
1365
1411

Is this working as intended? Is there some other way to attach a subsystem to *distinct* hierarchies? Distinct hierarchies would allow distinct cgroups, distinct settings (e.g. memory.limit_in_bytes) and distinct partitions of the tasks in the system.

Note: I don’t have a good use for this function - I’m simply trying to reverse engineer the semantics of cgroups to get a precise understanding.

Glyn

Re: Attaching a cgroup subsystem to multiple hierarchies

[Michal Hocko]

On Wed 05-02-14 14:39:52, Glyn Normington wrote:
> Reading cgroups.txt and casting around the net leads me to believe
> that it is possible to attach a cgroup subsystem (e.g. memory) to
> multiple hierarchies, but this seems to result in “mirrored”
> hierarchies which are automatically kept in step with each other -
> essentially it looks like the same hierarchy at multiple file system
> paths.
> 
> Take the following interaction for example:
> 
> \begin{verbatim}
> $ pwd   
> /home/vagrant
> $ mkdir mem1
> $ mkdir mem2
> $ sudo su
> # mount -t cgroup -o memory none /home/vagrant/mem1
> # mount -t cgroup -o memory none /home/vagrant/mem2
> # cd mem1
> # mkdir inst1  
> # ls inst1 
> cgroup.clone_children  memory.failcnt ...
> # ls ../mem2
> cgroup.clone_children  inst1 memory.limit_in_bytes ...
> # cd inst1
> # echo 1000000 > memory.limit_in_bytes 
> # cat memory.limit_in_bytes 
> 1003520
> # cat ../../mem2/inst1/memory.limit_in_bytes 
> 1003520
> # echo $$ > tasks
> # cat tasks
> 1365
> 1409
> # cat ../../mem2/inst1/tasks
> 1365
> 1411
> 
> Is this working as intended?

Yes, it doesn't make any sense to have two different views on the same
controllers.

> Is there some other way to attach a subsystem to *distinct*
> hierarchies?

No.

> Distinct hierarchies would allow distinct cgroups, distinct settings
> (e.g. memory.limit_in_bytes) and distinct partitions of the tasks in
> the system.

Which one should be applied then?

> 
> Note: I don’t have a good use for this function - I’m simply
> trying to reverse engineer the semantics of cgroups to get a precise
> understanding.

I think there is no need to reverse engineer ;)
Documentation/cgroups/cgroups.txt in the kernel tree does give a decent
description IMO.

-- 
Michal Hocko
SUSE Labs

Re: Attaching a cgroup subsystem to multiple hierarchies

[Glyn Normington]

Hi Michal

On 6 Feb 2014, at 18:59, Michal Hocko <mhocko@suse.cz> wrote:

> On Wed 05-02-14 14:39:52, Glyn Normington wrote:
>> Reading cgroups.txt and casting around the net leads me to believe
>> that it is possible to attach a cgroup subsystem (e.g. memory) to
>> multiple hierarchies, but this seems to result in “mirrored”
>> hierarchies which are automatically kept in step with each other -
>> essentially it looks like the same hierarchy at multiple file system
>> paths.
>> 
>> Take the following interaction for example:
>> 
>> \begin{verbatim}
>> $ pwd   
>> /home/vagrant
>> $ mkdir mem1
>> $ mkdir mem2
>> $ sudo su
>> # mount -t cgroup -o memory none /home/vagrant/mem1
>> # mount -t cgroup -o memory none /home/vagrant/mem2
>> # cd mem1
>> # mkdir inst1  
>> # ls inst1 
>> cgroup.clone_children  memory.failcnt ...
>> # ls ../mem2
>> cgroup.clone_children  inst1 memory.limit_in_bytes ...
>> # cd inst1
>> # echo 1000000 > memory.limit_in_bytes 
>> # cat memory.limit_in_bytes 
>> 1003520
>> # cat ../../mem2/inst1/memory.limit_in_bytes 
>> 1003520
>> # echo $$ > tasks
>> # cat tasks
>> 1365
>> 1409
>> # cat ../../mem2/inst1/tasks
>> 1365
>> 1411
>> 
>> Is this working as intended?
> 
> Yes, it doesn't make any sense to have two different views on the same
> controllers.

Then wouldn’t it be better for the second mount to fail?

> 
>> Is there some other way to attach a subsystem to *distinct*
>> hierarchies?
> 
> No.
> 
>> Distinct hierarchies would allow distinct cgroups, distinct settings
>> (e.g. memory.limit_in_bytes) and distinct partitions of the tasks in
>> the system.
> 
> Which one should be applied then?

Good question. All of them, I would say: the constraints due to distinct settings would be ANDed together.

The implementation would be more complex and less efficient as a subsystem's resources consumed by a process would need charging against each hierarchy to which the subsystem was attached.

I very much doubt this would be worth implementing and I’m not at all suggesting it.

> 
>> 
>> Note: I don’t have a good use for this function - I’m simply
>> trying to reverse engineer the semantics of cgroups to get a precise
>> understanding.
> 
> I think there is no need to reverse engineer ;)
> Documentation/cgroups/cgroups.txt in the kernel tree does give a decent
> description IMO.

I disagree. For example, cgroups.txt does not clearly state whether or not a single subsystem may be attached to distinct hierarchies.

This seems to have caused confusion elsewhere. For example, Red Hat write “… a single subsystem can be attached to two hierarchies if both of those hierarchies have only that subsystem attached.” ([1]).

> 
> -- 
> Michal Hocko
> SUSE Labs

Regards,
Glyn

[1] See “Rule 2” on p. 10 of https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/pdf/Resource_Management_Guide/Red_Hat_Enterprise_Linux-6-Resource_Management_Guide-en-US.pdf

Re: Attaching a cgroup subsystem to multiple hierarchies

[Li Zefan]

(Add Michal back to the Cc list, and Cc cgroup mailing list)

On 2014/2/7 17:21, Glyn Normington wrote:
> Hi Michal
> 
> On 6 Feb 2014, at 18:59, Michal Hocko <mhocko@suse.cz> wrote:
> 
>> On Wed 05-02-14 14:39:52, Glyn Normington wrote:
>>> Reading cgroups.txt and casting around the net leads me to believe
>>> that it is possible to attach a cgroup subsystem (e.g. memory) to
>>> multiple hierarchies, but this seems to result in “mirrored”
>>> hierarchies which are automatically kept in step with each other -
>>> essentially it looks like the same hierarchy at multiple file system
>>> paths.
>>>
>>> Take the following interaction for example:
>>>
>>> \begin{verbatim}
>>> $ pwd   
>>> /home/vagrant
>>> $ mkdir mem1
>>> $ mkdir mem2
>>> $ sudo su
>>> # mount -t cgroup -o memory none /home/vagrant/mem1
>>> # mount -t cgroup -o memory none /home/vagrant/mem2
>>> # cd mem1
>>> # mkdir inst1  
>>> # ls inst1 
>>> cgroup.clone_children  memory.failcnt ...
>>> # ls ../mem2
>>> cgroup.clone_children  inst1 memory.limit_in_bytes ...
>>> # cd inst1
>>> # echo 1000000 > memory.limit_in_bytes 
>>> # cat memory.limit_in_bytes 
>>> 1003520
>>> # cat ../../mem2/inst1/memory.limit_in_bytes 
>>> 1003520
>>> # echo $$ > tasks
>>> # cat tasks
>>> 1365
>>> 1409
>>> # cat ../../mem2/inst1/tasks
>>> 1365
>>> 1411
>>>
>>> Is this working as intended?
>>
>> Yes, it doesn't make any sense to have two different views on the same
>> controllers.
> 
> Then wouldn’t it be better for the second mount to fail?
> 

We don't disallow mounting procfs/sysfs to more than one mount point.
Why we want to do this to cgroupfs?

>>
>>> Is there some other way to attach a subsystem to *distinct*
>>> hierarchies?
>>
>> No.
>>
>>> Distinct hierarchies would allow distinct cgroups, distinct settings
>>> (e.g. memory.limit_in_bytes) and distinct partitions of the tasks in
>>> the system.
>>
>> Which one should be applied then?
> 
> Good question. All of them, I would say: the constraints due to distinct settings would be ANDed together.
> 
> The implementation would be more complex and less efficient as a subsystem's resources consumed by a process would need charging against each hierarchy to which the subsystem was attached.
> 
> I very much doubt this would be worth implementing and I’m not at all suggesting it.
> 

Don't even think about it. :)

>>
>>>
>>> Note: I don’t have a good use for this function - I’m simply
>>> trying to reverse engineer the semantics of cgroups to get a precise
>>> understanding.
>>
>> I think there is no need to reverse engineer ;)
>> Documentation/cgroups/cgroups.txt in the kernel tree does give a decent
>> description IMO.
> 
> I disagree. For example, cgroups.txt does not clearly state whether or not a single subsystem may be attached to distinct hierarchies.
> 
> This seems to have caused confusion elsewhere. For example, Red Hat write “… a single subsystem can be attached to two hierarchies if both of those hierarchies have only that subsystem attached.” ([1]).
> 

No documentation is perfect, but you can make it better by sending us
a patch.

Re: Attaching a cgroup subsystem to multiple hierarchies

[Glyn Normington]

On 8 February 2014 02:26, Li Zefan <lizefan@huawei.com> wrote:
> (Add Michal back to the Cc list, and Cc cgroup mailing list)
>
> On 2014/2/7 17:21, Glyn Normington wrote:
>> Hi Michal
>>
>> On 6 Feb 2014, at 18:59, Michal Hocko <mhocko@suse.cz> wrote:
>>
>>> On Wed 05-02-14 14:39:52, Glyn Normington wrote:
>>>> Reading cgroups.txt and casting around the net leads me to believe
>>>> that it is possible to attach a cgroup subsystem (e.g. memory) to
>>>> multiple hierarchies, but this seems to result in "mirrored"
>>>> hierarchies which are automatically kept in step with each other -
>>>> essentially it looks like the same hierarchy at multiple file system
>>>> paths.
>>>>
>>>> Take the following interaction for example:
>>>>
>>>> \begin{verbatim}
>>>> $ pwd
>>>> /home/vagrant
>>>> $ mkdir mem1
>>>> $ mkdir mem2
>>>> $ sudo su
>>>> # mount -t cgroup -o memory none /home/vagrant/mem1
>>>> # mount -t cgroup -o memory none /home/vagrant/mem2
>>>> # cd mem1
>>>> # mkdir inst1
>>>> # ls inst1
>>>> cgroup.clone_children  memory.failcnt ...
>>>> # ls ../mem2
>>>> cgroup.clone_children  inst1 memory.limit_in_bytes ...
>>>> # cd inst1
>>>> # echo 1000000 > memory.limit_in_bytes
>>>> # cat memory.limit_in_bytes
>>>> 1003520
>>>> # cat ../../mem2/inst1/memory.limit_in_bytes
>>>> 1003520
>>>> # echo $$ > tasks
>>>> # cat tasks
>>>> 1365
>>>> 1409
>>>> # cat ../../mem2/inst1/tasks
>>>> 1365
>>>> 1411
>>>>
>>>> Is this working as intended?
>>>
>>> Yes, it doesn't make any sense to have two different views on the same
>>> controllers.
>>
>> Then wouldn't it be better for the second mount to fail?
>>
>
> We don't disallow mounting procfs/sysfs to more than one mount point.
> Why we want to do this to cgroupfs?
>

If we accept that this scenario does make sense (even if it's not
particularly useful), then yes, we should continue to allow it. But it
would be worth documenting the behaviour in cgroups.txt.

>>>
>>>> Is there some other way to attach a subsystem to *distinct*
>>>> hierarchies?
>>>
>>> No.
>>>
>>>> Distinct hierarchies would allow distinct cgroups, distinct settings
>>>> (e.g. memory.limit_in_bytes) and distinct partitions of the tasks in
>>>> the system.
>>>
>>> Which one should be applied then?
>>
>> Good question. All of them, I would say: the constraints due to distinct settings would be ANDed together.
>>
>> The implementation would be more complex and less efficient as a subsystem's resources consumed by a process would need charging against each hierarchy to which the subsystem was attached.
>>
>> I very much doubt this would be worth implementing and I'm not at all suggesting it.
>>
>
> Don't even think about it. :)
>
>>>
>>>>
>>>> Note: I don't have a good use for this function - I'm simply
>>>> trying to reverse engineer the semantics of cgroups to get a precise
>>>> understanding.
>>>
>>> I think there is no need to reverse engineer ;)
>>> Documentation/cgroups/cgroups.txt in the kernel tree does give a decent
>>> description IMO.
>>
>> I disagree. For example, cgroups.txt does not clearly state whether or not a single subsystem may be attached to distinct hierarchies.
>>
>> This seems to have caused confusion elsewhere. For example, Red Hat write "... a single subsystem can be attached to two hierarchies if both of those hierarchies have only that subsystem attached." ([1]).
>>
>
> No documentation is perfect, but you can make it better by sending us
> a patch.
>

I certainly intend to, after some more investigation.

-- 
Regards,
Glyn

Re: Attaching a cgroup subsystem to multiple hierarchies

[Glyn Normington]

On 08/02/2014 02:26, Li Zefan wrote:
> No documentation is perfect, but you can make it better by sending us
> a patch.
Done: http://lkml.iu.edu//hypermail/linux/kernel/1403.1/00683.html