[PATCH/RFC] Have sane default values for cpusets

[PATCH/RFC] Have sane default values for cpusets

[Dhaval Giani]

Hi folks,

This is a patch (against a somewhat older kernel) which proposes to set
a default value for a cpuset cgroup that is created. At this point in
time, this is just half done since I would prefer some comments, and see
if it is acceptable, and how.

First the description of the patch.

This patch basically sets up default values for the a cpuset that is
created. By default right now, cpuset.cpus_allowed and
cpuset.mems_allowed is empty. This does not allow a task to be attached
to the cpuset. This patch sets the default value of the cpus_allowed and
mems_allowed as the same as that of the parent.

TODO:
1. Set the value depending on the exclusive flags set in other cpusets.

This does not break ABI since applications which were explicitly setting
up the cpusets will still be setting them up anyway. And if someone was
checking if a cpuset was setup or not by checking the state of
cpuset.cpus_allowed, then it was broken and should be fixed.

Now the motivation.

Looking from an application programmer's point of view, when using
cgroups, he does not want to care about unrelated subsystem and would
only manipulate the subsystem which he is concerned with. But this is a
decision that is not just limited to the application programmer. It is a
decision that is very strongly dependent on the underlying system as
well. Cgroups allows multiple subsystems to be mounted together, which
then implies they have a common hierarchy.

Now to take an example, consider a system where cpu and memory are
mounted together, since the user wants to have the same hierarchy for
both cpu and memory. Since the application cares only about memory, it
manipulates all those values. But since they are mounted together, every
time it creates a cgroup for a task, that task will also be moved to the
corresponding cpu cgroup. The solution to this is (and the one we
recommend is) to mount all cgroups separately, but this is not always
going to happen, because it is quite painful to do this. If you use
libcgroup, you need to add additional parameters to your configuration
file. If you mount it manually, you have to specify multiple mount
commands.

Anyway, coming back to the original issue. Consider that the usecase
that the user has is a valid use case, and just mix in cpuset into this
case. Now, if the application creates a cgroup, for memory, but not
knowing that the user has mounted cpusets together, it is unable to
attach a task to its newly created cgroup because cpusets is not setup.
Now the programmer is forced to know about cpusets as well.

In order to handle this situation, libcgroup has an API which takes the
parameters from the parent cgroup. But that is also broken. Consider
this same example. If there is a cgroup, that has its cpu.rt_runtime_us
parameter setup in the another child, then the create from parent API
will fail since we tried to assign too much rt bandwidth to that cgroup.
So you can neither create a cgroup nor can you assign parameters from
its parents.

Now rt-cgroups handles this situation quite well. Since real-time is
obviously a special case, the default is to have no rt bandwidth for
that cgroup. Where cpusets goes wrong is to have a *no* default values.
So the question now is, do we expect to have this non uniform policy in
implementing subsystems, or do we enforce a policy to have sane defaults
for subsystems if they prevent attaching "regular" tasks by default.

Solving it in userspace is just adding another layer, and asking either
libcgroup to have a lot of code for just one subsystem, or expecting the
programmer to know about every subsystem, just in order to handle every
corner case.

Comments?

Thanks!
Dhaval

---
 kernel/cpuset.c |   13 +++++++++++++
 1 file changed, 13 insertions(+)

Index: linux-2.6/kernel/cpuset.c
===================================================================
--- linux-2.6.orig/kernel/cpuset.c
+++ linux-2.6/kernel/cpuset.c
@@ -1824,6 +1824,17 @@ static void cpuset_post_clone(struct cgr
 }
 
 /*
+ * Inherit the parent's cpus/mems values. Do not inhert the
+ * exclusivity flag
+ *
+ */
+static void cpuset_inherit_parent_values(struct cpuset *child)
+{
+	cpumask_copy(child->cpus_allowed, child->parent->cpus_allowed);
+	child->mems_allowed = child->parent->mems_allowed;
+}
+
+/*
  *	cpuset_create - create a cpuset
  *	ss:	cpuset cgroup subsystem
  *	cont:	control group that the new cpuset will be part of
@@ -1860,6 +1871,8 @@ static struct cgroup_subsys_state *cpuse
 	cs->relax_domain_level = -1;
 
 	cs->parent = parent;
+	cpuset_inherit_parent_values(cs);
+
 	number_of_cpusets++;
 	return &cs->css ;
 }

Re: [PATCH/RFC] Have sane default values for cpusets

[Peter Zijlstra]

On Wed, 2010-05-12 at 15:05 +0200, Dhaval Giani wrote:
> Where cpusets goes wrong is to have a *no* default values.

It has a default, empty is still a valid value.

Re: [PATCH/RFC] Have sane default values for cpusets

[Dhaval Giani]

On Wed, May 12, 2010 at 3:40 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Wed, 2010-05-12 at 15:05 +0200, Dhaval Giani wrote:
>> Where cpusets goes wrong is to have a *no* default values.
>
> It has a default, empty is still a valid value.
>

Well, it is still not sane. And in the part you snipped, I did mention,

>> do we enforce a policy to have sane defaults
>> for subsystems if they prevent attaching "regular" tasks by default.

And to add to it, a sane default can be defined as one, where a task
can be attached to a cgroup on creation without changing any other
parameter.

Dhaval

Re: [PATCH/RFC] Have sane default values for cpusets

[James Kosin]

On 5/12/2010 9:50 AM, Dhaval Giani wrote:
> On Wed, May 12, 2010 at 3:40 PM, Peter Zijlstra <peterz@infradead.org> wrote:
>> On Wed, 2010-05-12 at 15:05 +0200, Dhaval Giani wrote:
>>> Where cpusets goes wrong is to have a *no* default values.
>>
>> It has a default, empty is still a valid value.
>>
> 
> Well, it is still not sane. And in the part you snipped, I did mention,
> 
>>> do we enforce a policy to have sane defaults
>>> for subsystems if they prevent attaching "regular" tasks by default.
> 
> And to add to it, a sane default can be defined as one, where a task
> can be attached to a cgroup on creation without changing any other
> parameter.
> 
> Dhaval

By keeping the insane policy, we force everyone to properly setup to
sane defaults.  By automatically inheriting the defaults, we would be
introducing the possibility of a lazy programmer forgetting to setup the
proper defaults for their application which may need different values
than the inherited settings.  This would lead to ensuing chaos eventually.

James

Re: [PATCH/RFC] Have sane default values for cpusets

[Dhaval Giani]

[Please do *NOT* drop the cc list]

On Wed, May 12, 2010 at 4:11 PM, James Kosin <jkosin@intcomgrp.com> wrote:
> On 5/12/2010 9:50 AM, Dhaval Giani wrote:
>> On Wed, May 12, 2010 at 3:40 PM, Peter Zijlstra <peterz@infradead.org> wrote:
>>> On Wed, 2010-05-12 at 15:05 +0200, Dhaval Giani wrote:
>>>> Where cpusets goes wrong is to have a *no* default values.
>>>
>>> It has a default, empty is still a valid value.
>>>
>>
>> Well, it is still not sane. And in the part you snipped, I did mention,
>>
>>>> do we enforce a policy to have sane defaults
>>>> for subsystems if they prevent attaching "regular" tasks by default.
>>
>> And to add to it, a sane default can be defined as one, where a task
>> can be attached to a cgroup on creation without changing any other
>> parameter.
>>
>> Dhaval
>
> By keeping the insane policy, we force everyone to properly setup to
> sane defaults.  By automatically inheriting the defaults, we would be
> introducing the possibility of a lazy programmer forgetting to setup the
> proper defaults for their application which may need different values
> than the inherited settings.  This would lead to ensuing chaos eventually.

Nope. Not really. What you are saying is that an application
programmer who wants to just use memory cgroups should also care about
cpusets and just about countless other cgroup subsystems that can
exist. A lazy programmer not setting up a sane value for what he cares
about will see abnormal execution of his application and be able to
fix it.

You are talking about moving the burden of setup to the people who
should be least concerned about it.

Dhaval

Re: [PATCH/RFC] Have sane default values for cpusets

[Peter Zijlstra]

On Wed, 2010-05-12 at 16:13 +0200, Dhaval Giani wrote:
> What you are saying is that an application
> programmer who wants to just use memory cgroups should also care about
> cpusets and just about countless other cgroup subsystems that can
> exist. 

That's exactly what he says if he mounts them together.

Re: [PATCH/RFC] Have sane default values for cpusets

[Dhaval Giani]

On Wed, May 12, 2010 at 4:20 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Wed, 2010-05-12 at 16:13 +0200, Dhaval Giani wrote:
>> What you are saying is that an application
>> programmer who wants to just use memory cgroups should also care about
>> cpusets and just about countless other cgroup subsystems that can
>> exist.
>
> That's exactly what he says if he mounts them together.
>

No. That's not under his control.

Dhaval

Re: [PATCH/RFC] Have sane default values for cpusets

[Jan Safranek]

On 05/12/2010 04:20 PM, Peter Zijlstra wrote:
> On Wed, 2010-05-12 at 16:13 +0200, Dhaval Giani wrote:
>> What you are saying is that an application
>> programmer who wants to just use memory cgroups should also care about
>> cpusets and just about countless other cgroup subsystems that can
>> exist.
>
> That's exactly what he says if he mounts them together.

No, the programmer does not mount anything. Programmer writes 
application which wants to create a subgroup. System admin is the one 
who decides what is mounted how. And the programmer (=me) needs a way 
how to reliably create a subgroup, without knowing details about all 
controllers. E.g. 'blkio' controller is quite new one, old applications 
do now know anything about it, yet according to your idea, the 
application *must* provide sane defaults to it.

Jan

Re: [PATCH/RFC] Have sane default values for cpusets

[Dhaval Giani]

On Wed, May 12, 2010 at 7:38 PM, James Kosin <jkosin@intcomgrp.com> wrote:
> On 5/12/2010 10:40 AM, Jan Safranek wrote:
>
> On 05/12/2010 04:20 PM, Peter Zijlstra wrote:
>
> On Wed, 2010-05-12 at 16:13 +0200, Dhaval Giani wrote:
>
> What you are saying is that an application
> programmer who wants to just use memory cgroups should also care about
> cpusets and just about countless other cgroup subsystems that can
> exist.
>
> That's exactly what he says if he mounts them together.
>
> No, the programmer does not mount anything. Programmer writes application
> which wants to create a subgroup. System admin is the one who decides what
> is mounted how. And the programmer (=me) needs a way how to reliably create
> a subgroup, without knowing details about all controllers. E.g. 'blkio'
> controller is quite new one, old applications do now know anything about it,
> yet according to your idea, the application *must* provide sane defaults to
> it.
>
> Jan
>
>
> Jan,
>
> I think sane defaults should be enforced only if the application actively
> uses the device.  Your right in that you don't want to have to guess as to
> which devices are mounted as long as you don't touch the devices that aren't
> mounted everything should be okay and work for the specific device or mount
> you are working with  (which by the way you need to know!!!)....
>

I seem to get the impression that there is a miscommunication here. We
are talking about the cgroups feature here (more details are available
at http://www.mjmwired.net/kernel/Documentation/cgroups.txt ). We
really are not talking about devices, but about specific cgroup
subsystems which can be mounted together, and are not under the
control of the programmer.

> PS: Providing a set of defaults on creation may lead to a security
> problem... just an afterthought.
>

Which is why you have *sane* defaults.

Dhaval

Re: [PATCH/RFC] Have sane default values for cpusets

[James Kosin]

On 5/12/2010 1:42 PM, Dhaval Giani wrote:
>
> I seem to get the impression that there is a miscommunication here. We
> are talking about the cgroups feature here (more details are available
> at http://www.mjmwired.net/kernel/Documentation/cgroups.txt ). We
> really are not talking about devices, but about specific cgroup
> subsystems which can be mounted together, and are not under the
> control of the programmer.
>
>   
>
> Dhaval
>
>   
Thanks for clearing that up.
It looks like they are not under the control of the programmer on
purpose.  It is not outlined in the description of how to use this feature.
The tasks have to be added separately by an administrator that is
operating the system to tune performance and to provide resources where
most needed.

Re: [PATCH/RFC] Have sane default values for cpusets

[Chris Friesen]

On 05/12/2010 11:58 AM, James Kosin wrote:

> Thanks for clearing that up.
> It looks like they are not under the control of the programmer on
> purpose.  It is not outlined in the description of how to use this feature.
> The tasks have to be added separately by an administrator that is
> operating the system to tune performance and to provide resources where
> most needed.

Depending on the type of system, they may or may not be under the
control of the programmer. In an embedded environment, the programmer is
the administrator and generally has control over how the groups are set
up. On a more generic time-sharing system it's likely that the
individual programmer won't have control over the groups.

Something I haven't tried is whether or not it's possible to set
ownership/permissions on subtrees within the cgroup hierarchy to allow
unprivileged users/groups to control their own tasks (basically letting
them prioritize their own jobs without being able to affect anything
else).  Does anyone know if this sort of thing is possible?

Chris

Re: [PATCH/RFC] Have sane default values for cpusets

[Balbir Singh]

* Chris Friesen <cfriesen@nortel.com> [2010-05-12 12:04:08]:

> Something I haven't tried is whether or not it's possible to set
> ownership/permissions on subtrees within the cgroup hierarchy to allow
> unprivileged users/groups to control their own tasks (basically letting
> them prioritize their own jobs without being able to affect anything
> else).  Does anyone know if this sort of thing is possible?

Yep, it is.

-- 
	Three Cheers,
	Balbir

Re: [PATCH/RFC] Have sane default values for cpusets

[Paul Menage]

What about the case where some subset of the parent's mems/cpus are
given to a child with the exclusive flag set?

Paul

On Wed, May 12, 2010 at 6:05 AM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
> Hi folks,
>
> This is a patch (against a somewhat older kernel) which proposes to set
> a default value for a cpuset cgroup that is created. At this point in
> time, this is just half done since I would prefer some comments, and see
> if it is acceptable, and how.
>
> First the description of the patch.
>
> This patch basically sets up default values for the a cpuset that is
> created. By default right now, cpuset.cpus_allowed and
> cpuset.mems_allowed is empty. This does not allow a task to be attached
> to the cpuset. This patch sets the default value of the cpus_allowed and
> mems_allowed as the same as that of the parent.
>
> TODO:
> 1. Set the value depending on the exclusive flags set in other cpusets.
>
> This does not break ABI since applications which were explicitly setting
> up the cpusets will still be setting them up anyway. And if someone was
> checking if a cpuset was setup or not by checking the state of
> cpuset.cpus_allowed, then it was broken and should be fixed.
>
> Now the motivation.
>
> Looking from an application programmer's point of view, when using
> cgroups, he does not want to care about unrelated subsystem and would
> only manipulate the subsystem which he is concerned with. But this is a
> decision that is not just limited to the application programmer. It is a
> decision that is very strongly dependent on the underlying system as
> well. Cgroups allows multiple subsystems to be mounted together, which
> then implies they have a common hierarchy.
>
> Now to take an example, consider a system where cpu and memory are
> mounted together, since the user wants to have the same hierarchy for
> both cpu and memory. Since the application cares only about memory, it
> manipulates all those values. But since they are mounted together, every
> time it creates a cgroup for a task, that task will also be moved to the
> corresponding cpu cgroup. The solution to this is (and the one we
> recommend is) to mount all cgroups separately, but this is not always
> going to happen, because it is quite painful to do this. If you use
> libcgroup, you need to add additional parameters to your configuration
> file. If you mount it manually, you have to specify multiple mount
> commands.
>
> Anyway, coming back to the original issue. Consider that the usecase
> that the user has is a valid use case, and just mix in cpuset into this
> case. Now, if the application creates a cgroup, for memory, but not
> knowing that the user has mounted cpusets together, it is unable to
> attach a task to its newly created cgroup because cpusets is not setup.
> Now the programmer is forced to know about cpusets as well.
>
> In order to handle this situation, libcgroup has an API which takes the
> parameters from the parent cgroup. But that is also broken. Consider
> this same example. If there is a cgroup, that has its cpu.rt_runtime_us
> parameter setup in the another child, then the create from parent API
> will fail since we tried to assign too much rt bandwidth to that cgroup.
> So you can neither create a cgroup nor can you assign parameters from
> its parents.
>
> Now rt-cgroups handles this situation quite well. Since real-time is
> obviously a special case, the default is to have no rt bandwidth for
> that cgroup. Where cpusets goes wrong is to have a *no* default values.
> So the question now is, do we expect to have this non uniform policy in
> implementing subsystems, or do we enforce a policy to have sane defaults
> for subsystems if they prevent attaching "regular" tasks by default.
>
> Solving it in userspace is just adding another layer, and asking either
> libcgroup to have a lot of code for just one subsystem, or expecting the
> programmer to know about every subsystem, just in order to handle every
> corner case.
>
> Comments?
>
> Thanks!
> Dhaval
>
> ---
>  kernel/cpuset.c |   13 +++++++++++++
>  1 file changed, 13 insertions(+)
>
> Index: linux-2.6/kernel/cpuset.c
> ===================================================================
> --- linux-2.6.orig/kernel/cpuset.c
> +++ linux-2.6/kernel/cpuset.c
> @@ -1824,6 +1824,17 @@ static void cpuset_post_clone(struct cgr
>  }
>
>  /*
> + * Inherit the parent's cpus/mems values. Do not inhert the
> + * exclusivity flag
> + *
> + */
> +static void cpuset_inherit_parent_values(struct cpuset *child)
> +{
> +       cpumask_copy(child->cpus_allowed, child->parent->cpus_allowed);
> +       child->mems_allowed = child->parent->mems_allowed;
> +}
> +
> +/*
>  *     cpuset_create - create a cpuset
>  *     ss:     cpuset cgroup subsystem
>  *     cont:   control group that the new cpuset will be part of
> @@ -1860,6 +1871,8 @@ static struct cgroup_subsys_state *cpuse
>        cs->relax_domain_level = -1;
>
>        cs->parent = parent;
> +       cpuset_inherit_parent_values(cs);
> +
>        number_of_cpusets++;
>        return &cs->css ;
>  }
>
>
>
>

Re: [PATCH/RFC] Have sane default values for cpusets

[Lennart Poettering]

On Wed, 12.05.10 16:20, Peter Zijlstra (peterz@infradead.org) wrote:

> 
> On Wed, 2010-05-12 at 16:13 +0200, Dhaval Giani wrote:
> > What you are saying is that an application
> > programmer who wants to just use memory cgroups should also care about
> > cpusets and just about countless other cgroup subsystems that can
> > exist. 
> 
> That's exactly what he says if he mounts them together.

Well, this is not realistic.

See Dhaval's patch on the background of systemd
(http://0pointer.de/blog/projects/systemd.html). When a service is
started in systemd, we create a cgroup for it, when it ends, we remove
it. While systemd does that to keep track of processes this has the nice
side effect that all services are properly (and without races) sorted
into different groups: if you start apache, then you get it into its own
group, if you started cups, you get your own group for that -- without
further configuration. Now, while the main reason to do that is for
keeping track of processes this is also useful to actually enforce
limits and suchlike on those groups and hence services. An admin can
choose to enforce limits on the groups systemd creates
for him, because most likely the grouping systemd does along service
lines is the one that matters the most.

I am not interested to make systemd aware of each and every controller
that exists and will exist in the future and encode specific inheritance
rules for them. That is simply not possible, we'd have to add a lot of
logic to systemd I simply don't want to maintain there, and I'd have to
constantly play catch-up with every controller that is added to the
kernel. However, if I don't have that in systemd, as it stands now and
an admin tells systemd to duplicate its groups tree in the cpuset
hierarchy, then systemd would fail to work. And that is not acceptable.

So, just for once, see this from the perspective of the people using
your code: if admins want to piggybick resource limiting onto the normal
systemd cgroup tree, then you make that impossible by having weird
inheritance rules that systemd would first have to learn. (And I am
sorry, but I refuse to teach those rules to systemd, anyway)

What I am arguing here is basically that it is really important to allow
userspace code to create groups in hierarchies where controllers are
active that the userspace code does not know.

Also, it's completely stupid anyway to ask userspace code to implement
inheritance rules for each cgroup controller, if that algorithm could
just as well with minimal work be implemented on the kernel side for free,
and then allows userspace to simply rely on "mkdir" to result in a
working subgroup.

Or let me say this with other words: if an "mkdir" is not enough to
create a working sub-cgroup then libcgroup would have to learn the
necessary inheritance rules and how to copy group params from the parent
to the child -- and that for each and every controller that exists and
will exist. If a new controller is added you'd have to patch libcgroup
and the kernel and make sure they always stay in sync. And that's just
crappy design, if you ask me, and doesn't scale.

Lennart

Re: [PATCH/RFC] Have sane default values for cpusets

[Dhaval Giani]

On Wed, May 12, 2010 at 9:01 PM, Paul Menage <menage@google.com> wrote:
> What about the case where some subset of the parent's mems/cpus are
> given to a child with the exclusive flag set?
>

As I mentioned in the TODO, it is still to be handled. But it should
simply exclude those mems/cpus which are exclusive. It was a bit more
involved than the effort I wanted to put in before gauging the
reactions.

Thanks!
Dhaval

Re: [PATCH/RFC] Have sane default values for cpusets

[Paul Menage]

On Wed, May 12, 2010 at 12:10 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
> On Wed, May 12, 2010 at 9:01 PM, Paul Menage <menage@google.com> wrote:
>> What about the case where some subset of the parent's mems/cpus are
>> given to a child with the exclusive flag set?
>>
>
> As I mentioned in the TODO, it is still to be handled.

Oops, sorry, just read the patch :-)

> But it should
> simply exclude those mems/cpus which are exclusive. It was a bit more
> involved than the effort I wanted to put in before gauging the
> reactions.

I think the idea is reasonable - the only way that I could see it
breaking someone would be code that currently does something like:

mkdir A
mkdir B
echo 1 > A/mem_exclusive
echo 1 > B/mem_exclusive
echo $mems_for_a > A/mems
echo $mems_for_b > B/mems

The attempts to set the mem_exclusive flags would fail, since A and B
would both have all of the parent's mems.

Paul

Re: [PATCH/RFC] Have sane default values for cpusets

[Dhaval Giani]

On Wed, May 12, 2010 at 9:20 PM, Paul Menage <menage@google.com> wrote:
> On Wed, May 12, 2010 at 12:10 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
>> On Wed, May 12, 2010 at 9:01 PM, Paul Menage <menage@google.com> wrote:
>>> What about the case where some subset of the parent's mems/cpus are
>>> given to a child with the exclusive flag set?
>>>
>>
>> As I mentioned in the TODO, it is still to be handled.
>
> Oops, sorry, just read the patch :-)
>
>> But it should
>> simply exclude those mems/cpus which are exclusive. It was a bit more
>> involved than the effort I wanted to put in before gauging the
>> reactions.
>
> I think the idea is reasonable - the only way that I could see it
> breaking someone would be code that currently does something like:
>
> mkdir A
> mkdir B
> echo 1 > A/mem_exclusive
> echo 1 > B/mem_exclusive
> echo $mems_for_a > A/mems
> echo $mems_for_b > B/mems
>
> The attempts to set the mem_exclusive flags would fail, since A and B
> would both have all of the parent's mems.
>

But would this not fail otherwise?

Dhaval

Re: [PATCH/RFC] Have sane default values for cpusets

[Balbir Singh]

* Dhaval Giani <dhaval.giani@gmail.com> [2010-05-12 15:05:41]:

> Hi folks,
> 
> This does not break ABI since applications which were explicitly setting
> up the cpusets will still be setting them up anyway. And if someone was
> checking if a cpuset was setup or not by checking the state of
> cpuset.cpus_allowed, then it was broken and should be fixed.
>

The only ABI breakage I would consider is

A user space application reading those values and expecting them to
be empty before taking any action

-- 
	Three Cheers,
	Balbir

Re: [PATCH/RFC] Have sane default values for cpusets

[Paul Menage]

On Wed, May 12, 2010 at 12:29 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
>> I think the idea is reasonable - the only way that I could see it
>> breaking someone would be code that currently does something like:
>>
>> mkdir A
>> mkdir B
>> echo 1 > A/mem_exclusive
>> echo 1 > B/mem_exclusive
>> echo $mems_for_a > A/mems
>> echo $mems_for_b > B/mems
>>
>> The attempts to set the mem_exclusive flags would fail, since A and B
>> would both have all of the parent's mems.
>>
>
> But would this not fail otherwise?
>

Assuming that mems_for_a and mems_for_b were disjoint, it would be
fine currently.

Paul

Re: [PATCH/RFC] Have sane default values for cpusets

[Balbir Singh]

* menage@google.com <menage@google.com> [2010-05-12 12:36:57]:

> On Wed, May 12, 2010 at 12:29 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
> >> I think the idea is reasonable - the only way that I could see it
> >> breaking someone would be code that currently does something like:
> >>
> >> mkdir A
> >> mkdir B
> >> echo 1 > A/mem_exclusive
> >> echo 1 > B/mem_exclusive
> >> echo $mems_for_a > A/mems
> >> echo $mems_for_b > B/mems
> >>
> >> The attempts to set the mem_exclusive flags would fail, since A and B
> >> would both have all of the parent's mems.
> >>
> >
> > But would this not fail otherwise?
> >
> 
> Assuming that mems_for_a and mems_for_b were disjoint, it would be
> fine currently.
>

Yep, that does seem like breakage. 

-- 
	Three Cheers,
	Balbir

Re: [PATCH/RFC] Have sane default values for cpusets

[Dhaval Giani]

On Wed, May 12, 2010 at 9:36 PM, Paul Menage <menage@google.com> wrote:
> On Wed, May 12, 2010 at 12:29 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
>>> I think the idea is reasonable - the only way that I could see it
>>> breaking someone would be code that currently does something like:
>>>
>>> mkdir A
>>> mkdir B
>>> echo 1 > A/mem_exclusive
>>> echo 1 > B/mem_exclusive
>>> echo $mems_for_a > A/mems
>>> echo $mems_for_b > B/mems
>>>
>>> The attempts to set the mem_exclusive flags would fail, since A and B
>>> would both have all of the parent's mems.
>>>
>>
>> But would this not fail otherwise?
>>
>
> Assuming that mems_for_a and mems_for_b were disjoint, it would be
> fine currently.
>

Ah my bad. I misread mems_for_a as taking the value from the parent.
You are right, that was a case I missed.

Hmm, so how do we fix this? Any solutions? Not fixing the kernel
pushes the problem to the userspace, making it hard for tons of more
applications to use cgroups without jumping through a lot of hoops.

Dhaval

Re: [PATCH/RFC] Have sane default values for cpusets

[Dhaval Giani]

On Wed, May 12, 2010 at 9:59 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
> On Wed, May 12, 2010 at 9:36 PM, Paul Menage <menage@google.com> wrote:
>> On Wed, May 12, 2010 at 12:29 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
>>>> I think the idea is reasonable - the only way that I could see it
>>>> breaking someone would be code that currently does something like:
>>>>
>>>> mkdir A
>>>> mkdir B
>>>> echo 1 > A/mem_exclusive
>>>> echo 1 > B/mem_exclusive
>>>> echo $mems_for_a > A/mems
>>>> echo $mems_for_b > B/mems
>>>>
>>>> The attempts to set the mem_exclusive flags would fail, since A and B
>>>> would both have all of the parent's mems.
>>>>
>>>
>>> But would this not fail otherwise?
>>>
>>
>> Assuming that mems_for_a and mems_for_b were disjoint, it would be
>> fine currently.
>>
>
> Ah my bad. I misread mems_for_a as taking the value from the parent.
> You are right, that was a case I missed.
>
> Hmm, so how do we fix this? Any solutions? Not fixing the kernel
> pushes the problem to the userspace, making it hard for tons of more
> applications to use cgroups without jumping through a lot of hoops.
>

OK, how about this. Introduce a new option, nodefaults (or some such
name) which would retain the existing behaviour while the default
mount options would moutn cpuset with the defaults. Also, make

mount -t cpuset cpuset /cpuset

equivalent to

mount -t cgroup -onoprefix,nodefaults,cpuset cpuset /cpuset

Would that work?

Thanks,
Dhaval

Re: [PATCH/RFC] Have sane default values for cpusets

[Balbir Singh]

* Dhaval Giani <dhaval.giani@gmail.com> [2010-05-13 12:26:30]:

> On Wed, May 12, 2010 at 9:59 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
> > On Wed, May 12, 2010 at 9:36 PM, Paul Menage <menage@google.com> wrote:
> >> On Wed, May 12, 2010 at 12:29 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
> >>>> I think the idea is reasonable - the only way that I could see it
> >>>> breaking someone would be code that currently does something like:
> >>>>
> >>>> mkdir A
> >>>> mkdir B
> >>>> echo 1 > A/mem_exclusive
> >>>> echo 1 > B/mem_exclusive
> >>>> echo $mems_for_a > A/mems
> >>>> echo $mems_for_b > B/mems
> >>>>
> >>>> The attempts to set the mem_exclusive flags would fail, since A and B
> >>>> would both have all of the parent's mems.
> >>>>
> >>>
> >>> But would this not fail otherwise?
> >>>
> >>
> >> Assuming that mems_for_a and mems_for_b were disjoint, it would be
> >> fine currently.
> >>
> >
> > Ah my bad. I misread mems_for_a as taking the value from the parent.
> > You are right, that was a case I missed.
> >
> > Hmm, so how do we fix this? Any solutions? Not fixing the kernel
> > pushes the problem to the userspace, making it hard for tons of more
> > applications to use cgroups without jumping through a lot of hoops.
> >
> 
> OK, how about this. Introduce a new option, nodefaults (or some such
> name) which would retain the existing behaviour while the default
> mount options would moutn cpuset with the defaults. Also, make
> 
> mount -t cpuset cpuset /cpuset
> 
> equivalent to
> 
> mount -t cgroup -onoprefix,nodefaults,cpuset cpuset /cpuset
>

Does something like cpusetinherits make more sense.

-- 
	Three Cheers,
	Balbir

Re: [PATCH/RFC] Have sane default values for cpusets

[Peter Zijlstra]

On Wed, 2010-05-12 at 21:07 +0200, Lennart Poettering wrote:
> See Dhaval's patch on the background of systemd
> (http://0pointer.de/blog/projects/systemd.html). When a service is
> started in systemd, we create a cgroup for it, when it ends, we remove
> it. 

I seriously hope that's optional, because I for one would really hate a
system that does that. I still mostly build kernels with only cpuset in
and really don't want anybody but me creating groups in there.

Re: [PATCH/RFC] Have sane default values for cpusets

[Lennart Poettering]

On Thu, 13.05.10 15:01, Peter Zijlstra (peterz@infradead.org) wrote:

> 
> On Wed, 2010-05-12 at 21:07 +0200, Lennart Poettering wrote:
> > See Dhaval's patch on the background of systemd
> > (http://0pointer.de/blog/projects/systemd.html). When a service is
> > started in systemd, we create a cgroup for it, when it ends, we remove
> > it. 
> 
> I seriously hope that's optional, because I for one would really hate a
> system that does that. I still mostly build kernels with only cpuset in
> and really don't want anybody but me creating groups in there.

By default systemd will create its groups in the "debug" hierarchy, (at
least for now, in the long run i'd like to see "noop" hierarchy or so,
that doesn't sound so temporary), since that controller is not useful
for anything but keeping track of processes. So it shouldn't bother you
at all.

Lennart

-- 
Lennart Poettering                        Red Hat, Inc.
lennart [at] poettering [dot] net
http://0pointer.net/lennart/           GnuPG 0x1A015CC4

Re: [PATCH/RFC] Have sane default values for cpusets

[Paul Menage]

On Thu, May 13, 2010 at 7:03 AM, Lennart Poettering
<mzxreary@0pointer.de> wrote:
> By default systemd will create its groups in the "debug" hierarchy, (at
> least for now, in the long run i'd like to see "noop" hierarchy or so,
> that doesn't sound so temporary), since that controller is not useful

If you just want to track processes, mount a (named) hierarchy with no
attached subsystems.

Paul

Re: [PATCH/RFC] Have sane default values for cpusets

[Paul Menage]

On Wed, May 12, 2010 at 12:59 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
> On Wed, May 12, 2010 at 9:36 PM, Paul Menage <menage@google.com> wrote:
>> On Wed, May 12, 2010 at 12:29 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
>>>> I think the idea is reasonable - the only way that I could see it
>>>> breaking someone would be code that currently does something like:
>>>>
>>>> mkdir A
>>>> mkdir B
>>>> echo 1 > A/mem_exclusive
>>>> echo 1 > B/mem_exclusive
>>>> echo $mems_for_a > A/mems
>>>> echo $mems_for_b > B/mems
>>>>
>>>> The attempts to set the mem_exclusive flags would fail, since A and B
>>>> would both have all of the parent's mems.
>>>>
>>>
>>> But would this not fail otherwise?
>>>
>>
>> Assuming that mems_for_a and mems_for_b were disjoint, it would be
>> fine currently.
>>
>
> Ah my bad. I misread mems_for_a as taking the value from the parent.
> You are right, that was a case I missed.
>
> Hmm, so how do we fix this? Any solutions? Not fixing the kernel
> pushes the problem to the userspace, making it hard for tons of more
> applications to use cgroups without jumping through a lot of hoops.
>

Well, it's not clear to me whether the case I outlined is actually one
that would bite people - it's likely a rare case.

Balbir's point that some apps might get upset by finding non-empty
mems/cpus in a newly-created cgroup is more reasonable.

How about a per-cgroup cpuset.inherit_defaults file that defaults to
false and is inherited from the parent. If the parent's file is set to
true, then the mems/cpus are also inherited?

Then the sysadmin who's giving out user-controllable cpuset-based
cgroups can just set it to true and the users don't need to worry
about setting up the defaults.

Paul

Re: [PATCH/RFC] Have sane default values for cpusets

[Dhaval Giani]

On Thu, May 13, 2010 at 10:12 PM, Paul Menage <menage@google.com> wrote:
> On Wed, May 12, 2010 at 12:59 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
>> On Wed, May 12, 2010 at 9:36 PM, Paul Menage <menage@google.com> wrote:
>>> On Wed, May 12, 2010 at 12:29 PM, Dhaval Giani <dhaval.giani@gmail.com> wrote:
>>>>> I think the idea is reasonable - the only way that I could see it
>>>>> breaking someone would be code that currently does something like:
>>>>>
>>>>> mkdir A
>>>>> mkdir B
>>>>> echo 1 > A/mem_exclusive
>>>>> echo 1 > B/mem_exclusive
>>>>> echo $mems_for_a > A/mems
>>>>> echo $mems_for_b > B/mems
>>>>>
>>>>> The attempts to set the mem_exclusive flags would fail, since A and B
>>>>> would both have all of the parent's mems.
>>>>>
>>>>
>>>> But would this not fail otherwise?
>>>>
>>>
>>> Assuming that mems_for_a and mems_for_b were disjoint, it would be
>>> fine currently.
>>>
>>
>> Ah my bad. I misread mems_for_a as taking the value from the parent.
>> You are right, that was a case I missed.
>>
>> Hmm, so how do we fix this? Any solutions? Not fixing the kernel
>> pushes the problem to the userspace, making it hard for tons of more
>> applications to use cgroups without jumping through a lot of hoops.
>>
>
> Well, it's not clear to me whether the case I outlined is actually one
> that would bite people - it's likely a rare case.
>
> Balbir's point that some apps might get upset by finding non-empty
> mems/cpus in a newly-created cgroup is more reasonable.
>
> How about a per-cgroup cpuset.inherit_defaults file that defaults to
> false and is inherited from the parent. If the parent's file is set to
> true, then the mems/cpus are also inherited?
>
> Then the sysadmin who's giving out user-controllable cpuset-based
> cgroups can just set it to true and the users don't need to worry
> about setting up the defaults.
>

Default for root being "true" should work. Anything else, and you
still require the programmer to know about cpuset and setting that
flag to true.

Dhaval

Re: [PATCH/RFC] Have sane default values for cpusets

[Dhaval Giani]

> Default for root being "true" should work. Anything else, and you
> still require the programmer to know about cpuset and setting that
> flag to true.
>

To be honest, I don't see an easy way out of this. I think it is more
reasonable to have the change since a lot more users are benefited by
it. And in most cases, where they are broken, those applications do
require some specialist work, and so it should be reasonable to expect
the administrator to set the variable to false at mount time.

Dhaval

Re: [PATCH/RFC] Have sane default values for cpusets

[Lennart Poettering]

On Thu, 13.05.10 13:06, Paul Menage (menage@google.com) wrote:

> 
> On Thu, May 13, 2010 at 7:03 AM, Lennart Poettering
> <mzxreary@0pointer.de> wrote:
> > By default systemd will create its groups in the "debug" hierarchy, (at
> > least for now, in the long run i'd like to see "noop" hierarchy or so,
> > that doesn't sound so temporary), since that controller is not useful
> 
> If you just want to track processes, mount a (named) hierarchy with no
> attached subsystems.

Oh, that is possible? How would I do that? 

This certainly doesn't work:

# mount waldo -t cgroup /tmp/xxxx -o defaults

Lennart

-- 
Lennart Poettering                        Red Hat, Inc.
lennart [at] poettering [dot] net
http://0pointer.net/lennart/           GnuPG 0x1A015CC4

Re: [PATCH/RFC] Have sane default values for cpusets

[Lennart Poettering]

On Thu, 13.05.10 22:36, Lennart Poettering (mzxreary@0pointer.de) wrote:

> On Thu, 13.05.10 13:06, Paul Menage (menage@google.com) wrote:
> 
> > 
> > On Thu, May 13, 2010 at 7:03 AM, Lennart Poettering
> > <mzxreary@0pointer.de> wrote:
> > > By default systemd will create its groups in the "debug" hierarchy, (at
> > > least for now, in the long run i'd like to see "noop" hierarchy or so,
> > > that doesn't sound so temporary), since that controller is not useful
> > 
> > If you just want to track processes, mount a (named) hierarchy with no
> > attached subsystems.
> 
> Oh, that is possible? How would I do that? 
> 
> This certainly doesn't work:
> 
> # mount waldo -t cgroup /tmp/xxxx -o defaults

An neither does this:

# mount waldo -t cgroup /tmp/xxxx -o name=systemd

(the mount() syscall fails with EINVAL here, the other one fails with EBUSY)

Lennart

-- 
Lennart Poettering                        Red Hat, Inc.
lennart [at] poettering [dot] net
http://0pointer.net/lennart/           GnuPG 0x1A015CC4

Re: [PATCH/RFC] Have sane default values for cpusets

[Balbir Singh]

* Lennart Poettering <mzxreary@0pointer.de> [2010-05-13 22:41:59]:

> On Thu, 13.05.10 22:36, Lennart Poettering (mzxreary@0pointer.de) wrote:
> 
> > On Thu, 13.05.10 13:06, Paul Menage (menage@google.com) wrote:
> > 
> > > 
> > > On Thu, May 13, 2010 at 7:03 AM, Lennart Poettering
> > > <mzxreary@0pointer.de> wrote:
> > > > By default systemd will create its groups in the "debug" hierarchy, (at
> > > > least for now, in the long run i'd like to see "noop" hierarchy or so,
> > > > that doesn't sound so temporary), since that controller is not useful
> > > 
> > > If you just want to track processes, mount a (named) hierarchy with no
> > > attached subsystems.
> > 
> > Oh, that is possible? How would I do that? 
> > 
> > This certainly doesn't work:
> > 
> > # mount waldo -t cgroup /tmp/xxxx -o defaults
> 
> An neither does this:
> 
> # mount waldo -t cgroup /tmp/xxxx -o name=systemd
> 
> (the mount() syscall fails with EINVAL here, the other one fails with EBUSY)
>

Can you try the command below

mount -t cgroup cgroup -o none,name=hello /cgroup/ 

Works for me.

-- 
	Three Cheers,
	Balbir

Re: [PATCH/RFC] Have sane default values for cpusets

[Peter Zijlstra]

On Thu, 2010-05-13 at 16:03 +0200, Lennart Poettering wrote:
> > On Wed, 2010-05-12 at 21:07 +0200, Lennart Poettering wrote:
> > > See Dhaval's patch on the background of systemd
> > > (http://0pointer.de/blog/projects/systemd.html). When a service is
> > > started in systemd, we create a cgroup for it, when it ends, we remove
> > > it. 
> > 
> > I seriously hope that's optional, because I for one would really hate a
> > system that does that. I still mostly build kernels with only cpuset in
> > and really don't want anybody but me creating groups in there.
> 
> By default systemd will create its groups in the "debug" hierarchy, (at
> least for now, in the long run i'd like to see "noop" hierarchy or so,
> that doesn't sound so temporary), since that controller is not useful
> for anything but keeping track of processes. So it shouldn't bother you
> at all. 

Will it still work with a CONFIG_CGROUP=n kernel? I see distributions
deteriorate, you cannot even boot a raw bzImage kernel without initrd on
most distros (sure, its not too hard to fix, but still).

Also, I get all kinds of dumb-ass init-script failures for not having
modules but stuff built-in. A prime example is NFS failing on start on
both fedora and ubuntu with a built-in nfs server (for different but
both retarded reasons).

Requiring CONFIG_CGROUP=y to even get init running seems like a final
straw to ensure nobody will ever get anything to boot these days.

Re: [PATCH/RFC] Have sane default values for cpusets

[Paul Menage]

On Thu, May 13, 2010 at 1:36 PM, Lennart Poettering
<mzxreary@0pointer.de> wrote:
>> If you just want to track processes, mount a (named) hierarchy with no
>> attached subsystems.
>
> Oh, that is possible? How would I do that?
>
> This certainly doesn't work:
>
> # mount waldo -t cgroup /tmp/xxxx -o defaults

mount waldo -t cgroup /tmp/xxxx -o none,name=foo

Paul

Re: [PATCH/RFC] Have sane default values for cpusets

[Lennart Poettering]

On Fri, 14.05.10 02:37, Balbir Singh (balbir@linux.vnet.ibm.com) wrote:

> > > Oh, that is possible? How would I do that? 
> > > 
> > > This certainly doesn't work:
> > > 
> > > # mount waldo -t cgroup /tmp/xxxx -o defaults
> > 
> > An neither does this:
> > 
> > # mount waldo -t cgroup /tmp/xxxx -o name=systemd
> > 
> > (the mount() syscall fails with EINVAL here, the other one fails with EBUSY)
> >
> 
> Can you try the command below
> 
> mount -t cgroup cgroup -o none,name=hello /cgroup/ 
> 
> Works for me.

That line certainly works, but unfortunately your own libcgroup
completely ignores mount points like this. Meh.

Lennart

-- 
Lennart Poettering                        Red Hat, Inc.
lennart [at] poettering [dot] net
http://0pointer.net/lennart/           GnuPG 0x1A015CC4

Re: [PATCH/RFC] Have sane default values for cpusets

[Lennart Poettering]

On Thu, 13.05.10 23:19, Peter Zijlstra (peterz@infradead.org) wrote:

> 
> On Thu, 2010-05-13 at 16:03 +0200, Lennart Poettering wrote:
> > > On Wed, 2010-05-12 at 21:07 +0200, Lennart Poettering wrote:
> > > > See Dhaval's patch on the background of systemd
> > > > (http://0pointer.de/blog/projects/systemd.html). When a service is
> > > > started in systemd, we create a cgroup for it, when it ends, we remove
> > > > it. 
> > > 
> > > I seriously hope that's optional, because I for one would really hate a
> > > system that does that. I still mostly build kernels with only cpuset in
> > > and really don't want anybody but me creating groups in there.
> > 
> > By default systemd will create its groups in the "debug" hierarchy, (at
> > least for now, in the long run i'd like to see "noop" hierarchy or so,
> > that doesn't sound so temporary), since that controller is not useful
> > for anything but keeping track of processes. So it shouldn't bother you
> > at all. 
> 
> Will it still work with a CONFIG_CGROUP=n kernel? I see distributions
> deteriorate, you cannot even boot a raw bzImage kernel without initrd on
> most distros (sure, its not too hard to fix, but still).

No it won't work without cgroups.

> Also, I get all kinds of dumb-ass init-script failures for not having
> modules but stuff built-in. A prime example is NFS failing on start on
> both fedora and ubuntu with a built-in nfs server (for different but
> both retarded reasons).
> 
> Requiring CONFIG_CGROUP=y to even get init running seems like a final
> straw to ensure nobody will ever get anything to boot these days.

Well, I wasn't aware that cgroups is now in the kernel for the purpose
that people should *not* use it.

Next time something is added to the kernel please mark it as "Hey,
please don't use it, this is only here so that you don't use
it. Thanks!" Maybe then dumb-ass folks like me will notice and refrain
from using it.

Requiring a single kernel options is not really too much to ask, is it?
Don't be that conservative. systemd certainly won't require an initrd
or anything else equally intrusive btw.

Lennart

-- 
Lennart Poettering                        Red Hat, Inc.
lennart [at] poettering [dot] net
http://0pointer.net/lennart/           GnuPG 0x1A015CC4

Re: [PATCH/RFC] Have sane default values for cpusets

[Balbir Singh]

* Lennart Poettering <mzxreary@0pointer.de> [2010-05-14 02:02:52]:

> On Thu, 13.05.10 23:19, Peter Zijlstra (peterz@infradead.org) wrote:
> 
> > 
> > On Thu, 2010-05-13 at 16:03 +0200, Lennart Poettering wrote:
> > > > On Wed, 2010-05-12 at 21:07 +0200, Lennart Poettering wrote:
> > > > > See Dhaval's patch on the background of systemd
> > > > > (http://0pointer.de/blog/projects/systemd.html). When a service is
> > > > > started in systemd, we create a cgroup for it, when it ends, we remove
> > > > > it. 
> > > > 
> > > > I seriously hope that's optional, because I for one would really hate a
> > > > system that does that. I still mostly build kernels with only cpuset in
> > > > and really don't want anybody but me creating groups in there.
> > > 
> > > By default systemd will create its groups in the "debug" hierarchy, (at
> > > least for now, in the long run i'd like to see "noop" hierarchy or so,
> > > that doesn't sound so temporary), since that controller is not useful
> > > for anything but keeping track of processes. So it shouldn't bother you
> > > at all. 
> > 
> > Will it still work with a CONFIG_CGROUP=n kernel? I see distributions
> > deteriorate, you cannot even boot a raw bzImage kernel without initrd on
> > most distros (sure, its not too hard to fix, but still).
> 
> No it won't work without cgroups.
> 
> > Also, I get all kinds of dumb-ass init-script failures for not having
> > modules but stuff built-in. A prime example is NFS failing on start on
> > both fedora and ubuntu with a built-in nfs server (for different but
> > both retarded reasons).
> > 
> > Requiring CONFIG_CGROUP=y to even get init running seems like a final
> > straw to ensure nobody will ever get anything to boot these days.
> 
> Well, I wasn't aware that cgroups is now in the kernel for the purpose
> that people should *not* use it.
> 
> Next time something is added to the kernel please mark it as "Hey,
> please don't use it, this is only here so that you don't use
> it. Thanks!" Maybe then dumb-ass folks like me will notice and refrain
> from using it.
> 
> Requiring a single kernel options is not really too much to ask, is it?
> Don't be that conservative. systemd certainly won't require an initrd
> or anything else equally intrusive btw.
>

I think the config options are the domains of the distributors and if
the code is there and works, most distros will enable it. As long as
they have a feature that uses that option or a need for it.

I am not sure why CONFIG_CGROUP=y is so bad, Peter, could you
elaborate?

-- 
	Three Cheers,
	Balbir

Re: [PATCH/RFC] Have sane default values for cpusets

[Balbir Singh]

* Lennart Poettering <mzxreary@0pointer.de> [2010-05-14 01:55:42]:

> On Fri, 14.05.10 02:37, Balbir Singh (balbir@linux.vnet.ibm.com) wrote:
> 
> > > > Oh, that is possible? How would I do that? 
> > > > 
> > > > This certainly doesn't work:
> > > > 
> > > > # mount waldo -t cgroup /tmp/xxxx -o defaults
> > > 
> > > An neither does this:
> > > 
> > > # mount waldo -t cgroup /tmp/xxxx -o name=systemd
> > > 
> > > (the mount() syscall fails with EINVAL here, the other one fails with EBUSY)
> > >
> > 
> > Can you try the command below
> > 
> > mount -t cgroup cgroup -o none,name=hello /cgroup/ 
> > 
> > Works for me.
> 
> That line certainly works, but unfortunately your own libcgroup
> completely ignores mount points like this. Meh.
>

Lets fix it, we never thought of using named hierarchies, but it
should not be too hard to fix. 

-- 
	Three Cheers,
	Balbir

Re: [PATCH/RFC] Have sane default values for cpusets

[Dhaval Giani]

On Fri, May 14, 2010 at 1:55 AM, Lennart Poettering
<mzxreary@0pointer.de> wrote:
> On Fri, 14.05.10 02:37, Balbir Singh (balbir@linux.vnet.ibm.com) wrote:
>
>> > > Oh, that is possible? How would I do that?
>> > >
>> > > This certainly doesn't work:
>> > >
>> > > # mount waldo -t cgroup /tmp/xxxx -o defaults
>> >
>> > An neither does this:
>> >
>> > # mount waldo -t cgroup /tmp/xxxx -o name=systemd
>> >
>> > (the mount() syscall fails with EINVAL here, the other one fails with EBUSY)
>> >
>>
>> Can you try the command below
>>
>> mount -t cgroup cgroup -o none,name=hello /cgroup/
>>
>> Works for me.
>
> That line certainly works, but unfortunately your own libcgroup
> completely ignores mount points like this. Meh.
>

That be my mistake. I missed this update to cgroups. A patch will be
coming out pretty soon!

Dhaval

Re: [PATCH/RFC] Have sane default values for cpusets

[Peter Zijlstra]

On Fri, 2010-05-14 at 02:02 +0200, Lennart Poettering wrote:
> > Will it still work with a CONFIG_CGROUP=n kernel? I see distributions
> > deteriorate, you cannot even boot a raw bzImage kernel without initrd on
> > most distros (sure, its not too hard to fix, but still).
> 
> No it won't work without cgroups. 

I guess we'll finally have an answer for people on #kernelnewbies on
what distro is best suited for kernel development :-)


> > Requiring CONFIG_CGROUP=y to even get init running seems like a final
> > straw to ensure nobody will ever get anything to boot these days.
> 
> Well, I wasn't aware that cgroups is now in the kernel for the purpose
> that people should *not* use it.

Init using the bare minimum possible only seems like a sensible thing.
Once the basic stuff is running you can detect whats available and use
it if needed/wanted.

I very often build a CONFIG_CGROUP=n kernel, I simply strip down
my .config to the bare minimum because that builds fastest.

Re: [PATCH/RFC] Have sane default values for cpusets

[Peter Zijlstra]

On Fri, 2010-05-14 at 11:13 +0530, Balbir Singh wrote:

> I think the config options are the domains of the distributors 

Whoever runs a distro kernel anyway? This is LKML, we're kernel devs.

> I am not sure why CONFIG_CGROUP=y is so bad, Peter, could you
> elaborate? 

Because it adds code I don't use and adds to the compile overhead.

Re: [PATCH/RFC] Have sane default values for cpusets

[Peter Zijlstra]

On Fri, 2010-05-14 at 08:51 +0200, Peter Zijlstra wrote:
> Init using the bare minimum possible only seems like a sensible thing.
> Once the basic stuff is running you can detect whats available and use
> it if needed/wanted. 

I really think init should be able to run regardless of any CONFIG_*
option. The only thing that should avoid init from running is not being
able to mount the filesystem with init on it.

Re: [PATCH/RFC] Have sane default values for cpusets

[Balbir Singh]

* Peter Zijlstra <peterz@infradead.org> [2010-05-14 09:23:41]:

> On Fri, 2010-05-14 at 08:51 +0200, Peter Zijlstra wrote:
> > Init using the bare minimum possible only seems like a sensible thing.
> > Once the basic stuff is running you can detect whats available and use
> > it if needed/wanted. 
> 
> I really think init should be able to run regardless of any CONFIG_*
> option. The only thing that should avoid init from running is not being
> able to mount the filesystem with init on it.
>

I suppose we'll continue to have alternative with the init=
commandline option. If you look at the boot process today, we do
already rely on features (CONFIG_*) for bootup. Changing the default
to have CONFIG_CGROUP=y (without any of the controllers) would work,
won't it? Is that too high an overhead/build or runtime wise? 

-- 
	Three Cheers,
	Balbir

Re: [PATCH/RFC] Have sane default values for cpusets

[Balbir Singh]

* Peter Zijlstra <peterz@infradead.org> [2010-05-14 08:53:48]:

> On Fri, 2010-05-14 at 11:13 +0530, Balbir Singh wrote:
> 
> > I think the config options are the domains of the distributors 
> 
> Whoever runs a distro kernel anyway? This is LKML, we're kernel devs.
>

Heh! agreed, but we are a small population and moreover a large
population would benefit from systemd and parallel boot.
 
> > I am not sure why CONFIG_CGROUP=y is so bad, Peter, could you
> > elaborate? 
> 
> Because it adds code I don't use and adds to the compile overhead.

Not anymore, if systemd becomes the default :-)


-- 
	Three Cheers,
	Balbir

Re: [PATCH/RFC] Have sane default values for cpusets

[Peter Zijlstra]

On Fri, 2010-05-14 at 13:42 +0530, Balbir Singh wrote:

> Heh! agreed, but we are a small population and moreover a large
> population would benefit from systemd and parallel boot.

CONFIG_CGROUP=y isn't a requirement for parallel boot. People have been
doing that for years without it.

And I'm not against systemd using cgroups per-se, I'm against it
mandating it. It could simply not use them and not provide whatever it
needs them for when not present.

CONFIG_CGROUP is an option, so people can say no. 

Same for CONFIG_SYSFS, udev gets highly unhappy when disabled, but init
still works and you do get a shell of some sort. If you pre-populate
your /dev with static device nodes you can actually make it all the way
to runlevel 3 the last time I tried.

Re: [PATCH/RFC] Have sane default values for cpusets

[Dhaval Giani]

On Fri, May 14, 2010 at 12:38 PM, Peter Zijlstra <peterz@infradead.org> wrote:
> On Fri, 2010-05-14 at 13:42 +0530, Balbir Singh wrote:
>
>> Heh! agreed, but we are a small population and moreover a large
>> population would benefit from systemd and parallel boot.
>
> CONFIG_CGROUP=y isn't a requirement for parallel boot. People have been
> doing that for years without it.
>
> And I'm not against systemd using cgroups per-se, I'm against it
> mandating it. It could simply not use them and not provide whatever it
> needs them for when not present.
>
> CONFIG_CGROUP is an option, so people can say no.
>

I think that is reasonable. There should be a fallback for folks who
want to run their own kernel (though I cannot imagine why they would
want to turn off cgroups ;-) ).

Thanks!
Dhaval

Re: [PATCH/RFC] Have sane default values for cpusets

[Lennart Poettering]

On Fri, 14.05.10 08:53, Peter Zijlstra (peterz@infradead.org) wrote:

> 
> On Fri, 2010-05-14 at 11:13 +0530, Balbir Singh wrote:
> 
> > I think the config options are the domains of the distributors 
> 
> Whoever runs a distro kernel anyway? This is LKML, we're kernel devs.

Well, sorry to bring that to you, but I didn't really design systemd as
a development tool for lkml developers. Instead I designed it as a
general purposes system manager.

Lennart

-- 
Lennart Poettering                        Red Hat, Inc.
lennart [at] poettering [dot] net
http://0pointer.net/lennart/           GnuPG 0x1A015CC4