Re: [Lse-tech] Re: CPUSET Proposal

Re: [Lse-tech] Re: CPUSET Proposal

[Hanna Linder]

--On Wednesday, September 24, 2003 03:06:08 PM -0700 David Mosberger <davidm@napali.hpl.hp.com> wrote:

> Simon,
> 
> Could you please make it VERY clear that the system call numbers the
> patch is using are NOT OFFICIAL numbers and are therefore likely to
> change and/or collide unless and until they get accepted into the
> official tree?
> 
> BTW: What do cpusets provide that couldn't be done with user-level
> tools on top of the existing sched_setaffinity() system call?
> 
> Thanks,
> 
> 	--david

Blatent plug for the group call:

We have time at the next conference call (Oct 1, 2pm PDT, GMT -0700)
to discuss this if you all can make it and are interested. The only
other topic on the agenda is "real time application latency when the
system is stressed" from Mark Gross at Intel. 

Simon/Sylvain, let me know if one or both of you would be interested in 
calling in to give a brief overview and answer questions of this proposal.
No need for slides or anything, it is a very informal group. If you are
outside the US I can arrange a toll-free number.

Hanna

Re: [Lse-tech] Re: CPUSET Proposal

[Paul Jackson]

> BTW: What do cpusets provide that couldn't be done with user-level
> tools on top of the existing sched_setaffinity() system call?

I don't see how you can do the migrate_cpuset_processes() from a user
level daemon.  Just because two tasks happen to be allowed on the same
CPUs doesn't mean they are in the same cpuset.  The kernel must track,
across forks, which tasks share a given cpuset.

There are also some resource management capabilities, such as tracking
and controlling how much memory a cpuset takes, and swapping (with
possible oom kill) against a cpuset that one can consider extending this
to, but only if it's in the kernel.  But I'm not ready to push this
point ... yet.

And the permission model has to remain a rather primitive "root can do
anything, anyone else can just subset their parent" if it lacks kernel
hooks to track uid/suid ownership of each cpuset.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: [Lse-tech] Re: CPUSET Proposal

[William Lee Irwin III]

At some point in the past, someone wrote:
>> BTW: What do cpusets provide that couldn't be done with user-level
>> tools on top of the existing sched_setaffinity() system call?

On Wed, Sep 24, 2003 at 10:39:44PM -0700, Paul Jackson wrote:
> I don't see how you can do the migrate_cpuset_processes() from a user
> level daemon.  Just because two tasks happen to be allowed on the same
> CPUs doesn't mean they are in the same cpuset.  The kernel must track,
> across forks, which tasks share a given cpuset.
> There are also some resource management capabilities, such as tracking
> and controlling how much memory a cpuset takes, and swapping (with
> possible oom kill) against a cpuset that one can consider extending this
> to, but only if it's in the kernel.  But I'm not ready to push this
> point ... yet.
> And the permission model has to remain a rather primitive "root can do
> anything, anyone else can just subset their parent" if it lacks kernel
> hooks to track uid/suid ownership of each cpuset.

This sounds like it has progressively more commonality with CKRM; the
notion is of a workclass, not of a purely cpu-oriented notion.


-- wli

Re: [Lse-tech] Re: CPUSET Proposal

[Paul Jackson]

> This sounds like it has progressively more commonality with CKRM; the
> notion is of a workclass, not of a purely cpu-oriented notion.

I _knew_ I shouldn't have thrown in that paragraph that began "There are
also some resource management capabilities, ...".

There are two aspects to CKRM - a common classification of service levels,
and hooks in each scheduler of resources to respect those levels.

These cpusets, either as proposed, or possible fancier forms that also
manage memory, do not replace, cannot be replaced by, and do not compete
with CKRM.  Rather they cooperate with CKRM, and represent one more
place, along side network drivers, schedulers and memory allocators,
that eventually will want to respect CKRM service levels.

The point of _this_ subthread was to consider whether this could more or
less entirely be done in user space.  The two aspects even of Simon's
current proposal that I don't see can be done in user space are the
migration, and the permission model.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: [Lse-tech] Re: CPUSET Proposal

[William Lee Irwin III]

At some point in the past, I wrote:
>> This sounds like it has progressively more commonality with CKRM; the
>> notion is of a workclass, not of a purely cpu-oriented notion.

On Wed, Sep 24, 2003 at 11:09:58PM -0700, Paul Jackson wrote:
> I _knew_ I shouldn't have thrown in that paragraph that began "There are
> also some resource management capabilities, ...".
> There are two aspects to CKRM - a common classification of service levels,
> and hooks in each scheduler of resources to respect those levels.
> These cpusets, either as proposed, or possible fancier forms that also
> manage memory, do not replace, cannot be replaced by, and do not compete
> with CKRM.  Rather they cooperate with CKRM, and represent one more
> place, along side network drivers, schedulers and memory allocators,
> that eventually will want to respect CKRM service levels.
> The point of _this_ subthread was to consider whether this could more or
> less entirely be done in user space.  The two aspects even of Simon's
> current proposal that I don't see can be done in user space are the
> migration, and the permission model.

Well, the thing is, CKRM essentially has the cross-resource bits and
makes up some group that can be joined and departed from and inherited
and so on with all the right knobs. So the question is "what does CKRM
lack that cpusets needs?" It sounds to me like the major point missing
is "nobody else can touch this cpu" if I read cpusets properly.


-- wli

Re: [Lse-tech] Re: CPUSET Proposal

[Paul Jackson]

> Well, the thing is, CKRM essentially has the cross-resource bits and
> makes up some group that can be joined and departed from and inherited
> and so on with all the right knobs ...

The hierarchies don't correspond, or do so only accidentally.

That is, cpusets, as proposed, have a hierarchy such that one
cpuset is the child of another if one cpuset describes a subset
of another's CPUs.

At first blush, I don't see a hierarchy of CKRM Classes, rather
just a flat space, say Gold, Silver and Bronze.

  "When all you have is a hammer, the whole world looks like a nail"

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: [Lse-tech] Re: CPUSET Proposal

[William Lee Irwin III]

At some point in the past, I wrote:
>> Well, the thing is, CKRM essentially has the cross-resource bits and
>> makes up some group that can be joined and departed from and inherited
>> and so on with all the right knobs ...

On Wed, Sep 24, 2003 at 11:38:04PM -0700, Paul Jackson wrote:
> The hierarchies don't correspond, or do so only accidentally.
> That is, cpusets, as proposed, have a hierarchy such that one
> cpuset is the child of another if one cpuset describes a subset
> of another's CPUs.
> At first blush, I don't see a hierarchy of CKRM Classes, rather
> just a flat space, say Gold, Silver and Bronze.

It's meant to flatten the hierarchy by using numerical measures of
share to precompute the effect of the hierarchy. A directly hierarchical
data structure representation's traversal is too inefficient to be
tolerated in certain performance-critical codepaths, e.g. schedule().

The hierarchy is meant to be there, just implemented without that
traversal in the scheduler (and elsewhere).


-- wli

Re: [Lse-tech] Re: CPUSET Proposal

[Paul Jackson]

> It's meant to flatten the hierarchy ...
>  ...
> The hierarchy is meant to be there, just ...


_What_ is meant to flatten the hierarchy ??  To what does "It" refer ??

So is the hierarchy of CKRM there or not -- you've confused me.

And in any case, are we in agreement that any such CKRM hierarchy
is not isomorphic to the cpuset hierarchy?

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: [Lse-tech] Re: CPUSET Proposal

[William Lee Irwin III]

At some point in the past, I wrote:
>> It's meant to flatten the hierarchy ...
>>  ...
>> The hierarchy is meant to be there, just ...

On Wed, Sep 24, 2003 at 11:51:20PM -0700, Paul Jackson wrote:
> _What_ is meant to flatten the hierarchy ??  To what does "It" refer ??
> So is the hierarchy of CKRM there or not -- you've confused me.
> And in any case, are we in agreement that any such CKRM hierarchy
> is not isomorphic to the cpuset hierarchy?

The numerical share assignments.

I don't have the intestinal fortitude to pore over the pronouns used
in the last message.

The hierarchy used in CKRM is there.

I don't know if the CKRM hierarchy is different from yours or not.
It's a subgraph of the process inheritance hierarchy.

At any rate, I'll hope my little request to have similar mechanisms
consolidated is heeded somehow and bow out at this point.


-- wli

Re: [Lse-tech] Re: CPUSET Proposal

[Paul Jackson]

> The hierarchy used in CKRM is there. ...
> It's a subgraph of the process inheritance hierarchy.

Any recommendations on how I might learn more of such?


> my little request to have similar mechanisms consolidated ...

Such requests are valuable ... especially if there is more similarity
than just "hierarchical resource grouping something or other ...".


> bow out at this point.

Ok - take care.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: [Lse-tech] Re: CPUSET Proposal

[Hubertus Franke]

Paul Jackson wrote:

>>This sounds like it has progressively more commonality with CKRM; the
>>notion is of a workclass, not of a purely cpu-oriented notion.
>>    
>>
>
>I _knew_ I shouldn't have thrown in that paragraph that began "There are
>also some resource management capabilities, ...".
>
>There are two aspects to CKRM - a common classification of service levels,
>and hooks in each scheduler of resources to respect those levels.
>
>  
>
That is correct (assuming slight modification of the schedulers 
qualifies as a hook).

>These cpusets, either as proposed, or possible fancier forms that also
>manage memory, do not replace, cannot be replaced by, and do not compete
>with CKRM.  Rather they cooperate with CKRM, and represent one more
>place, along side network drivers, schedulers and memory allocators,
>that eventually will want to respect CKRM service levels.
>
>  
>
Yes, to my understanding of cpusets (and I haven't looked into it with 
great detail) its a
virtualization layer above physical binding. One really doesn't care to 
which CPU a process is
bound as long as it is bound to one. One might care that tasks are 
constraint to a particular
number of tasks and not beyond, thus leading to the partitioning 
capabilities.
So I agree here with Paul that it addresses more a physical separation 
of processes, or say
partitioning of machine which CKRM is targeted towards resource 
utilization within a class.
Just like cpu_affinity, CKRM could tolerate cpusets.

>The point of _this_ subthread was to consider whether this could more or
>less entirely be done in user space.  The two aspects even of Simon's
>current proposal that I don't see can be done in user space are the
>migration, and the permission model.
>
>  
>

-- Hubertus Franke     ( CKRM team )

Re: [Lse-tech] Re: CPUSET Proposal

[Hubertus Franke]

Paul Jackson wrote:

>>Well, the thing is, CKRM essentially has the cross-resource bits and
>>makes up some group that can be joined and departed from and inherited
>>and so on with all the right knobs ...
>>    
>>
>
>The hierarchies don't correspond, or do so only accidentally.
>
>That is, cpusets, as proposed, have a hierarchy such that one
>cpuset is the child of another if one cpuset describes a subset
>of another's CPUs.
>
>At first blush, I don't see a hierarchy of CKRM Classes, rather
>just a flat space, say Gold, Silver and Bronze.
>  
>
Paul, yes CKRM classes at this point are flat, we looked initially at 
hierarchies and
determined that for the first release might add a lot of complexity with 
questionable
benefits for the community at large. So we left hierarchies out.
Based on the general community feedback we might have to revisit this issue.

Again, I see cpusets and CKRM as addressing two orthogonal issues wrt to 
cpu's

cpusets  (partitioning in space)   with hierarchies
CKRM  (time partitioning)         how much of  time does a class get...

-- Hubertus Franke   ( CKRM team )

Re: [Lse-tech] Re: CPUSET Proposal

[Hubertus Franke]

Paul Jackson wrote:

>>It's meant to flatten the hierarchy ...
>> ...
>>The hierarchy is meant to be there, just ...
>>    
>>
>
>
>_What_ is meant to flatten the hierarchy ??  To what does "It" refer ??
>
>So is the hierarchy of CKRM there or not -- you've confused me.
>
>And in any case, are we in agreement that any such CKRM hierarchy
>is not isomorphic to the cpuset hierarchy?
>
>  
>
No hierarchy in CKRM  (as of now).

IHMO     even if we introduce a hierarchies in CKRM, I don't see that 
they are isomorphic to cpuset
hiearchies.

-- Hubertus Franke ( CKRM team )      c

Re: [Lse-tech] Re: CPUSET Proposal

[Simon Derr]

On Wed, 24 Sep 2003, David Mosberger wrote:

> BTW: What do cpusets provide that couldn't be done with user-level
> tools on top of the existing sched_setaffinity() system call?
This is a question we had a long in-house debate about.

The main reason of the inclusion of cpusets *inside* the kernel, is that
we have to deal with applications that may call sched_setaffinity() to
bind their processes to CPUs. Therefore we have to intercept these calls.
We could try to do some LD_PRELOAD userland trick or modify the libc, but
that would not work for statically linked programs.

As pointed by Paul, another reason is the possibility to change on the fly
the size/location of cpusets, and the need to apply the change on the
attached processes.


Thanks a lot for your comments,

	Simon.

Re: [Lse-tech] Re: CPUSET Proposal

[Dave Hansen]

On Thu, 2003-09-25 at 06:26, Simon Derr wrote:
> On Wed, 24 Sep 2003, David Mosberger wrote:
> 
> > BTW: What do cpusets provide that couldn't be done with user-level
> > tools on top of the existing sched_setaffinity() system call?
> This is a question we had a long in-house debate about.
> 
> The main reason of the inclusion of cpusets *inside* the kernel, is that
> we have to deal with applications that may call sched_setaffinity() to
> bind their processes to CPUs. Therefore we have to intercept these calls.
> We could try to do some LD_PRELOAD userland trick or modify the libc, but
> that would not work for statically linked programs.

You could also do a big chunk of this by allowing normal privledge users
to sched_setaffinity() a *subset* of their current allowed CPU set, but
not expand it.  sched_setaffinity() isn't *that* old of an interface, so
I'm not sure why you can't just change the application at this point.  

-- 
Dave Hansen
haveblue@us.ibm.com

RE: [Lse-tech] Re: CPUSET Proposal

[Luck, Tony]

> You could also do a big chunk of this by allowing normal 
> privledge users
> to sched_setaffinity() a *subset* of their current allowed 
> CPU set, but
> not expand it.  sched_setaffinity() isn't *that* old of an 
> interface, so
> I'm not sure why you can't just change the application at 
> this point.  

Because you need virtual cpu numbers.  Suppose I have a 16-way
system, and *two* applications that know nothing about each other,
but each happens to want 5 cpus to run.

With cpusets each application can ask the kernel for 5 cpus
(or a wrapper that invokes the application can do so), and then
the application can happily divide its work between virtual cpus
0, 1, 2, 3, 4 (using sched_setaffinity()).  The kernel can be smart
and provide different physical cpus for each of the cpusets.

-Tony

RE: [Lse-tech] Re: CPUSET Proposal

[Yu, Fenghua]

>> You could also do a big chunk of this by allowing normal 
>> privledge users
>> to sched_setaffinity() a *subset* of their current allowed 
>> CPU set, but
>> not expand it.  sched_setaffinity() isn't *that* old of an 
>> interface, so
>> I'm not sure why you can't just change the application at 
>> this point.  

>Because you need virtual cpu numbers.  Suppose I have a 16-way
>system, and *two* applications that know nothing about each other,
>but each happens to want 5 cpus to run.

>With cpusets each application can ask the kernel for 5 cpus
>(or a wrapper that invokes the application can do so), and then
>the application can happily divide its work between virtual cpus
>0, 1, 2, 3, 4 (using sched_setaffinity()).  The kernel can be smart
>and provide different physical cpus for each of the cpusets.

In this case, the applications can get all of current affinity info by
calling sched_getaffinity(). Thus, the applications can schedule the
processes to different cpus without bothering kernel.

Re: [Lse-tech] Re: CPUSET Proposal

[Sylvain Jeaugey]

On Thu, 25 Sep 2003, Hubertus Franke wrote:

> Again, I see cpusets and CKRM as addressing two orthogonal issues wrt to 
> cpu's
> 
> cpusets  (partitioning in space)   with hierarchies
> CKRM  (time partitioning)         how much of  time does a class get...
We do agree. We took a look at CKRM and that is the conclusion we 
achieved. At first sight, it could look like the goal are the same -and in 
some points it is- but the two approaches are different. It looks 
like it would be better to combine them rather to try to merge them.

Sylvain

RE: [Lse-tech] Re: CPUSET Proposal

[Sylvain Jeaugey]

On Thu, 25 Sep 2003, Yu, Fenghua wrote:

> >> You could also do a big chunk of this by allowing normal 
> >> privledge users
> >> to sched_setaffinity() a *subset* of their current allowed 
> >> CPU set, but
> >> not expand it.  sched_setaffinity() isn't *that* old of an 
> >> interface, so
> >> I'm not sure why you can't just change the application at 
> >> this point.  
> 
> >Because you need virtual cpu numbers.  Suppose I have a 16-way
> >system, and *two* applications that know nothing about each other,
> >but each happens to want 5 cpus to run.
That's exactly the point in virtualizating CPUs. 

> 
> >With cpusets each application can ask the kernel for 5 cpus
> >(or a wrapper that invokes the application can do so), and then
> >the application can happily divide its work between virtual cpus
> >0, 1, 2, 3, 4 (using sched_setaffinity()).  The kernel can be smart
> >and provide different physical cpus for each of the cpusets.
Actually, the intelligence is located in the wrapper application. The 
kernel part of cpusets is given absolute masks. The virtualization is 
located in the fact that calls to sched_setaffinity() are interpreted 
within the cpuset, and that /proc/cpuinfo is also modified to reflect 
available CPUs.

> In this case, the applications can get all of current affinity info by
> calling sched_getaffinity(). Thus, the applications can schedule the
> processes to different cpus without bothering kernel.
The problem is not only to run on a different CPU. We want that the used 
CPU is decided by the admin and not the application. Otherwise you would 
have no control on it. It could have been a solution for applications we 
write like MPI (and that is the first thing we did).
But we wanted to create something that could be a little united for all 
applications, without rewrite of applications. We add a lot of customized 
apps, each having been hacked to do something like this, but this resulted 
in a huge mess, lots of different configuration files, and no 
interoperability. By pushing it in the kernel, everything is _simple_ and 
_united_.

Sylvain

Re: [Lse-tech] Re: CPUSET Proposal

[Hubertus Franke]

Sylvain Jeaugey wrote:

>On Thu, 25 Sep 2003, Hubertus Franke wrote:
>
>  
>
>>Again, I see cpusets and CKRM as addressing two orthogonal issues wrt to 
>>cpu's
>>
>>cpusets  (partitioning in space)   with hierarchies
>>CKRM  (time partitioning)         how much of  time does a class get...
>>    
>>
>We do agree. We took a look at CKRM and that is the conclusion we 
>achieved. At first sight, it could look like the goal are the same -and in 
>some points it is- but the two approaches are different. It looks 
>like it would be better to combine them rather to try to merge them.
>
>Sylvain
>
>
>  
>
Correct.  These are both worthwhile efforts and they do different things.
A combination of  both at some point (not now) should be investigated.
On the CPU front, which cpusets at this point provide its simply orthogonal.

cpusets provide you means to lock process down to cpus through some
abstraction/virtual layer that does not determine the exact cpu but 
guarantees that
some cpu will be choosen to represent that number.

This is analogous to MPI applications which provide communicators which 
effectively
are "cpusets" in the broader sense. On top of that they provide topology 
information
as such..

Actually I don't see why CKRM can't enforce class shares on top of cpu sets.
They simply don't need to know about each others presense. CKRM through
its loadbalancing algorithm enforces shares for SMPs while at the same time
observes cpu_affinity constraints, which effectively cpusets boil down 
to ....

So  "combine" is the correct wording here....

-- Hubertus Franke   (CKRM team)

Re: [Lse-tech] Re: CPUSET Proposal

[Hubertus Franke]

Sylvain Jeaugey wrote:

>>>With cpusets each application can ask the kernel for 5 cpus
>>>(or a wrapper that invokes the application can do so), and then
>>>the application can happily divide its work between virtual cpus
>>>0, 1, 2, 3, 4 (using sched_setaffinity()).  The kernel can be smart
>>>and provide different physical cpus for each of the cpusets.
>>>      
>>>
>Actually, the intelligence is located in the wrapper application. The 
>kernel part of cpusets is given absolute masks. The virtualization is 
>located in the fact that calls to sched_setaffinity() are interpreted 
>within the cpuset, and that /proc/cpuinfo is also modified to reflect 
>available CPUs.
>
>  
>
Exactly ....   virtualization of the virtual cpu number is the trick to 
isolation from a code base
perspective .... it

>>In this case, the applications can get all of current affinity info by
>>calling sched_getaffinity(). Thus, the applications can schedule the
>>processes to different cpus without bothering kernel.
>>    
>>
>The problem is not only to run on a different CPU. We want that the used 
>CPU is decided by the admin and not the application. Otherwise you would 
>have no control on it. It could have been a solution for applications we 
>write like MPI (and that is the first thing we did).
>But we wanted to create something that could be a little united for all 
>applications, without rewrite of applications. We add a lot of customized 
>apps, each having been hacked to do something like this, but this resulted 
>in a huge mess, lots of different configuration files, and no 
>interoperability. By pushing it in the kernel, everything is _simple_ and 
>_united_.
>
>Sylvain
>
>
>  
>
Yipp, the lesson from the MPI ( I was involved in that as the 
implementer of the first IBM MPI implementation
for the IBM SP2) is that these abstractions (such as communicators, 
topologies) really mattered. It allowed application
writers to code algorithms without worrying how many cpu's which cpu, 
who's the next physical neighbor.
It stayed at the virtual level.
cpusets are the same .

Enough on this topic now .. :-)

-- Hubertus Franke  (CKRM  team)