Minutes from 5/19 CKRM/PAGG discussion

Minutes from 5/19 CKRM/PAGG discussion

[Hanna Linder]

Minutes from LSE call on CKRM and PAGG on May 19, 2004. 

Here are the websites for the respective projects:

http://ckrm.sf.net
http://sourceforge.net/projects/ckrm/
http://oss.sgi.com/projects/pagg
http://oss.sgi.com/projects/csa
http://elsa.sf.net

Conclusion:

        CSA/PAGG look at CKRM
        CKRM look at PAGG

Shailabh presented a short overview of CKRM.
	- for the latest design description read LinuxTAG paper
	- working prototype on ckrm.sf.net
	- Major concepts:
		-> Class type refers to kernel objects being controlled
		-> Class -> grouping of objects	
			e.g. task_class is equivalent to a job 
			container which is a collection of tasks
			e.g. socket_class
		-> Resource Controller
			-> control and modifications to basic schedulers
				- enforce resource shares
				- gather statistics
		-> Classification Engine
			-> automatic classification of kernel objects 
				into classes
		-> RCFS:
			- a file system interface 
			- Class types are top level directories
			- Classes are sub-directories under the 
			  class types
			- resource controllers show up as lines in the 
			  files contained in the directories
		-> flexible and generic
			- Modular. One can include only what one 
			needs and/or prefers.
			- development of various components is 
			independent of others
		-> one run queue per class
			-> controllers need read write
			-> hierarchical sub classes requires rewrite
		-> monitoring module
			-> class level monitoring module
			-> per class monitoring to be developed fully
		-> flexibility to the user
			-> dynamic grouping defined by the user.
			-> emphasis on rapidly moving from one class 
			   to another 
		-> performance impact of hierarchies not looked at.
			-> if hiearrchies in kernel is costly then we 
			   will move to flat hierarchies.
		-> working prototype of all 

Discussion on hierarchies:
(could not map all voices to names...also might have guessed wrong.)

	-> Rik: hierarchical schedulers exist in the kernel. 
		Does have hierarchical network schedulers
	-> if you go hierarchical cpu scheduler will not scale 
	-> you don't have to check limits for each walk. 
	   keep track of usage and once in a while update it.
			(edit: lazy update and lazy evaluation)
	-> you have to take walks -- potentially every scheduling decision 
	   might becoming serialised. -- peter Williams....
	-> instead do it for your hierarchies once in a while take a global view.
	->Shailabh: Such evaluation might introduce inaccuracies in share 
	 enforcement. Therefore it is a balance between accurate enforcement 
	of share vs. hierarchy.
		(edit: maybe a hierarchy only a few levels deep -upto 3/4-
		 will be accurate enough??)
	-> then do it in user space.
	-> leave the scheduler alone
		-> do it from the outside -- from kernel module or user space
	-> Gerrit: NUMA hierarchy/hyper threading is a heirarchy. 
		Kernel scheduler has NUMA hooks
	-> NUMA -- only load balancing not scheduling -- 
	-> Rik: class is similar  -- only once in a while
		-> need not be in the main scheduler

	------> might be implementable in differnet ways. Need to discuss in
		more detail.
	
Overview of PAGG:
	process aggregate
	kernel modules to group process together
	child will inherit the pagg and notify the kernel module
	
(John)	PAGG is simple...way of grouping process together. 
	have multiple simultaneous groupings.
	first modules is job module.
		- implements an inescapable set of processes
		- can't change job ids
	- working on code that implements cluster wide ids
	- processes that are related from the standpoint of 
	  NUMA placement
	-> associate process beyond process groups is very useful.
	->  There is similarity between PAGG and task_class
	-> Except a task can belong to different jobs	
	-> CSA on top of PAGG are like resource controllers.
	-> PAGG is more flexibile in one dimension
		-> how much resource is being used
		-> memory placement
	-> PAGG is more flexible due to this
	-> CKRM's objective is resource management
		-> choose your controllers
	-> then all controllers are active
	-> can have classtypes with different controllers
	-> PAGG Structure: Attach and detach functions and private data
	-> ... Makes it easy to load/unload a module.
	-> What was the real motivation for PAGG?
	-> multiple cases where we want to group processes
	-> What is using PAGG?
		->job container
		-> CSA using job container
		-> Working on:
			cluster wide collection of process
	-> What is the kernel impact of cluster wide id
		-> Unique id is needed for the cluster group
			-> cleanup reason ..track down all processes 
	->For NUMA placement
		- CPU Memset does work with it
		- has intenions of using it
		(edit: didn't catch if using if at present).
	-> How much is CSA code is all by itself?
		->  ...in theory independent but not in current 
			implementation. Need work. 
		Three ways:
		-> PAGG + job container
		-> PAGG + job container + CSA patch
		-> PAGG + NUMA
	-> What is required to integrate? 
		-> CKRM--- It is for resource control
	-> it is not all or nothing.
	-> does not allow NUMA grouping vs. resource grouping
	-> Question: Can one implement job container function using CKRM
	-> Answer: Take CKRM core patch -- write your own CE -- write your 
		set of fucntion hooks load your CE and that is all you need.
	-> Why are you against PAGG? Implementing on top of PAGG will reduce 
	the work CKRM needs?  Won't need to implement job container part.

	-> Not against. Trying to find out exploiters of PAGG functionlity:

	-> Does PAGG have capability to get notified on events...
	-> Callback? that adds setuid/setgid etc. 
	-> yes, just added...
	-> clone?
	-> go through fork

	-> What is the use of setuid/setgid for PAGG?
	-> affinity yes

Combining? Coming up with one solution:
	-> Break CKRM into small pieces...
	-> perception issue of breaking up the CKRM.... had 6 differnet patches
	-> Breaking up the patches...changes things
	-> Conceptually done it separately...but not presented it as well.
	-> (Suse: Chris Mason?) people really identify the common functionality.
	    hard to pick from among you.

	-> Something that is not going to change choose interfaces for 
	   userland and kernel
	->(Gerrit) Have opensource applications that use them
	-> PAGG/Peter? 
		Don't need to invent new ones but need to come together on it.	
	-> Shailabh: kernel - user interface e.g. /rcfs

	-> whether CPUMEMSETS and CKRM can be combined?
	- Paul Jackson : Not focussed on a particular technology
		CKRM purposes are distinct from NUMA purposes. Independent goals
		
		Have plans to use PAGG's lowest level hooks in CPUMEMSET.

	-> once the hook is there anyone can use it..

	- if it is useful and does't cause harm then use it.

	-> Hanna : discuss on lse-tech@lists.sf.net.
	-> can use pagg list
	-> Hanna/Gerrit : Talking about interop - scaling from desktop to 
		enterprise. 
	<Agreement on lse-tech...post minutes on lse-tech>

	-> Join forces and come up with a plan
	-> If PAGG functionality to include what CKRM needs.  
		- Expanding the goal makes it more complex....
		- harder to get in
		- performace implications

	-> Shailabh: CKRM will consider looking at the lowest level of PAGG 
			- Not committing to use it
			- Evaluating it

	Suse ? --- certainly willing to consolidate these projects.

	Ralf -- CKRM is in beta5
		- Looks like the teams can benefit from each other.
		- need to consider if CKRM needs to be pulled out now and
		  added later. 

	-> What are the users of PAGG other than CSA? at least one other user?
		-> (Peter) ARMTech  solaris and windows 
		-> Is it opensourced? 
		-> (Peter) should be somewhere....
		- (Peter?)Will look at CKRM hooks
		-> (Paul Jackson) CPUMEMSETS. don't know the timeframe 

Conclusion:

	CSA/PAGG look at CKRM 
	CKRM look at PAGG

Re: Minutes from 5/19 CKRM/PAGG discussion

[Hanna Linder]

--On Monday, May 24, 2004 10:09:07 AM -0700 Hanna Linder <hannal@us.ibm.com> wrote:

> 
> Minutes from LSE call on CKRM and PAGG on May 19, 2004. 

I forgot to give credit to Vivek Kashyap for writing the minutes. 

Thanks a lot!

Hanna

Re: Minutes from 5/19 CKRM/PAGG discussion

[Rik van Riel]

On Mon, 24 May 2004, Hanna Linder wrote:

> Minutes from LSE call on CKRM and PAGG on May 19, 2004. 

Thanks for organising this call, Hanna!

> Conclusion:
> 
> 	CSA/PAGG look at CKRM 
> 	CKRM look at PAGG

I really hope the projects will be able to agree on one
framework.  As long as there are multiple competing
frameworks the chance of any of them being merged into
the upstream kernel is exceedingly low...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: Minutes from 5/19 CKRM/PAGG discussion

[Hubertus Franke]

Rik van Riel wrote:

>On Mon, 24 May 2004, Hanna Linder wrote:
>
>  
>
>>Minutes from LSE call on CKRM and PAGG on May 19, 2004. 
>>    
>>
>
>Thanks for organising this call, Hanna!
>
Thanks 2.. sorry I couldn't make it ...

>
>  
>
>>Conclusion:
>>
>>	CSA/PAGG look at CKRM 
>>	CKRM look at PAGG
>>    
>>
>
>I really hope the projects will be able to agree on one
>framework.  As long as there are multiple competing
>frameworks the chance of any of them being merged into
>the upstream kernel is exceedingly low...
>
>  
>
Agreed, so let's get the ball rolling.

We (CKRM team) will look at PAGG (again) ... Actually we did
last year and basically assumed due to the inactivity that
PAGG was not actively persued anymore.

I believe at our end we will come to the conclusion that CKRM can serve
as the base for CSA, as PAGG seems to be lowest level layer of
this management silo. The level CSA would hook to is that of
the CKRM event hooking which is part of the CKRM core.

One important input the PAGG team could give is some real
examples where actually multiple associations to different groups
is required and help us appreciate that position and let us
see how this would/could be done in CKRM.

 From our point of view we don't see this requirement. In contrast
we use the modified rbce (CRBCE) to push the "interesting" kernel events
to userspace where any kind of accounting aggregation can take place.
Yet, we believe the integrated resource scheduling (e.g. cpu) will
always happen at the dominant class - object association in the kernel.

Another point that has not been made is that CKRM's philosophy
is to manage any kind of objects wrt to some class type.
By definition, PAGG is a Process Aggregation, which is a subset
of what CKRM needs namely (obj->class) associations.

In our hooking scheme we therefore provide the ability to attach
to so called kernel events a callback function. Any kernel code
can attach a callback function. This is part of our core.

Any classtype (not part of the core) can register a callback at
any of those events, so typically only limited
events are "hooked" for a particular type. Regardless, we have
function stacking, rather then object stacking.

PAGG by itself manages the proper association of a (or several)
"transparent" group object with the task. The functionality
hidden behind the group object
still needs to be implemented by the group object itself.

In CKRM this is similar, yet, the class object is associated with
a particular class type. All interactions with the user component
and classification engine are architected by the higher layers of CKRM,
in that classes have automatic representation in RCFS and RBCE if
those are loaded.

So here are some of the stickling points, we need to work on ...

(a) how can PAGG be made general enough so we can provide generic
       KernelObject <-> ClassObject associations .. not just tasks groups.

(b) Can CSA use the extended rbce (CRBCE) instead of PAGG to
       do its accounting ?

-- Hubertus Franke



 

Re: Minutes from 5/19 CKRM/PAGG discussion

[Peter Williams]

Hubertus Franke wrote:
> Rik van Riel wrote:
> 
>> On Mon, 24 May 2004, Hanna Linder wrote:
>>
>>  
>>
>>> Minutes from LSE call on CKRM and PAGG on May 19, 2004.   
>>
>>
>> Thanks for organising this call, Hanna!
>>
> Thanks 2.. sorry I couldn't make it ...
> 
>>
>>  
>>
>>> Conclusion:
>>>
>>>     CSA/PAGG look at CKRM     CKRM look at PAGG
>>>   
>>
>>
>> I really hope the projects will be able to agree on one
>> framework.  As long as there are multiple competing
>> frameworks the chance of any of them being merged into
>> the upstream kernel is exceedingly low...
>>
>>  
>>
> Agreed, so let's get the ball rolling.
> 
> We (CKRM team) will look at PAGG (again) ... Actually we did
> last year and basically assumed due to the inactivity that
> PAGG was not actively persued anymore.
> 
> I believe at our end we will come to the conclusion that CKRM can serve
> as the base for CSA, as PAGG seems to be lowest level layer of
> this management silo. The level CSA would hook to is that of
> the CKRM event hooking which is part of the CKRM core.
> 
> One important input the PAGG team could give is some real
> examples where actually multiple associations to different groups
> is required and help us appreciate that position and let us
> see how this would/could be done in CKRM.
> 
>  From our point of view we don't see this requirement. In contrast
> we use the modified rbce (CRBCE) to push the "interesting" kernel events
> to userspace where any kind of accounting aggregation can take place.
> Yet, we believe the integrated resource scheduling (e.g. cpu) will
> always happen at the dominant class - object association in the kernel.
> 
> Another point that has not been made is that CKRM's philosophy
> is to manage any kind of objects wrt to some class type.
> By definition, PAGG is a Process Aggregation, which is a subset
> of what CKRM needs namely (obj->class) associations.
> 
> In our hooking scheme we therefore provide the ability to attach
> to so called kernel events a callback function. Any kernel code
> can attach a callback function. This is part of our core.
> 
> Any classtype (not part of the core) can register a callback at
> any of those events, so typically only limited
> events are "hooked" for a particular type. Regardless, we have
> function stacking, rather then object stacking.

I'm assuming that this means several clients can use the same callback 
simultaneously?

> 
> PAGG by itself manages the proper association of a (or several)
> "transparent" group object with the task. The functionality
> hidden behind the group object
> still needs to be implemented by the group object itself.
> 
> In CKRM this is similar, yet, the class object is associated with
> a particular class type. All interactions with the user component
> and classification engine are architected by the higher layers of CKRM,
> in that classes have automatic representation in RCFS and RBCE if
> those are loaded.
> 
> So here are some of the stickling points, we need to work on ...
> 
> (a) how can PAGG be made general enough so we can provide generic
>       KernelObject <-> ClassObject associations .. not just tasks groups.

I would suggest that rather than extending PAGG, separate mechanisms 
similar in design to PAGG be created for each KernelObject for which 
such associations are required.  Perhaps the generic component of such 
mechanisms could be separated out into a generic package and then each 
specific mechanism just provides the specialization for its KernelObject 
type.  I'd imagine that the main difference between the specialized 
packages would be (apart from the different type of KernelObject) in the 
set of callbacks provided.

> 
> (b) Can CSA use the extended rbce (CRBCE) instead of PAGG to
>       do its accounting ?

 From my (possibly incorrect) understanding of the above description, 
one thing that PAGG provides to its clients that CKRM doesn't is the 
ability to attach some private data to task structs and it passes that 
data to the client as part of the callback.  Am I correct in this 
interpretation?

Peter
-- 
Dr Peter Williams, Chief Scientist                peterw@aurema.com
Aurema Pty Limited                                Tel:+61 2 9698 2322
PO Box 305, Strawberry Hills NSW 2012, Australia  Fax:+61 2 9699 9174
79 Myrtle Street, Chippendale NSW 2008, Australia http://www.aurema.com

Re: Minutes from 5/19 CKRM/PAGG discussion

[Peter Williams]

Hubertus Franke wrote:
> 
> One important input the PAGG team could give is some real
> examples where actually multiple associations to different groups
> is required and help us appreciate that position and let us
> see how this would/could be done in CKRM.

One example would be the implementation of CPU sets (or pools) a la 
Solaris where there are named CPU pools to which processors and 
processes are assigned.   Processors can be moved between CPU pools and 
when this happens it is necessary to visit all the processes that are 
assigned to the pools involved (one losing and one gaining the 
processor) and change their CPU affinity masks to reflect the new 
assignment of processors.  PAGG would be ideal for implementing this.

At the same time, a resource management client could be controlling 
resources allocated to processes based on some other criteria such as 
the real user or the application being run without regard to which CPU 
pool they are running in.

Peter
-- 
Dr Peter Williams, Chief Scientist                peterw@aurema.com
Aurema Pty Limited                                Tel:+61 2 9698 2322
PO Box 305, Strawberry Hills NSW 2012, Australia  Fax:+61 2 9699 9174
79 Myrtle Street, Chippendale NSW 2008, Australia http://www.aurema.com

Re: Minutes from 5/19 CKRM/PAGG discussion

[Peter Williams]

Peter Williams wrote:
> Hubertus Franke wrote:
> 
>>
>> One important input the PAGG team could give is some real
>> examples where actually multiple associations to different groups
>> is required and help us appreciate that position and let us
>> see how this would/could be done in CKRM.
> 
> 
> One example would be the implementation of CPU sets (or pools) a la 
> Solaris where there are named CPU pools to which processors and 
> processes are assigned.   Processors can be moved between CPU pools and 
> when this happens it is necessary to visit all the processes that are 
> assigned to the pools involved (one losing and one gaining the 
> processor) and change their CPU affinity masks to reflect the new 
> assignment of processors.  PAGG would be ideal for implementing this.
> 
> At the same time, a resource management client could be controlling 
> resources allocated to processes based on some other criteria such as 
> the real user or the application being run without regard to which CPU 
> pool they are running in.

Additionally, it seems to me that even within the field of resource 
management it is not necessarily the case that the same grouping is 
required for different resource types e.g. the grouping for control of 
CPU resources might be different to the grouping for control of network 
bandwidth allocation or disk space or disk I/O bandwidth, etc.

Peter
-- 
Dr Peter Williams, Chief Scientist                peterw@aurema.com
Aurema Pty Limited                                Tel:+61 2 9698 2322
PO Box 305, Strawberry Hills NSW 2012, Australia  Fax:+61 2 9699 9174
79 Myrtle Street, Chippendale NSW 2008, Australia http://www.aurema.com

Re: Minutes from 5/19 CKRM/PAGG discussion

[Hubertus Franke]

Peter Williams wrote:

> Hubertus Franke wrote:
>
>>
>> In our hooking scheme we therefore provide the ability to attach
>> to so called kernel events a callback function. Any kernel code
>> can attach a callback function. This is part of our core.
>>
>> Any classtype (not part of the core) can register a callback at
>> any of those events, so typically only limited
>> events are "hooked" for a particular type. Regardless, we have
>> function stacking, rather then object stacking.
>
>
> I'm assuming that this means several clients can use the same callback 
> simultaneously?

Yes it means exactly that ....

>
>>
>> PAGG by itself manages the proper association of a (or several)
>> "transparent" group object with the task. The functionality
>> hidden behind the group object
>> still needs to be implemented by the group object itself.
>>
>> In CKRM this is similar, yet, the class object is associated with
>> a particular class type. All interactions with the user component
>> and classification engine are architected by the higher layers of CKRM,
>> in that classes have automatic representation in RCFS and RBCE if
>> those are loaded.
>>
>> So here are some of the stickling points, we need to work on ...
>>
>> (a) how can PAGG be made general enough so we can provide generic
>>       KernelObject <-> ClassObject associations .. not just tasks 
>> groups.
>
>
> I would suggest that rather than extending PAGG, separate mechanisms 
> similar in design to PAGG be created for each KernelObject for which 
> such associations are required.  Perhaps the generic component of such 
> mechanisms could be separated out into a generic package and then each 
> specific mechanism just provides the specialization for its 
> KernelObject type.  I'd imagine that the main difference between the 
> specialized packages would be (apart from the different type of 
> KernelObject) in the set of callbacks provided.
>
Yes, and that has been done. Currently we have two such classtype  
(class task and network class) both supplied as independent "modules" 
atop the ckrm core.
And you are right again wrt the difference aspect. Each class registers 
the callbacks its interested in.

>>
>> (b) Can CSA use the extended rbce (CRBCE) instead of PAGG to
>>       do its accounting ?
>
>
> From my (possibly incorrect) understanding of the above description, 
> one thing that PAGG provides to its clients that CKRM doesn't is the 
> ability to attach some private data to task structs and it passes that 
> data to the client as part of the callback.  Am I correct in this 
> interpretation?
>
> Peter

That is the "stickling" point. Yes, PAGG provides this feature that one 
can chain private data to the attach/detach callback. CKRM at this point 
does not do that as we do not see the need for multiple class 
associations in the core. Instead we can drive such things through the 
extended RBCE interface. Here you register callbacks to the task 
classtype to be notified of the ckrm events.

Since we do networking, PAGG is not sufficient for us as it only deals 
with processes. Hence we need our generic infrastructure at the core 
level. Sure we can try to modularize further to take the CKRM EVENTS 
out. Then potentially one could implement task types on top of PAGG on 
top of CKRM Events (which are needed anyway for other the task class 
associations), but then again PAGG brings nothing but another indirections.

It worthwhile to consider to bite the bullet and allow PAGG to enter its 
task class association chain (1 word) and allow CKRM its own. CKRM is 
going after the integrated resource schedulers, PAGG/CSA (afaik) does not.

-- Hubertus

Re: [Lse-tech] Re: Minutes from 5/19 CKRM/PAGG discussion

[Hubertus Franke]

Peter Williams wrote:

> Hubertus Franke wrote:
>
>>
>> One important input the PAGG team could give is some real
>> examples where actually multiple associations to different groups
>> is required and help us appreciate that position and let us
>> see how this would/could be done in CKRM.
>
>
> One example would be the implementation of CPU sets (or pools) a la 
> Solaris where there are named CPU pools to which processors and 
> processes are assigned.   Processors can be moved between CPU pools 
> and when this happens it is necessary to visit all the processes that 
> are assigned to the pools involved (one losing and one gaining the 
> processor) and change their CPU affinity masks to reflect the new 
> assignment of processors.  PAGG would be ideal for implementing this.
>
> At the same time, a resource management client could be controlling 
> resources allocated to processes based on some other criteria such as 
> the real user or the application being run without regard to which CPU 
> pool they are running in.
>
> Peter

Good one, at question though is again though is whether the very 
communalities warrant some realignment. We want to hook into the base 
schedulers and be the clearing house or umbrella to consolidate all the 
ideas, such as the well defined RCFS recently introduced together with 
Rik van Riel. PAGG is as stated a way of doing things outside the core 
kernel and hookable schedulers have been several times rejected at the 
lkml base.

One potential is to agree that the communalities are so few that it 
makes sense to continue with both approaches independently.

-- Hubertus

Re: Minutes from 5/19 CKRM/PAGG discussion

[Hubertus Franke]

Peter Williams wrote:

> Peter Williams wrote:
>
>> Hubertus Franke wrote:
>>
>>>
>>> One important input the PAGG team could give is some real
>>> examples where actually multiple associations to different groups
>>> is required and help us appreciate that position and let us
>>> see how this would/could be done in CKRM.
>>
>>
>>
>> One example would be the implementation of CPU sets (or pools) a la 
>> Solaris where there are named CPU pools to which processors and 
>> processes are assigned.   Processors can be moved between CPU pools 
>> and when this happens it is necessary to visit all the processes that 
>> are assigned to the pools involved (one losing and one gaining the 
>> processor) and change their CPU affinity masks to reflect the new 
>> assignment of processors.  PAGG would be ideal for implementing this.
>>
>> At the same time, a resource management client could be controlling 
>> resources allocated to processes based on some other criteria such as 
>> the real user or the application being run without regard to which 
>> CPU pool they are running in.
>
>
> Additionally, it seems to me that even within the field of resource 
> management it is not necessarily the case that the same grouping is 
> required for different resource types e.g. the grouping for control of 
> CPU resources might be different to the grouping for control of 
> network bandwidth allocation or disk space or disk I/O bandwidth, etc.
>
> Peter

Correct, that is in principle possible. We went through this discussion 
on the mailing list  .. (Rik, Shailabh help me out here) and decided 
that mostly that is
not being required. The extra overhead of putting that we deemed 
unnecessary.

Instead, if really desired, one would require a different classtype for 
each of those groupings. Note however, our entire infrastructure already 
supports that in that RCFS would pick these specialized class types up, 
provide the proper hierarchy as an FS subdirectory structure. For us it 
was mainly a question how we can get the most out of the initial 
prototype without having general users have to go and specify all kinds 
of hierarchies, i.e. one per resource.

-- Hubertus

Re: Minutes from 5/19 CKRM/PAGG discussion

[Peter Williams]

Hubertus Franke wrote:
> Peter Williams wrote:
>> From my (possibly incorrect) understanding of the above description, 
>> one thing that PAGG provides to its clients that CKRM doesn't is the 
>> ability to attach some private data to task structs and it passes that 
>> data to the client as part of the callback.  Am I correct in this 
>> interpretation?
>>
>> Peter
> 
> 
> That is the "stickling" point. Yes, PAGG provides this feature that one 
> can chain private data to the attach/detach callback. CKRM at this point 
> does not do that as we do not see the need for multiple class 
> associations in the core.

I think that you are looking at this issue too much from a CKRM point of 
view.  I.e. just because CKRM doesn't need it doesn't mean that it isn't 
  a good idea.  In fact the issue should be viewed more broadly than 
just a "resource management" point of view.

If there are multiple clients then having their per KernelObject data 
managed using the PAGG mechanism greatly simplifies the task of 
implementing a client AND reduces the potential overhead on the system 
as the alternative is for the client to use some type of search 
mechanism to find its copy of its per KernelObject specific data when 
servicing its callback functions.

> Instead we can drive such things through the 
> extended RBCE interface. Here you register callbacks to the task 
> classtype to be notified of the ckrm events.
> 
> Since we do networking, PAGG is not sufficient for us as it only deals 
> with processes.

I think that this is just a detail and that what should happen is that a 
PAGG like mechanism be applied to sockets.  Similarly, to enable memory 
management/monitoring one attached to address space structures would be 
useful.  And so on.

> Hence we need our generic infrastructure at the core 
> level. Sure we can try to modularize further to take the CKRM EVENTS 
> out.

I think that breaking these up into smaller chunks (based on the type of 
KernelObject to which they apply) would be a good idea.  The fact that 
CKRM wants to use them all isn't sufficient justification to lump them 
all together.

> Then potentially one could implement task types on top of PAGG on 
> top of CKRM Events (which are needed anyway for other the task class 
> associations), but then again PAGG brings nothing but another indirections.
> 
> It worthwhile to consider to bite the bullet and allow PAGG to enter its 
> task class association chain (1 word) and allow CKRM its own. CKRM is 
> going after the integrated resource schedulers, PAGG/CSA (afaik) does not.

I think that this is another example of you taking a too CKRM centric 
point of view.  What I'm trying to say is that I think that these lower 
level interfaces need to be more independent of CKRM's requirements.

Peter
-- 
Dr Peter Williams, Chief Scientist                peterw@aurema.com
Aurema Pty Limited                                Tel:+61 2 9698 2322
PO Box 305, Strawberry Hills NSW 2012, Australia  Fax:+61 2 9699 9174
79 Myrtle Street, Chippendale NSW 2008, Australia http://www.aurema.com

Re: [Lse-tech] Re: Minutes from 5/19 CKRM/PAGG discussion

[Peter Williams]

Hubertus Franke wrote:
> Peter Williams wrote:
> 
>> Hubertus Franke wrote:
>>
>>>
>>> One important input the PAGG team could give is some real
>>> examples where actually multiple associations to different groups
>>> is required and help us appreciate that position and let us
>>> see how this would/could be done in CKRM.
>>
>>
>>
>> One example would be the implementation of CPU sets (or pools) a la 
>> Solaris where there are named CPU pools to which processors and 
>> processes are assigned.   Processors can be moved between CPU pools 
>> and when this happens it is necessary to visit all the processes that 
>> are assigned to the pools involved (one losing and one gaining the 
>> processor) and change their CPU affinity masks to reflect the new 
>> assignment of processors.  PAGG would be ideal for implementing this.
>>
>> At the same time, a resource management client could be controlling 
>> resources allocated to processes based on some other criteria such as 
>> the real user or the application being run without regard to which CPU 
>> pool they are running in.
>>
>> Peter
> 
> 
> Good one, at question though is again though is whether the very 
> communalities warrant some realignment. We want to hook into the base 
> schedulers and be the clearing house or umbrella to consolidate all the 
> ideas, such as the well defined RCFS recently introduced together with 
> Rik van Riel. PAGG is as stated a way of doing things outside the core 
> kernel and hookable schedulers have been several times rejected at the 
> lkml base.

PAGG does not provide any control capabilities and anyone who wished to 
use PAGG for control would have to use existing control interfaces.  So 
this argument does not apply.

> 
> One potential is to agree that the communalities are so few that it 
> makes sense to continue with both approaches independently.
> 

I don't think that it's that bad.  The problem is that CKRM (which is 
huge) wants to do everything itself (and its way).  In reality, CKRM 
needs a lot of lower level features that don't currently exist and many 
of these features have merit in their own right independent of CKRM's 
need for them.  These lower level features should be separated out from 
CKRM and presented as the (potentially) small and easy to understand 
mechanisms that they are.  Other developers would then be free to use 
these mechanisms as they see fit (probably using them for innovative 
things not envisaged by any of us).  To fully realize their potential 
these features need to (where it's sensible) support multiple 
simultaneous clients and be as efficient and easy to use as possible 
(and some of the ideas present in PAGG would help in this regard).

Peter
-- 
Dr Peter Williams, Chief Scientist                peterw@aurema.com
Aurema Pty Limited                                Tel:+61 2 9698 2322
PO Box 305, Strawberry Hills NSW 2012, Australia  Fax:+61 2 9699 9174
79 Myrtle Street, Chippendale NSW 2008, Australia http://www.aurema.com

Re: Minutes from 5/19 CKRM/PAGG discussion

[Peter Williams]

Hubertus Franke wrote:
> Peter Williams wrote:
> 
>> Peter Williams wrote:
>>
>>> Hubertus Franke wrote:
>>>
>>>>
>>>> One important input the PAGG team could give is some real
>>>> examples where actually multiple associations to different groups
>>>> is required and help us appreciate that position and let us
>>>> see how this would/could be done in CKRM.
>>>
>>>
>>>
>>>
>>> One example would be the implementation of CPU sets (or pools) a la 
>>> Solaris where there are named CPU pools to which processors and 
>>> processes are assigned.   Processors can be moved between CPU pools 
>>> and when this happens it is necessary to visit all the processes that 
>>> are assigned to the pools involved (one losing and one gaining the 
>>> processor) and change their CPU affinity masks to reflect the new 
>>> assignment of processors.  PAGG would be ideal for implementing this.
>>>
>>> At the same time, a resource management client could be controlling 
>>> resources allocated to processes based on some other criteria such as 
>>> the real user or the application being run without regard to which 
>>> CPU pool they are running in.
>>
>>
>>
>> Additionally, it seems to me that even within the field of resource 
>> management it is not necessarily the case that the same grouping is 
>> required for different resource types e.g. the grouping for control of 
>> CPU resources might be different to the grouping for control of 
>> network bandwidth allocation or disk space or disk I/O bandwidth, etc.
>>
>> Peter
> 
> 
> Correct, that is in principle possible. We went through this discussion 
> on the mailing list  .. (Rik, Shailabh help me out here) and decided 
> that mostly that is

Mostly isn't good enough :-)

> not being required. The extra overhead of putting that we deemed 
> unnecessary.

I think that this would be minimal.

> 
> Instead, if really desired, one would require a different classtype for 
> each of those groupings. Note however, our entire infrastructure already 
> supports that in that RCFS would pick these specialized class types up, 
> provide the proper hierarchy as an FS subdirectory structure. For us it 
> was mainly a question how we can get the most out of the initial 
> prototype without having general users have to go and specify all kinds 
> of hierarchies, i.e. one per resource.

If that is what you want (and it's not an unreasonable desire) there is 
nothing to stop CKRM from imposing this view of the world on top but you 
shouldn't be forcing this view on others.  I.e. it's easy to consolidate 
separate grouping mechanisms to have a common grouping for all 
KernelObjects but it would be very close to impossible to try to present 
separate groupings for each KernelObject if there was only one 
underlying grouping.

Peter
-- 
Dr Peter Williams, Chief Scientist                peterw@aurema.com
Aurema Pty Limited                                Tel:+61 2 9698 2322
PO Box 305, Strawberry Hills NSW 2012, Australia  Fax:+61 2 9699 9174
79 Myrtle Street, Chippendale NSW 2008, Australia http://www.aurema.com