Re: [Lse-tech] [PATCH] cpusets - big numa cpu and memory placement

[Jack Steiner <steiner@sgi.com>]

On Thu, Aug 12, 2004 at 12:15:22AM -0700, Paul Jackson wrote:
> Shailabh wrote:
> > But when apps are being confined to a set of cpus *only* for purposes of 
> > getting a certain fraction of the total compute power, cpusets are not 
> > orthogonal in intent, not implementation, from a CKRM CPU class 
> > implementing hard limits.
> 
> So if someone wanted to constrain a group of tasks to using 50% of all
> the available CPU ticks on a 32 CPU system, they could either use a CKRM
> CPU class with a hard limit of 50%, or a cpuset that contained 16 CPUs.
> 
> Yes, for that purpose, except for NUMA placement (the cache affinity and
> memory latency you mention), these two approaches are similar in affect.

One other important attribute of a cpuset is that, used properly, cpusets
will guarantee exclusive use of a set of cpus for an application.

MPI jobs are frequently consist of a number of threads that communicate
via message passing interfaces. All threads need to be executing
at the same time. If a single thread loses a cpu, all threads stop making
forward progress and spin at a barrier.

Cpusets can eliminate the need for a gang scheduler.


> 
> So, yes, my absolute insistence that CKRM and cpusets are orthogonal is
> overstated.  Well, I could quibble that orthogonal doesn't imply disjoint.
> Whatever.
> 
> 
> > What's your opinion on the commonalities between the two interfaces 
> > pointed out in my previous mail ?
> 
> My apologies for not yet replying to your mail of a couple of days ago.
> It was valuable to me, and I've taken a bit of time to digest it.
> Meanwhile, newer stuff keeps overruning my reply.  Soon, hopefully.
> 
> 
> > Also, if CKRM were to move to the "each controller exports its own 
> > interface" model, how would this affect the discussion ?
> 
> I cannot speak for the discussion, only for myself.  I am clearly
> sensitive to the downsides of trying to integrate these interfaces.
> 
> Hopefully I can find the time tonight to study your earlier replies more
> closely, and better understand the potential benefits of such an
> integration.  So far, I don't see them.  I will do my best to keep my
> eyes and mind open.  Thanks especially to your posts, I have learned
> quite a bit about CKRM this week.
> 
> I will confess to a strong bias toward a minimum of abstraction at the
> kernel-user boundary, and towards providing a one-to-one map between the
> mechanisms and the interfaces a kernel provides.  Let the user level
> assemble the pieces as it will.  If combining interfaces (CKRM and
> cpuset) caused any unwarranted change in or obfuscation of the semantics
> provided by either, that would be unfortunate, in my view.
> 
> 
> > Do you think there's *any* merit to cpusets sharing the rcfs 
> > interface *if* the latter were to make the changes mentioned in earlier 
> > mail ?
> 
> Not yet, but I need to go back over your replies, and others, with this
> question more clearly in focus.
> 
> 
> > If not (and others agree), lets end this discussion and move on - both 
> > projects have enough to do ...
> 
> At least Martin does not yet agree, if I understand his posts.  But,
> yes, either way, we are close to where it is best to table this
> discussion, for the moment at least.
> 
> Thank-you for your constructive and enlightening comments so far.
> 
> -- 
>                           I won't rest till it's the best ...
>                           Programmer, Linux Scalability
>                           Paul Jackson <pj@sgi.com> 1.650.933.1373

-- 
Thanks

Jack Steiner (steiner@sgi.com)          651-683-5302
Principal Engineer                      SGI - Silicon Graphics, Inc.