Re: [Lse-tech] [PATCH] cpusets - big numa cpu and memory placement

[Paul Jackson <pj@sgi.com>]

Matthew wrote:
>   The reason Paul and I decided that they weren't totally reconcilable is 
> because of the memory binding side of the CPUSETs code.

Speak for yourself, Matthew ;).

I agree you that the scheduler experts (I'm not one, nor do I aspire to
be one) may well find that it makes sense someday to better integrate
scheduler domains and cpusets.  It seems a little inefficient on the
surface for scheduler domain code to spend time trying to choose the
best task to run on a CPU, only to find out that the chosen task is not
allowed, because that tasks cpus_allowed does not allow execution on the
intended CPU.  Since in some systems, cpusets will provide a better
indication of the natural clustering of various cpus_allowed values than
a simple boottime hierarchical partitioning of the system, it makes
sense to me that there might be a way to improve the integration of
cpusets and scheduler domains, at least as an option on systems that are
making heavy use of cpusets.  This might have the downside of making
sched domains more dynamic than they are now, which might cost more
performance than it gained.  Others will have to evaluate those
tradeoffs.

But when you write the phrase "they weren't totally reconcilable,"
I presume you mean "cpusets and CKRM weren't totally reconcilable."

I would come close to turning this phrasing around, and state that
they were (nearly) totally unreconcilable <grin>.

I found no useful and significant basis for integration of cpusets and
CKRM either involving CPU or Memory Node management.

As best as I can figure out, CKRM is a fair share scheduler with a
gussied up more modular architecture, so that the components to track
usage, control (throttle) tasks, and classify tasks are separate
plugins.  I can find no significant and useful overlap on any of these
fronts, either the existing plugins or their infrastructure, with what
cpusets has and needs.

There are claims that CKRM has some generalized resource management
architecture that should be able to handle cpusets needs, but despite my
repeated (albeit not entirely successful) efforts to find documentation
and read source and my pleadings with Matthew and earlier on this
thread, I was never able to figure out what this meant, or find anything
that could profitably integrate with cpusets.

In sum -- I see a potential for useful integration of cpusets and
scheduler domains, I'll have to leave it up to those with expertise in
the scheduler to evaluate and perhaps accomplish this.  I do not see any
useful integration of cpusets and CKRM.

I continue to be befuddled as to why, Matthew, you confound potential
cpuset-scheddomain integration with potential cpuset-CKRM integration.
Scheduler domains and CKRM are distinct beasts, in my book, and the
contemplations of cpuset integration with these two beasts are also
distinct efforts.

And cpusets and CKRM are distinct beasts.

But I repeat myself ...

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.650.933.1373, 1.925.600.0401