Re: [Lse-tech] [PATCH] cpusets - big numa cpu and memory placement

[Paul Jackson <pj@sgi.com>]

Erich, responding to Hubertus:
> > (a) is it a guarantee/property that cpusets at with the same
> >      parent cpuset do not overlap ?
> 
> Right now it isn't AFAIK. Paul, if all cpusets on the same level are
> disjunct this certainly simplifies life. Would this be a too strong
> limitation for you? We could live with it.

Correct, Erich, it is not a guarantee that sibling cpusets don't
overlap, unless, as Simon noted, they are all marked exclusive.

Yes, it would be a stronger limitation than I would agree to, but that's
ok, because in my humble opinion, CKRM doesn't need it to operate within
cpusets.

I think what's needed for CKRM to operate within cpusets is clear
ownership.

Each instance of CKRM needs (tell me if I'm wrong here):
 1) to have a clear and unambiguous answer to the question of
    which CPUs, which Memory Nodes, and which Tasks it is
    controlling,
 2) no overlap of these sets with another instance of CKRM,
 3) the CPUs and Memory Nodes on which any of these Tasks are
    allowed to run must be a subset of those controlled by
    this instance of CKRM, and
 4) all Tasks allowed to run on any of the CPUs and Memory
    Nodes controlled by this CKRM instance are in the list
    of Tasks this CKRM knows it controls.

In short - each CKRM instance needs clear, unambiguous, non-overlapping
ownership of all it surveys.

Requesting that all cpusets be marked exclusive for both CPU and Memory
is an overzealous precondition for the above.

Another way to obtain the above requirements would be to assign each
CKRM instance to a separate cpuset subtree, where the root of the
subtree is marked exclusive for cpu and memory, where that CKRM instance
controls all CPUs and Memory owned by that subtree and all Tasks
attached to any cpuset in that subtree, and where any tasks attached to
ancestors of the root are either (1) not allowed to use any of the CPUs
and Memory assigned to the subtree, or (2) are both [2a] allowed to use
only some subset of the CPUs and Memory assigned to the subtree and [2b]
are included in the list of tasks to be managed by that CKRM instance.

(The last 4.5 lines above are the special case required to handle the
indigenous per-cpu tasks, such as the migration threads - sorry.)

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373