Re: [Lse-tech] [PATCH] cpusets - big numa cpu and memory placement

Simon Derr
2004-10-06T08:06:45Z

On Tue, 5 Oct 2004, Matthew Dobson wrote:

> On Sun, 2004-10-03 at 16:53, Martin J. Bligh wrote:
> > > Martin wrote:
> > >> Matt had proposed having a separate sched_domain tree for each cpuset, which
> > >> made a lot of sense, but seemed harder to do in practice because "exclusive"
> > >> in cpusets doesn't really mean exclusive at all.
> > >
> > > See my comments on this from yesterday on this thread.
> > >
> > > I suspect we don't want a distinct sched_domain for each cpuset, but
> > > rather a sched_domain for each of several entire subtrees of the cpuset
> > > hierarchy, such that every CPU is in exactly one such sched domain, even
> > > though it be in several cpusets in that sched_domain.
> >
> > Mmmm. The fundamental problem I think we ran across (just whilst pondering,
> > not in code) was that some things (eg ... init) are bound to ALL cpus (or
> > no cpus, depending how you word it); i.e. they're created before the cpusets
> > are, and are a member of the grand-top-level-uber-master-thingummy.
> >
> > How do you service such processes? That's what I meant by the exclusive
> > domains aren't really exclusive.
> >
> > Perhaps Matt can recall the problems better. I really liked his idea, aside
> > from the small problem that it didn't seem to work ;-)
>
> Well that doesn't seem like a fair statement. It's potentially true,
> but it's really hard to say without an implementation! ;)
>
> I think that the idea behind cpusets is really good, essentially
> creating isolated areas of CPUs and memory for tasks to run
> undisturbed. I feel that the actual implementation, however, is taking
> a wrong approach, because it attempts to use the cpus_allowed mask to
> override the scheduler in the general case. cpus_allowed, in my
> estimation, is meant to be used as the exception, not the rule. If we
> wish to change that, we need to make the scheduler more aware of it, so
> it can do the right thing(tm) in the presence of numerous tasks with
> varying cpus_allowed masks. The other option is to implement cpusets in
> a way that doesn't use cpus_allowed. That is the option that I am
> pursuing.

I like this idea.

The current implementation uses cpus_allowed because it is non-intrusive,
as it does not touch the scheduler at all, and also maybe because it was
easy to do this way since the cpuset development team seems to lack
scheduler gurus.

The 'non intrusive' part was also important as long as the cpusets were
mostly 'on their own', but if now it appears that more cooperation with
other functions such as CKRM is needed, I suppose a deeper impact on the
scheduler code might be OK. Especially if we intend to enforce 'real
exclusive' cpusets or things like that.

So I'm really interested in any design/bits of code that would go in that
direction.

Simon.