Re: [Lse-tech] [PATCH] cpusets - big numa cpu and memory placement

Tim Hockin
2004-10-03T20:15:53Z

On Sun, Oct 03, 2004 at 07:36:46AM -0700, Martin J. Bligh wrote:
> > This is where I see the need for "CPU sets". I.e. as a
> > replacement/modification to the CPU affinity mechanism basically adding
> > an extra level of abstraction to make it easier to use for implementing
> > the type of isolation that people seem to want. I say this because,
> > strictly speaking and as you imply, the current affinity mechanism is
> > sufficient to provide that isolation BUT it would be a huge pain to
> > implement.
>
> The way cpusets uses the current cpus_allowed mechanism is, to me, the most
> worrying thing about it. Frankly, the cpus_allowed thing is kind of tacked
> onto the existing scheduler, and not at all integrated into it, and doesn't
> work well if you use it heavily (eg bind all the processes to a few CPUs,
> and watch the rest of the system kill itself).

7 years ago, before cpus_allowed was dreamed up, I proposed a pset patch
and was shot down hard. Now it's back, and we're trying to find a way to
cram it in on top.

Yeah, it does not fit nicely with cpus_allowed.

I have to ask - do we REALLY need cpusets? I meant, even SGI dropped
PSET at some point, because (if I recall) NO ONE USED IT.

What's the problem being solved that *requires* psets?

I have a customer I work with periodically who was using my pset patch up
until they moved to RH8, when the O(1) scheduler and cpus_allowed changed
everything. This was their requirement for pset:

1. Take a processor out of the general execution pool (call it
PROC_RESTRICTED). This processor will not schedule general tasks.
2. Assign a task to the PROC_RESTRICTED cpu. Now that CPU will only
schedule the assigned task (and it's children).
3. Repeat for every CPU, with the caveat that one CPU must remain
PROC_ENABLED.

I had an array of enum procstate and a new syscall pair:
sched_{gs}etprocstate(). The scheduler checks the procstate, and if it is
not ENABLED, it checks that (cpus_allowed == 1<<cpu). Simple, but works.
Could be baked a bit more, for general use.

What if I proposed a patch like this, now? It would require cleanup for
2.6, but I'm game if it's useful.

Tim