Re: [Lse-tech] [PATCH] cpusets - big numa cpu and memory placement

Simon Derr
2004-10-05T09:31:49Z

On Mon, 4 Oct 2004, Martin J. Bligh wrote:

> OK, then your "exclusive" cpusets aren't really exclusive at all, since
> they have other stuff running in them. The fact that you may institute
> the stuff early enough to avoid most things falling into this doesn't
> really solve the problems, AFAICS.

I'd like to present you at this point what was the original decision for
having exclusive (called strict, at this point in history) and
non-exclusive cpusets.

The idea was to have a system, and run all jobs on it through a batch
scheduler. Some jobs cared about performance, some didn't.

The ones who cared about performance got an 'exclusive' cpuset, the ones
who didn't got a 'non exclusive' cpuset.

Now there is a possibility, that at a given time, only 'exclusive' jobs
are running, and hence that 'exclusive' cpusets have been created for jobs
on all the CPUs.

Our system (at Bull) is both a big and a small machine:
-big: we have NUMA constraints.
-small: we don't have enough CPUs to spare one, we need to use ALL CPUs
for our jobs.

There are still processes running outside the job cpusets (i.e in the root
cpuset), sshd, the batch scheduler. These tasks use a low amount of CPU,
so it is okay if they happen to run inside even 'exclusive' cpusets. For
us, 'exclusive' only means that no other CPU-hungry job is going to share
our CPU.

Of course, in our case, a valid argument is that 'exclusiveness' should
not be enforced by the kernel but rather by the job scheduler. Probably.

But now I see that the discussion is going towards:
-fully exclusive cpusets, maybe even with no interrupts handling
-maybe only allow exclusive cpusets, since non-exclusive cpusets are
tricky wrt CKRM.

That would be a no-go for us.

Simon.