Re: [Lse-tech] [PATCH] cpusets - big numa cpu and memory placement

Martin J. Bligh
2004-10-03T23:48:48Z

--Paul Jackson <pj@sgi.com> wrote (on Sunday, October 03, 2004 09:02:09 -0700):

> Martin wrote:
>> The way cpusets uses the current cpus_allowed mechanism is, to me, the most
>> worrying thing about it. Frankly, the cpus_allowed thing is kind of tacked
>> onto the existing scheduler, and not at all integrated into it, and doesn't
>> work well if you use it heavily (eg bind all the processes to a few CPUs,
>> and watch the rest of the system kill itself).
>
> True. One detail of what you say I'm unclear on -- how will the rest of
> the system kill itself? Why wouldn't the unemployed CPUs just idle
> around, waiting for something to do?

I think last time I looked they just sat there saying:

Rebalance!
Ooooh, CPU 3 over there looks heavily loaded, I'll steal something.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
Humpf. I give up.
Rebalance!
Ooooh, CPU 3 over there looks heavily loaded, I'll steal something.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
Humpf. I give up.
Rebalance!
Ooooh, CPU 3 over there looks heavily loaded, I'll steal something.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
That one. Try to migrate. Oops, no cpus_allowed bars me.
Humpf. I give up.
... ad infinitum.

Desperately boring, and rather ineffective.

> As I recall, Ingo added task->cpus_allowed for the Tux in-kernel web
> server a few years back, and I piggy backed the cpuset stuff on that, to
> keep my patch size small.
>
> Likely your same concerns apply to the task->mems_allowed field that
> I added, in the same fashion, in my cpuset patch of recent.

Mmm, I'm less concerned about that one, or at least I can't specifically
see how it breaks.

> We need a mechanism that the cpuset apparatus respects that maps each
> CPU to a sched_domain, exactly one sched_domain for any given CPU at any
> point in time, regardless of which task it is considering running at the
> moment. Somewhat like dual-channeled disks, having more than one
> sched_domain apply at the same time to a given CPU leads to confusions
> best avoided unless desparately needed.

Agreed. The cpus_allowed mechanism doesn't seem well suited to heavy use
anyway (I think John Hawkes had problems with it too). That's not your
fault ... but I'm not convinced it's a good foundation to be building
further things on either ;-)

M.