Re: [Lse-tech] [PATCH] cpusets - big numa cpu and memory placement

Nick Piggin
2004-10-11T23:19:04Z

Matthew Dobson wrote:
> On Fri, 2004-10-08 at 17:18, Nick Piggin wrote:
>
>>Matthew Dobson wrote:
>>
>>>I think this example is easily achievable with the sched_domains
>>>modifications I am proposing. You can still create your 128 CPU
>>>exclusive domain, called big_domain (due to my lack of naming
>>>creativity), and further divide big_domain into smaller, non-exclusive
>>>sched_domains. We do this all the time, albeit statically at boot time,
>>>with the current sched_domains code. When we create a 4-node domain on
>>>IA64, and underneath it we create 4 1-node domains. We've now
>>>partitioned the system into 4 sched_domains, each containing 4 cpus.
>>>Balancing between these 4 node-level sched_domains is allowed, but can
>>>be disallowed by not setting the SD_LOAD_BALANCE flag. Your example
>>>does show that it can be more than just a convenient way to group tasks,
>>>but your example can be done with what I'm proposing.
>>
>>You wouldn't be able to do this just with sched domains, because
>>it doesn't know anything about individual tasks. As soon as you
>>have some overlap, all your tasks can escape out of your domain.
>>
>>I don't think there is a really nice way to do overlapping sets.
>>Those that want them need to just use cpu affinity for now.
>
>
> Well, the tasks can escape out of the domain iff you have the
> SD_LOAD_BALANCE flag set on that domain. If SD_LOAD_BALANCE isn't set,
> then when the scheduler tick goes off, and the code looks at the domain,
> it will see the lack of the flag and will not attempt to balance the
> domain, correct? This is what we currently do with the 'isolated'
> domains, right?
>

Yeah that's right. Well you have to remove some of the other SD_
flags as well (eg. SD_BALANCE_EXEC, SD_WAKE_BALANCE).

But I don't think there is much point in overlapping sets which
don't do any balancing. They might as well not exist at all.

> You're right that you can get some of the more obscure semantics of the
> various flavors of cpusets by leveraging sched_domains AND
> cpus_allowed. I don't have any desire to remove that ability, just keep
> it as the exception.
>

I think at this stage, overlapping cpu sets are the exception. It
is pretty logical that they're going to require some per-task info,
because the balancer can't otherwise differentiate between two tasks
on the same runqueue but in different cpu sets.

sched-domains gives you a nice clean way to do exclusive partitioning,
and I can't imagine it would be too common to want to do overlapping
partitioning.