Re: [RFC PATCH] Dynamic sched domains aka Isolated cpusets

[Matthew Dobson <colpatch@us.ibm.com>]

Paul Jackson wrote:
> A few days ago, Nick wrote:
> 
>>Well the scheduler simply can't handle it, so it is not so much a
>>matter of pushing - you simply can't use partitioned domains and
>>meaningfully have a cpuset above them.
> 
> 
> And I (pj) replied:
> 
>>Translating that into cpuset-speak, I think what you mean is ...
> 
> 
> I then went on to ask some questions.  I haven't seen a reply.
> I probably wrote too many words, and you had more pressing matters
> to deal with.  Which is fine.
> 
> Let's make this simpler.
> 
> Ignore cpusets -- let's just talk about a tasks cpus_allowed value,
> and scheduler domains.  Think of cpusets as just a strange way of
> setting a tasks cpus_allowed value.
> 
> Question:
> 
>     What happens if we have say two isolated scheduler domains
>     on a system, covering say two halves of the system, and
>     some task has its cpus_allowed set to allow _all_ CPUs?
> 
> What kind of pain does that cause?  I'm hoping you will say that
> the only pain it causes is that the task will only run on one
> half of the system, even if the other half is idle.  And that
> so long as I don't mind that, it's no problem to do this.

I'm not the sched_domains guru that Nick is, but as your question has gone
unanswered for a few days I'll chime in and see if I can't help you provoke
a more definitive response.

Your assumptions above are correct to the best of my knowledge.  The only
pain it causes is that the scheduler will not be able to "see" outside of
the span of the highest sched_domain attached to a particular CPU.


        A          B
       / \        / \
      X   Y      Z   W
     /\   /\    /\   /\
    0  1 2  3  4  5 6  7

In this setup, with your "Alpha" & "Beta" domains splitting the system in
half, a process in a cpuset spanning cpus 0..7 will get "stuck" in
whichever domain it happens to be in when the Alpha & Beta domains get
created.  Explicit sys_sched_setaffinity() calls will still move it between
domains, but it will not move between Alpha & Beta on its own.
Loadbalancing from CPU 0's perspective (in Alpha) sees only CPUs 0..3.

Right Nick? ;)

-Matt