Re: scheduler scalability - cgroups, cpusets and load-balancing

[Paul Jackson <pj@sgi.com>]

Peter wrote:
> Thanks for the link. Yes I think your last suggestion of creating
> rt-domains ( http://lkml.org/lkml/2007/10/23/419 ) is a good one.

We now have a per-cpuset Boolean flag file called 'sched_load_balance'.

In the default case, this flag is set on, and the kernel does its
usual load balancing across all CPUs in that cpuset.  This means, under
the covers, that there exists some sched domain such that all CPUs in
that cpuset are in that same sched domain.  That sched domain might
contain additional CPUs from outside that cpuset as well.  Indeed,
in the default vanilla configuration, that sched domain contains all
CPUs in the system.

If we turn the sched_load_balance flag off for some cpuset, we are
telling the kernel it's ok not to load balance on the CPUs in that
cpuset (unless those CPUs are in some other cpuset that needed load
balancing anyway.)

This 'sched_load_balance' flag is, thus far, "the" cpuset hook
supporting realtime.  One can use it to configure a system so that
the kernel does not do normal load balancing on select CPUs, such
as those CPUs dedicated to realtime use.

It sounds like Peter is reminding us that we really have three choices
for a handling a given CPU's load balancing:
 1) normal kernel scheduler load balancing,
 2) RT load balancing, or
 3) no load balancing whatsoever.

If that's the case (if we really need choice 3) then a single Boolean
flag, such as sched_load_balance, is not sufficient to select from
the three choices, and it might make sense to add a second per-cpuset
Boolean flag, say "sched_rt_balance", default off, which if turned on,
enabled choice 2.

If that's not the case (we only need choices 1 and 2) then -logically-
we could overload the meaning of the current sched_load_balance,
to mean, if turned off, not only to stop doing normal balancing, but
to further mean that we should commence RT balancing.  However bits
aren't -that- precious here, and this sounds unnecessarily confusing.

So ... would a new per-cpuset Boolean flag such as sched_rt_balance be
appropriate and sufficient to mark those cpusets whose set of CPUs
required RT balancing?

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.940.382.4214