Re: scheduler scalability - cgroups, cpusets and load-balancing

Gregory Haskins
2008-01-29T16:48:44Z

>>> On Tue, Jan 29, 2008 at 11:28 AM, in message
<20080129102836.be614579.pj@sgi.com>, Paul Jackson <pj@sgi.com> wrote:
> Gregory wrote:
>> I am a bit confused as to why you disable load-balancing in the
>> RT cpuset? It shouldn't be strictly necessary in order for the
>> RT scheduler to do its job (unless I am misunderstanding what you
>> are trying to accomplish?). Do you do this because you *have*
>> to in order to make real-time deadlines, or because its just a
>> further optimization?
>
> My primary motivation for cpusets originally, and for the
> sched_load_balance flag now, was not realtime, but "soft partitioning"
> of big NUMA systems, especially for batch schedulers. They sometimes
> have large cpusets which are only being used to hold smaller, per-job,
> cpusets. It is a waste of time (CPU cycles in the kernel sched code)
> to load balance those large cpusets. Load balancing doesn't scale
> easily to high CPU counts, and it's nice to avoid doing that where
> not needed.

Understood, and that makes tons of sense.

>
> See the following lkml message for a fuller explanation:
>
> http://lkml.org/lkml/2008/1/29/85
>
> As a secondary motivation, I thought that disabling load balancing on
> the RT cpuset was the right thing to do for RT needs, but I make no
> claim to knowing much about RT.

Well, I make no claim to understand the large batch systems you work on either ;) Everything you said made a ton of sense other than the RT/load-balance thing, but I think we are on the same page now.

>
> I just now realized that you added a 'root_domain' in a patch in
> late Nov and early Dec. I was on the road then, moving from
> California to Texas, and not paying much attention to Linux.

np (though I was wondering why you had no comment before ;)

>
> A couple of questions on that patch, both involving a comment it adds
> to kernel/sched.c:
>
> /*
> * We add the notion of a root-domain which will be used to define per-domain
> * variables. Each exclusive cpuset essentially defines an island domain by
> * fully partitioning the member cpus from any other cpuset. Whenever a new
> * exclusive cpuset is created, we also create and attach a new root-domain
> * object.
> */
>
> 1) What are 'per-domain' variables?

s/per-domain/per-root-domain

>
> 2) The mention of 'exclusive cpuset' is no longer correct.
>
> With the patch 'remove sched domain hooks from cpusets' cpusets
> no longer defines sched domains using the cpu_exclusive flag.
>
> With the subsequent sched_load_balance patch (see
> http://lkml.org/lkml/2007/10/6/19) cpusets uses a new per-cpuset
> flag 'sched_load_balance' to define sched domains.

Doh! Thanks for the heads up.

>
> The following revised comment might be more accurate:
>
> /*
> * We add the notion of a root-domain which will be used to define per-domain
> * variables. Each non-overlapping sched domain defines an island domain by
> * fully partitioning the member cpus from any other cpuset. Whenever a new
> * such a sched domain is created, we also create and attach a new
> root-domain
> * object. These non-overlapping sched domains are determined by the cpuset
> * configuration, via a call to partition_sched_domains().
> */
>
> It sounds like you (Gregory, others) want your RT CPUs to be in a sched
> domain, unlike the current way things are, where my cpuset code
> carefully avoids setting up a sched domain for those CPUs. However I
> still have need, in the batch scheduler case explained above, to have
> some CPUs not in any sched domain.
>
> If you require these RT sched domains to be setup differently somehow,
> in some way that is visible to partition_sched_domains, then that
> apparently means we need a per-cpuset flag to mark those RT cpusets.

I think we only need a plain-vanilla partition, so no flags should be necessary.

-Greg

>
> If you just want an ordinary sched domain setup (just so long as it
> contains only the intended RT CPUs, not others) then I guess we don't
> technically need any more per-cpuset flags, but I'm worried, because
> the API we're presenting to users for this has just gone from subtle to
> bizarre. I suspect I'll want to add a flag anyway, if by doing so, I
> can make the kernel-user API, via cpusets, easier to understand.