Re: [PATCH] cpuset and sched domains: sched_load_balance flag

[Nick Piggin <nickpiggin@yahoo.com.au>]

On Tuesday 02 October 2007 04:15, Paul Jackson wrote:
> Nick wrote:
> > which you could equally achieve by adding
> > a second set of sched domains (and the global domains could keep
> > globally balancing).
>
> Hmmm ... this could be the key to this discussion.
>
> Nick - can two sched domains overlap?  And if they do, what does that
> mean on any user or application behaviour.

Yes, sched domains can be completely arbitrary, and of course in the
current kernel, parent domains always overlap their children.

A sched domain usually means that the scheduler can move tasks
around among that group of CPUs, given the correct flags (but if
there are no flags, then it would be a superfluous domain and should
get trimmed away I think).

BTW. as far as the sched.c changes in your patch go, I much prefer
the partition_sched_domains API: http://lkml.org/lkml/2006/10/19/85

The caller should manage everything itself, rather than
partition_sched_domains doing half of the memory allocation.


> From the cpuset side - this patch handles overlap by joining the 'cpus'
> into one sched domain.  If two cpusets with overlapping 'cpus' are both
> marked 'sched_load_balance', then this patch forms a single, combined
> sched domain.
>
> As best as I can tell, you and I are actually in agreement in the
> case that there is no overlap.  If the several cpusets which have
> 'sched_load_balance' enabled have mutually disjoint 'cpus' (no
> overlap), then my patch forms exactly one sched domain for each such
> cpuset, having the same 'cpus'.

OK, I don't think your patch actually does the wrong thing
technically (although admittedly your rebuild_sched_domains
isn't something I really applied my poor brain to).

> The issue is the overlapping cases - are overlapping sched domains
> allowed, and if so, how do they affect user space?

For hard partitions, you don't want them of course. And I think
we should come up with a cpusets solution for that first.

Afterwards, overlapping sched domains are allowed and could be
used to make balancing more efficient (rather than any real
affect on userspace). At the moment, the domain builder probably
wouldn't cope very well, though.