Re: + sched-use-tasklet-to-call-balancing.patch added to -mm tree

[Ingo Molnar <mingo@elte.hu>]

* Christoph Lameter <clameter@sgi.com> wrote:

> I have done some testing on NUMA with 8p / 4n and 256 p / 128n which 
> seems to indicate that this is doing very well. Having a global 
> tasklet avoids concurrent load balancing from multiple nodes which 
> smoothes out the load balancing load over all nodes in a NUMA system. 
> It fixes the issues that we saw with contention during concurrent load 
> balancing.

ok, that's what i suspected - what made the difference wasnt the fact 
that it was moved out of irqs-off section, but that it was running 
globally, instead of in parallel on every cpu. I have no conceptual 
problem with single-threading the more invasive load-balancing bits. 
(since it has to touch every runqueue anyway there's probably little 
parallelism possible) But it's a scary change nevertheless, it 
materially affects every SMP system's balancing characteristics.

Also, tasklets are a pretty bad choice for scalable stuff - it's 
basically a shared resource between all CPUs and the tasklet-running 
cacheline will bounce between all the CPUs. Also, a tasklet can be 
'reactivated', which means that a CPU will do more rebalancing albeit 
it's /another/ CPU that wanted that.

So could you try another implementation perhaps, which would match the 
'single thread of rebalancing' model better? For example, try a global 
spinlock that is trylocked upon rebalance. If the trylock succeeds then 
do the rebalance and unlock. If the trylock fails, just skip the 
rebalance. (this roughly matches the tasklet logic but has lower 
overhead and doesnt cause false, repeat rebalancing)

	Ingo