big bw_pipe drop due to sched-domain patch?

big bw_pipe drop due to sched-domain patch?

[David Mosberger]

Today I noticed that going from 2.6.6 to 2.6.7-rc1 caused a big drop
in LMbench2 bw_pipe throughput on a dual-CPU machine (2.6.7-rc1 shows
10 times lower bandwidth than 2.6.6).  It appears that the drop is due
to the two tasks being distributed across the two CPUs, rather than
being kept on the same CPU.  If I force the tasks to be pinned on a
single CPU, e.g., like so:

	$ taskset 1 ./bw_pipe

then performance is "only" about 10% worse than with 2.6.6.

Is this kind of drop expected?

	--david

Re: big bw_pipe drop due to sched-domain patch?

[Nick Piggin]

David Mosberger wrote:
> Today I noticed that going from 2.6.6 to 2.6.7-rc1 caused a big drop
> in LMbench2 bw_pipe throughput on a dual-CPU machine (2.6.7-rc1 shows
> 10 times lower bandwidth than 2.6.6).  It appears that the drop is due
> to the two tasks being distributed across the two CPUs, rather than
> being kept on the same CPU.  If I force the tasks to be pinned on a
> single CPU, e.g., like so:
> 
> 	$ taskset 1 ./bw_pipe
> 
> then performance is "only" about 10% worse than with 2.6.6.
> 
> Is this kind of drop expected?
> 

(please CC Ingo for scheduler related stuff too)


Previously (ie. pre sched-domains), pipe based wakeups would
unconditionally migrate the "wakee" onto the same CPU as the waker.
Great for benchmarks, but possibly not really essential for real
workloads, especially when tasks start getting pulled across NUMA
nodes (I'd be happy to be shown otherwise though).

sched-domains introduces instead a "soft bias" that attempts to
get wakers and wakees together. The concept is applied to _all_
types of wakeups, not just pipe based.

I think this is a slightly better scheme, but we could increase
the aggressiveness of the bias if it is proven to be useful.

I don't know why your taskset performance is lower. taskset should
completely take the scheduler out of the equation AFAIKS...