Re: Database regression due to scheduler changes ?

[Nick Piggin <nickpiggin@yahoo.com.au>]

Brian Twichell wrote:
> David Lang wrote:
> 
>>   If I am understanding the data you posted, it looks like you are 
>> useing sched_yield extensivly in your database. 
> 
> 
> Yes, I've seen problems in the past with workloads that use sched_yield 
> heavily.
> 
> But bear in mind, the ~2700 sched_yields shown in the schedstats 
> occurred over a 9 minute period. That means that sched_yield is being 
> called at a rate of around 5 per second -- this is not a heavy user of 
> sched_yield.
> 
> To put this into a broader perspective, this workload has around 270 
> tasks, and the context switch rate is around
> 45,000 per second.
> 

Hi,

Thanks for your detailed report (and schedstats analysis). Sorry
I didn't see it until now.

I think you are right that the NUMA domain is probably being too
constrictive of task balancing, and that is where the regression
is coming from.

For some workloads it is definitely important to have the NUMA
domain, because it helps spread load over memory controllers as
well as CPUs - so I guess eliminating that domain is not a good
long term solution.

I would look at changing parameters of SD_NODE_INIT in include/
asm-powerpc/topology.h so they are closer to SD_CPU_INIT parameters
(ie. more aggressive).

Reducing min_ and max_interval, busy_factor, cache_hot_time, will
all do this.

I would also take a look at removing SD_WAKE_IDLE from the flags.
This flag should make balancing more aggressive, but it can have
problems when applied to a NUMA domain due to too much task
movement.

I agree that sched_yield would be unlikely to be a problem at
those rates, and either way it doesn't explain the regression.

-- 
SUSE Labs, Novell Inc.

Send instant messages to your online friends http://au.messenger.yahoo.com