Re: [patch] sched: auto-tune migration costs [was: Re: Industry db benchmark result on recent 2.6 kernels]

[Ingo Molnar <mingo@elte.hu>]

* Chen, Kenneth W <kenneth.w.chen@intel.com> wrote:

> > tested on x86, the calibration results look ok there.
> 
> Calibration result on ia64 (1.5 GHz, 9 MB), somewhat smaller in this 
> version compare to earlier estimate of 10.4ms.  The optimal setting 
> found by a db workload is around 16 ms.

with idle time in the system i'd first concentrate on getting rid of the 
idle time, and then re-measuring the sweet spot. 9.3 msec is certainly a 
correct ballpark figure.

There will also be workload related artifacts: you may speculatively 
delay migration to another CPU, in the hope of the currently executing 
task scheduling away soon. (I have played with that in the past - the 
scheduler has some idea about how scheduling-happy a task is, based on 
the interactivity estimator.)

The cost matrix on the other hand measures the pure cache-related 
migration cost, on a quiet system. There can be an up to factor 2 
increase in the 'effective migration cost', depending on the average 
length of the scheduling atom of the currently executing task. Further 
increases may happen if the system does not scale and interacting 
migrations slow down each other. So with the 9.3msec estimate, the true 
migration factor might be between 9.3 and 18.6 msecs. The bad news would 
be if the estimator gave a higher value than your sweet spot.

once we have a good estimate of the migration cost between domains 
(ignoring permanent penalties such as NUMA), there's a whole lot of 
things we can do with that, to apply it in a broader sense.

> ---------------------
> | migration cost matrix (max_cache_size: 9437184, cpu: -1 MHz):
> ---------------------
>           [00]    [01]    [02]    [03]
> [00]:     -     9.3(0)  9.3(0)  9.3(0)
> [01]:   9.3(0)    -     9.3(0)  9.3(0)
> [02]:   9.3(0)  9.3(0)    -     9.3(0)
> [03]:   9.3(0)  9.3(0)  9.3(0)    -
> --------------------------------
> | cacheflush times [1]: 9.3 (9329800)
> | calibration delay: 16 seconds
> --------------------------------

ok, the delay of 16 secs is alot better. Could you send me the full 
detection log, how stable is the curve?

	Ingo