Re: [rfc][patch] sched: remove smpnice

[Peter Williams <pwil3058@bigpond.net.au>]

Peter Williams wrote:
> Siddha, Suresh B wrote:
> 
>> On Wed, Feb 15, 2006 at 09:40:32AM +1100, Peter Williams wrote:
>>
>>> Siddha, Suresh B wrote:
>>>
>>>> On a 4P(8-way with HT), if you run a -20 task(a simple infinite loop)
>>>> it hops from one processor to another processor... you can observe it
>>>> using top.
>>>
>>>
>>> How do you get top to display which CPU tasks are running on?
>>
>>
>>
>> In the interactive mode, you can select the "last used cpu" field to 
>> display.

Thanks this works and I'm now convinced that there's hopping occurring 
but I disagree on the cause (see below).

>> or you can use /proc/pid/stat
>>
>>
>>>> find_busiest_group() thinks there is an imbalance and ultimately the
>>>> idle cpu kicks active load balance on busy cpu, resulting in the 
>>>> hopping.
>>>
>>>
>>> I'm still having trouble getting my head around this.  A task 
>>> shouldn't be moved unless there's at least one other task on its 
>>> current CPU, it 
>>
>>
>>
>> Because of the highest priority task, weighted load of that cpu
>> will be > SCHED_LOAD_SCALE. Because of this, an idle cpu in 
>> find_busiest_group() thinks that there is an imbalance.. This is due to
>> the code near the comment "however we may be ablet to increase total 
>> CPU power used by ...". That piece of code assumes that a unit load
>> is represented by SCHED_LOAD_SCALE (which is no longer true with smpnice)
>> and finally results in "pwr_move > pwr_now".. This will make the idle cpu
>> try to pull that process from busiest cpu and the process will 
>> ultimately move
>> with the help of active load balance...
> 
> 
> But as I pointed out, with the code as it is in the recent -mm kernels, 
> the amount of biased load (i.e. NICE_TO_BIAS_PRIO(0)) that 
> find_busiest_group() sets *imbalance to in these circumstances is too 
> small for a task with nice less than zero to be moved i.e. move_tasks() 
> will skip it.  Or are you just referring to the vanilla kernels?

The upshot of this is that the code near "however we may" etc. never 
gets executed.  So what must be happening is that *imbalance is greater 
than SCHED_LOAD_SCALE so it goes out unchanged (as the fact that the 
problem occurs with hard spinners lets try_to_wake_up() off the hook). 
But, as I also said in another e-mail, there must be another task with 
equal or higher priority than the task (that's doing the hopping) on its 
CPU for it to be booted off.  I would surmise that the prime candidate 
here would be the migration thread for the CPU in question as it's a 
real time task and seems to run fairly frequently.

The obvious way to handle this is too NOT move tasks from a run queue 
where the migration thread is running unless the number of running tasks 
on that CPU is greater than 2.  I'll try to come up with a patch that 
does this without causing too much complexity.

Peter
-- 
Peter Williams                                   pwil3058@bigpond.net.au

"Learning, n. The kind of ignorance distinguishing the studious."
  -- Ambrose Bierce