Re: [PATCH RESEND] sched: prefer an idle cpu vs an idle sibling for BALANCE_WAKE

[Josef Bacik <jbacik@fb.com>]

On 06/03/2015 12:53 PM, Mike Galbraith wrote:
> On Wed, 2015-06-03 at 11:57 -0400, Josef Bacik wrote:
>> On 06/03/2015 11:30 AM, Mike Galbraith wrote:
>>> On Wed, 2015-06-03 at 16:24 +0200, Peter Zijlstra wrote:
>>>> On Wed, 2015-06-03 at 10:12 -0400, Rik van Riel wrote:
>>>>
>>>>> There is a policy vs mechanism thing here. Ingo and Peter
>>>>> are worried about the overhead in the mechanism of finding
>>>>> an idle CPU.  Your measurements show that the policy of
>>>>> finding an idle CPU is the correct one.
>>>>
>>>> For his workload; I'm sure I can find a workload where it hurts.
>>>>
>>>> In fact, I'm fairly sure Mike knows one from the top of his head, seeing
>>>> how he's the one playing about trying to shrink that idle search :-)
>>>
>>> Like anything where scheduling latency doesn't heavily dominate.  Even
>>> if searching were free, bounces aren't, even for the very light.
>>>
>>
>> If scheduling latency doesn't hurt then making the search shouldn't
>> matter should it?  I get that migrations aren't free, but it seems like
>> they can't hurt that much.
>
> Nah, they don't hurt much :)
>
> commit e0a79f529d5ba2507486d498b25da40911d95cf6
> Author: Mike Galbraith <bitbucket@online.de>
> Date:   Mon Jan 28 12:19:25 2013 +0100
>
>      sched: Fix select_idle_sibling() bouncing cow syndrome
>
>      If the previous CPU is cache affine and idle, select it.
>
>      The current implementation simply traverses the sd_llc domain,
>      taking the first idle CPU encountered, which walks buddy pairs
>      hand in hand over the package, inflicting excruciating pain.
>
>      1 tbench pair (worst case) in a 10 core + SMT package:
>
>        pre   15.22 MB/sec 1 procs
>        post 252.01 MB/sec 1 procs
>
>
>>    This application is huge, it's our
>> webserver, we're doing like 400 requests per second on these things, and
>> hands down moving stuff to idle cpus is beating the pants off of staying
>> on the same cpu.  Is there a specific workload I could build a test for
>> that you think this approach would hurt?  Thanks,
>
> Search cost hurts fast movers, as does dragging even a small footprint
> all over the place, as you can see above.
>

Eesh ok, do you happen to remember how you ran tbench so I can add it to 
my tests here?  In addition to fixing this problem we're also interested 
in tracking performance of new kernels so we don't have to do this "what 
the hell went wrong in the last 6 releases" dance every year, so I'm 
throwing every performance thing we find useful in our test 
infrastructure.  Thanks,

Josef