Re: balance storm

[Mike Galbraith <umgwanakikbuti@gmail.com>]

On Tue, 2014-05-27 at 15:56 +0800, Libo Chen wrote: 
> On 2014/5/26 22:19, Mike Galbraith wrote:
> > On Mon, 2014-05-26 at 20:16 +0800, Libo Chen wrote: 
> >> On 2014/5/26 13:11, Mike Galbraith wrote:
> > 
> >>> Your synthetic test is the absolute worst case scenario.  There has to
> >>> be work between wakeups for select_idle_sibling() to have any chance
> >>> whatsoever of turning in a win.  At 0 work, it becomes 100% overhead.
> >>
> >> not synthetic, it is a real problem in our product. under no load, waste
> >> much cpu time.
> > 
> > What happens in your product if you apply the commit I pointed out?
> 
> under no load, cpu usage is up to 60%, but the same apps cost 10% on
> susp sp1.  The apps use a lot of timer.

Something is rotten.  3.14-rt contains that commit, I ran your test with
256 threads on 64 core box, saw ~4%.

Putting master/nopreempt config on box and doing the same test, box is
chewing up truckloads of CPU, but not from migrations. 

perf top -g --sort=symbol

Samples: 7M of event 'cycles', Event count (approx.): 1316249172581                                                                                                                                                                         
-   82.56%  [k] _raw_spin_lock_irqsave                                                                                                                                                                                                     ▒
   - _raw_spin_lock_irqsave                                                                                                                                                                                                                ▒
      - 96.59% __nanosleep_nocancel                                                                                                                                                                                                        ◆
           100.00% __libc_start_main                                                                                                                                                                                                       ▒
        2.88% __poll                                                                                                                                                                                                                       ▒
           0                                                                                                                                                                                                                               ▒
+    1.56%  [k] native_write_msr_safe                                                                                                                                                                                                      ▒
+    1.21%  [k] update_cfs_shares                                                                                                                                                                                                          ▒
+    0.92%  [k] __schedule                                                                                                                                                                                                                 ▒
+    0.88%  [k] _raw_spin_lock                                                                                                                                                                                                             ▒
+    0.73%  [k] update_cfs_rq_blocked_load                                                                                                                                                                                                 ▒
+    0.62%  [k] idle_cpu                                                                                                                                                                                                                   ▒
+    0.47%  [.] usleep                                                                                                                                                                                                                     ▒
+    0.41%  [k] cpuidle_enter_state                                                                                                                                                                                                        ▒
+    0.37%  [k] set_task_cpu

Oh, 256 * usleep(100) is not a great idea.

	-Mike