Re: [PATCH RFC 1/2] kvm: Handle undercommitted guest case in PLE handler

[Raghavendra K T <raghavendra.kt@linux.vnet.ibm.com>]

* Avi Kivity <avi@redhat.com> [2012-10-04 17:00:28]:

> On 10/04/2012 03:07 PM, Peter Zijlstra wrote:
> > On Thu, 2012-10-04 at 14:41 +0200, Avi Kivity wrote:
> >> 
> >> Again the numbers are ridiculously high for arch_local_irq_restore.
> >> Maybe there's a bad perf/kvm interaction when we're injecting an
> >> interrupt, I can't believe we're spending 84% of the time running the
> >> popf instruction. 
> > 
> > Smells like a software fallback that doesn't do NMI, hrtimer based
> > sampling typically hits popf where we re-enable interrupts.
> 
> Good nose, that's probably it.  Raghavendra, can you ensure that the PMU
> is properly exposed?  'dmesg' in the guest will tell.  If it isn't, -cpu
> host will expose it (and a good idea anyway to get best performance).
> 

Hi Avi, you are right. SandyBridge machine result was not proper.
I cleaned up the services, enabled PMU, re-ran all the test again.

Here is the summary:
We do get good benefit by increasing ple window. Though we don't
see good benefit for kernbench and sysbench, for ebizzy, we get huge
improvement for 1x scenario. (almost 2/3rd of ple disabled case).

Let me know if you think we can increase the default ple_window
itself to 16k.

I am experimenting with V2 version of undercommit improvement(this) patch
series, But I think if you wish  to go for increase of
default ple_window, then we would have to measure the benefit of patches
when ple_window = 16k.

I can respin the whole series including this default ple_window change.

I also have the perf kvm top result for both ebizzy and kernbench.
I think they are in expected lines now.

Improvements
================

16 core PLE machine with 16 vcpu guest

base = 3.6.0-rc5 + ple handler optimization patches
base_pleopt_16k = base + ple_window = 16k
base_pleopt_32k = base + ple_window = 32k
base_pleopt_nople = base + ple_gap = 0
kernbench, hackbench, sysbench (time in sec lower is better)
ebizzy (rec/sec higher is better)

% improvements w.r.t base (ple_window = 4k)
---------------+---------------+-----------------+-------------------+
               |base_pleopt_16k| base_pleopt_32k | base_pleopt_nople |
---------------+---------------+-----------------+-------------------+
kernbench_1x   |  0.42371      |  1.15164        |   0.09320         |
kernbench_2x   | -1.40981      | -17.48282       |  -570.77053       |
---------------+---------------+-----------------+-------------------+
sysbench_1x    | -0.92367      | 0.24241         | -0.27027          |
sysbench_2x    | -2.22706      |-0.30896         | -1.27573          |
sysbench_3x    | -0.75509      | 0.09444         | -2.97756          |
---------------+---------------+-----------------+-------------------+
ebizzy_1x      | 54.99976      | 67.29460        |  74.14076         |
ebizzy_2x      | -8.83386      |-27.38403        | -96.22066         |
---------------+---------------+-----------------+-------------------+

perf kvm top observation for kernbench and ebizzy (nople, 4k, 32k window) 
========================================================================
pleopt   ple_gap=0
--------------------
ebizzy : 18131 records/s
63.78%  [guest.kernel]  [g] _raw_spin_lock_irqsave
    5.65%  [guest.kernel]  [g] smp_call_function_many
    3.12%  [guest.kernel]  [g] clear_page
    3.02%  [guest.kernel]  [g] down_read_trylock
    1.85%  [guest.kernel]  [g] async_page_fault
    1.81%  [guest.kernel]  [g] up_read
    1.76%  [guest.kernel]  [g] native_apic_mem_write
    1.70%  [guest.kernel]  [g] find_vma

kernbench :Elapsed Time 29.4933 (27.6007)
   5.72%  [guest.kernel]  [g] async_page_fault
    3.48%  [guest.kernel]  [g] pvclock_clocksource_read
    2.68%  [guest.kernel]  [g] copy_user_generic_unrolled
    2.58%  [guest.kernel]  [g] clear_page
    2.09%  [guest.kernel]  [g] page_cache_get_speculative
    2.00%  [guest.kernel]  [g] do_raw_spin_lock
    1.78%  [guest.kernel]  [g] unmap_single_vma
    1.74%  [guest.kernel]  [g] kmem_cache_alloc

pleopt ple_window = 4k
---------------------------
ebizzy: 10176 records/s
   69.17%  [guest.kernel]  [g] _raw_spin_lock_irqsave
    3.34%  [guest.kernel]  [g] clear_page
    2.16%  [guest.kernel]  [g] down_read_trylock
    1.94%  [guest.kernel]  [g] async_page_fault
    1.89%  [guest.kernel]  [g] native_apic_mem_write
    1.63%  [guest.kernel]  [g] smp_call_function_many
    1.58%  [guest.kernel]  [g] SetPageLRU
    1.37%  [guest.kernel]  [g] up_read
    1.01%  [guest.kernel]  [g] find_vma


kernbench: 29.9533
nts: 240K cycles
    6.04%  [guest.kernel]  [g] async_page_fault
    4.17%  [guest.kernel]  [g] pvclock_clocksource_read
    3.28%  [guest.kernel]  [g] clear_page
    2.57%  [guest.kernel]  [g] copy_user_generic_unrolled
    2.30%  [guest.kernel]  [g] do_raw_spin_lock
    2.13%  [guest.kernel]  [g] _raw_spin_lock_irqsave
    1.93%  [guest.kernel]  [g] page_cache_get_speculative
    1.92%  [guest.kernel]  [g] unmap_single_vma
    1.77%  [guest.kernel]  [g] kmem_cache_alloc
    1.61%  [guest.kernel]  [g] __d_lookup_rcu
    1.19%  [guest.kernel]  [g] find_vma
    1.19%  [guest.kernel]  [g] __list_del_entry


pleopt: ple_window=16k
-------------------------
ebizzy: 16990
 62.35%  [guest.kernel]  [g] _raw_spin_lock_irqsave
    5.22%  [guest.kernel]  [g] smp_call_function_many
    3.57%  [guest.kernel]  [g] down_read_trylock
    3.20%  [guest.kernel]  [g] clear_page
    2.16%  [guest.kernel]  [g] up_read
    1.89%  [guest.kernel]  [g] find_vma
    1.86%  [guest.kernel]  [g] async_page_fault
    1.81%  [guest.kernel]  [g] native_apic_mem_write

kernbench: 28.5
 6.24%  [guest.kernel]  [g] async_page_fault
    4.16%  [guest.kernel]  [g] pvclock_clocksource_read
    3.33%  [guest.kernel]  [g] clear_page
    2.50%  [guest.kernel]  [g] copy_user_generic_unrolled
    2.08%  [guest.kernel]  [g] do_raw_spin_lock
    1.98%  [guest.kernel]  [g] unmap_single_vma
    1.89%  [guest.kernel]  [g] kmem_cache_alloc
    1.82%  [guest.kernel]  [g] page_cache_get_speculative
    1.46%  [guest.kernel]  [g] __d_lookup_rcu
    1.42%  [guest.kernel]  [g] _raw_spin_lock_irqsave
    1.15%  [guest.kernel]  [g] __list_del_entry
    1.10%  [guest.kernel]  [g] find_vma



Detailed result for the run
=============================
patched = base_pleopt_16k 
+-----------+-----------+-----------+------------+-----------+
                              kernbench 
+-----------+-----------+-----------+------------+-----------+
   base        stddev       patched    stdev        %improve     
+-----------+-----------+-----------+------------+-----------+
1x    30.0440     1.1896    29.9167     1.6755	   0.42371
2x    62.0083     3.4884    62.8825     2.5509	  -1.40981
+-----------+-----------+-----------+------------+-----------+
+-----------+-----------+-----------+------------+-----------+
                              sysbench 
+-----------+-----------+-----------+------------+-----------+
1x     7.1779     0.0577     7.2442     0.0479	  -0.92367
2x    15.5362     0.3370    15.8822     0.3591	  -2.22706
3x    23.8249     0.1513    24.0048     0.1844	  -0.75509
+-----------+-----------+-----------+------------+-----------+
+-----------+-----------+-----------+------------+-----------+
                              ebizzy 
+-----------+-----------+-----------+------------+-----------+
1x 10358.0000   442.6598   16054.8750  252.5088    54.99976
2x  2705.5000   130.0286   2466.5000   120.0024	  -8.83386
+-----------+-----------+-----------+------------+-----------+

patched = base_pleopt_32k
+-----------+-----------+-----------+------------+-----------+
                              kernbench 
+-----------+-----------+-----------+------------+-----------+
   base        stddev       patched    stdev        %improve     
+-----------+-----------+-----------+------------+-----------+
1x    30.0440     1.1896    29.6980     0.6760	   1.15164
2x    62.0083     3.4884    72.8491     4.4616	 -17.48282
+-----------+-----------+-----------+------------+-----------+
+-----------+-----------+-----------+------------+-----------+
                              sysbench 
+-----------+-----------+-----------+------------+-----------+
1x     7.1779     0.0577     7.1605     0.0447	   0.24241
2x    15.5362     0.3370    15.5842     0.1731	  -0.30896
3x    23.8249     0.1513    23.8024     0.2342	   0.09444
+-----------+-----------+-----------+------------+-----------+
+-----------+-----------+-----------+------------+-----------+
                              ebizzy 
+-----------+-----------+-----------+------------+-----------+
1x  10358.0000   442.6598   17328.3750   281.4569   67.29460
2x  2705.5000   130.0286    1964.6250   143.0793   -27.38403
+-----------+-----------+-----------+------------+-----------+

patched = base_pleopt_nople
+-----------+-----------+-----------+------------+-----------+
                              kernbench 
+-----------+-----------+-----------+------------+-----------+
   base        stddev       patched    stdev        %improve     
+-----------+-----------+-----------+------------+-----------+
1x    30.0440     1.1896    30.0160     0.7523	   0.09320
2x    62.0083     3.4884   415.9334   189.9901	  -570.77053
+-----------+-----------+-----------+------------+-----------+
+-----------+-----------+-----------+------------+-----------+
                              sysbench 
+-----------+-----------+-----------+------------+-----------+
1x     7.1779     0.0577     7.1973     0.0354	  -0.27027
2x    15.5362     0.3370    15.7344     0.2315	  -1.27573
3x    23.8249     0.1513    24.5343     0.3437	  -2.97756
+-----------+-----------+-----------+------------+-----------+
+-----------+-----------+-----------+------------+-----------+
                              ebizzy 
+-----------+-----------+-----------+------------+-----------+
1x 10358.0000   442.6598 18037.5000   315.2074	   74.14076
2x  2705.5000   130.0286   102.2500   104.3521	  -96.22066
+-----------+-----------+-----------+------------+-----------+