Re: KVM performance vs. Xen

[Andrew Theurer <habanero@linux.vnet.ibm.com>]

Here are the SMT off results.  This workload is designed to not 
over-saturate the CPU, so you have to pick a number of server sets to 
ensure that.  With SMT on, 4 sets was enough for KVM, but 5 was too much 
(start seeing response time errors).  For SMT off, I tried to size the 
load as high as we can go without running into these errors.  For KVM, 
thats 3 (18 guests) and for Xen, that's 4 (24 guests).  The throughout 
has a fairly linear relationship to the number of server sets used, but 
has a bit of wiggle room (mostly affected by response times getting 
longer and longer, but not exceeding the requirement set forth).  
Anyway, the relative throughput for these are "1.0" for KVM and "1.34" 
for Xen.  The CPU is 78.71% for KVM the CPU is 87.83%. 

If we normalize to CPU utilization, Xen is doing 20% more throughput.

Avi Kivity wrote:
> Anthony Liguori wrote:
>>
>> Previously, the block API only exposed non-vector interfaces and 
>> bounced vectored operations to a linear buffer.  That's been 
>> eliminated now though so we need to update the linux-aio patch to 
>> implement a vectored backend interface.
>>
>> However, it is an apples to apples comparison in terms of copying 
>> since the same is true with the thread pool.  My take away was that 
>> the thread pool overhead isn't the major source of issues.
>
> If the overhead is dominated by copying, then you won't see the 
> difference.  Once the copying is eliminated, the comparison may yield 
> different results.  We should certainly see a difference in context 
> switches.
I would like to test this the proper way.  What do I need to do to 
ensure these copies are eliminated?  I am on a 2.6.27 kernel, am I 
missing anything there?  Anthony, would you be willing to provide a 
patch to support the changes in the block API?
>
> One cause of context switches won't be eliminated - the non-saturating 
> workload causes us to switch to the idle thread, which incurs a 
> heavyweight exit.  This doesn't matter since we're idle anyway, but 
> when we switch back, we incur a heavyweight entry.
I have not looked at the schedstat or ftrace yet, but will soon.  Maybe 
it will tell us a little more about the context switches.

Here's a sample of the kvm_stat:

 efer_relo      exits  fpu_reloa  halt_exit  halt_wake  host_stat  hypercall  insn_emul  insn_emul     invlpg   io_exits  irq_exits  irq_injec  irq_windo  kvm_reque  largepage  mmio_exit  mmu_cache  mmu_flood  mmu_pde_z  mmu_pte_u  mmu_pte_w  mmu_recyc  mmu_shado  mmu_unsyn  mmu_unsyn  nmi_injec  nmi_windo   pf_fixed   pf_guest  remote_tl  request_n  signal_ex  tlb_flush
         0     233866      53994      20353      16209     119812          0      48879          0          0      75666      44917      34772       3984          0        187          0         10          0          0          0          0          0          0          0          0          0          0        202          0          0          0          0      17698
         0     244556      67321      15570      12364     116226          0      49865          0          0      69357      56131      32860       4449          0      -1895          0         19          0          0          0          0         21         21          0          0          0          0       1117          0          0          0          0      21586
         0     230788      71382      10619       7920     109151          0      44354          0          0      62561      60074      28322       4841          0        103          0         13          0          0          0          0          0          0          0          0          0          0        122          0          0          0          0      22702
         0     275259      82605      14326      11148     127293          0      53738          0          0      73438      70707      34724       5373          0        859          0         15          0          0          0          0         21         21          0          0          0          0        874          0          0          0          0      26723
         0     250576      58760      20368      16476     128296          0      50936          0          0      80439      51219      36329       4621          0      -1170          0          8          0          0          0          0         22         22          0          0          0          0       1333          0          0          0          0      18508
         0     244746      59650      19480      15657     122721          0      49882          0          0      76011      50453      35352       4523          0        201          0         11          0          0          0          0         21         21          0          0          0          0        212          0          0          0          0      19163
         0     251724      71715      14049      10920     117255          0      49924          0          0      70173      58040      32328       5058          0         94          0          7          0          0          0          0          0          0          0          0          0          0        105          0          0          0          0      25405
         0     247873      75212      12397       9465     117299          0      47402          0          0      68435      62901      30999       5289          0         36          0          9          0          0          0          0          0          0          0          0          0          0         47          0          0          0          0      24400
         0     259105      79515      14060      10713     121489          0      50106          0          0      71847      62392      33165       4802          0        358          0         17          0          0          0          0          0          0          0          0          0          0        375          0          0          0          0      26420
         0     255283      74818      13847      10642     120632          0      48832          0          0      70851      65453      32520       5032          0        752          0          6          0          0          0          0          0          0          0          0          0          0        759          0          0          0          0      23764
         0     268411      78048      15231      11707     123642          0      52845          0          0      74031      64919      34404       4765          0        639          0         15          0          0          0          0          0          0          0          0          0          0        653          0          0          0          0      25992
         0     247064      73794      12554       9522     115026          0      47878          0          0      66357      64359      30727       4884          0         97          0          8          0          0          0          0          0          0          0          0          0          0        107          0          0          0          0      23545
         0     259641      79179      11953       9247     117090          0      49836          0          0      68561      67053      31171       5435          0      -2759          0         11          0          0          0          0         21         21          0          0          0          0        245          0          0          0          0      26858
         0     258109      77455      13997      10732     121578          0      50559          0          0      71833      63841      33404       4980          0        484          0         14          0          0          0          0         21         21          0          0          0          0        495          0          0          0          0      24509
         0     250245      74357      13611      10459     117791          0      49733          0          0      68471      65089      31943       4797          0        581          0         13          0          0          0          0          0          0          0          0          0          0        596          0          0          0          0      22517
         0     262114      77257      13614      10499     121082          0      50683          0          0      71242      67844      33234       5031          0       1125          0          8          0          0          0          0          0          0          0          0          0          0       1133          0          0          0          0      24370
         0     254914      75937      12784       9809     116020          0      50562          0          0      67452      62673      31249       4903          0        786          0         19          0          0          0          0          0          0          0          0          0          0        806          0          0          0          0      25931
         0     249421      75642      12704       9805     116039          0      48426          0          0      66972      62276      31068       4999          0      -1817          0          6          0          0          0          0         21         21          0          0          0          0        187          0          0          0          0      25169
         0     274205      79561      13992      10844     126452          0      53165          0          0      74522      68844      34131       5529          0        123          0         20          0          0          0          0         42         42          0          0          0          0        152          0          0          0          0      26633
         0     267310      77262      15092      11705     125139          0      52891          0          0      74651      64647      34938       5018          0        195          0         18          0          0          0          0          0          0          0          0          0          0        213          0          0          0          0      25161
 

-Andrew