[Qemu-devel] QEMU/KVM performance gets worser - high load - high interrupts - high context switches

[Gerhard Wiesinger <lists@wiesinger.com>]

Hello,

Yesterday I looked at my munin statistics on my KVM host and I swar that 
performance gets worser: load is getting higher, interrupts are getting 
higher and are high as well as context switches. VMs and applications 
didn't change that way.

You can find graphics at: http://www.wiesinger.com/tmp/kvm/
Last spike I guess was upgrade from FC22 to FC23 or a kernel update. And 
it was even lower on older versions

For me it looks like the high interrupt load and context switches are 
the root cause. Interrupts inside the VM are <100, so with 10 VMs I'm 
expecting 1000+baseload => <2000, see statistics below.

All VMs are virtio on disk/network except one (IDE/rtl8139).

# Host as well as all guests (except 2 VMs):
uname -a
Linux kvm 4.2.6-301.fc23.x86_64 #1 SMP Fri Nov 20 22:22:41 UTC 2015 
x86_64 x86_64 x86_64 GNU/Linux

qemu-system-x86-2.4.1-1.fc23.x86_64

Platform:

All VMs have the pc-i440fx-2.4 profile (I upgraded yesterday from 
pc-i440fx-2.3 without any change).

Any ideas, anyone having same issues?

Ciao,
Gerhard

kvm: no VM running
  r  b   swpd   free   buff  cache   si   so    bi    bo in   cs us sy 
id wa st
  0  0      0 3308516 102408 3798568    0    0     0    12 197  679  0  
0 99  0  0
  0  0      0 3308516 102416 3798564    0    0     0    42 197  914  0  
0 99  1  0
  0  0      0 3308516 102416 3798568    0    0     0     0 190  791  0  
0 100  0  0
  2  0      0 3308484 102416 3798568    0    0     0     0 129  440  0  
0 100  0  0

kvm: 2 VMs running
  procs -----------memory---------- ---swap-- -----io---- -system-- 
------cpu-----
  r  b   swpd   free   buff  cache   si   so    bi    bo in   cs us sy 
id wa st
  1  0      0 2641464 103052 3814700    0    0     0     0 2715 5648  3  
2 95  0  0
  0  0      0 2641340 103052 3814700    0    0     0     0 2601 5555  1  
2 97  0  0
  1  0      0 2641308 103052 3814700    0    0     0     5 2687 5708  3  
2 95  0  0
  0  0      0 2640620 103060 3814628    0    0     0    30 2779 5756  4  
3 93  1  0
  0  0      0 2640644 103060 3814636    0    0     0     0 2436 5364  1  
2 97  0  0
  1  0      0 2640520 103060 3814636    0    0     0   119 2734 5975  3  
2 95  0  0

kvm: all 10 VMs running
procs -----------memory---------- ---swap-- -----io---- -system-- 
------cpu-----
  r  b   swpd   free   buff  cache   si   so    bi    bo in   cs us sy 
id wa st
  1  0      0  60408  78892 3371984    0    0     0    85 9015 17357  4  
9 87  0  0
  2  0      0  60408  78892 3371968    0    0     0    47 9375 17797  9  
9 82  0  0
  0  0      0  60472  78892 3372092    0    0    40    60 8882 17343  4  
8 86  1  0
  1  0      0  60316  78892 3372080    0    0     0    59 8863 17517  4  
8 87  0  0
  0  0      0  59540  78900 3372092    0    0     0    55 9135 17796  8  
9 81  1  0
  0  0      0  59168  78900 3372112    0    0     0    51 8931 17484  4  
9 87  0  0

cat /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 Quad CPU           @ 2.66GHz
stepping        : 7