Re: [User Question] Repeated severe performance problems on guest

[Martin Wawro <martin.wawro@gmail.com>]

Hi Stefan,

This morning we had to reboot the guest system again. pidstat logging
was not
enabled on the host (planned to do that later, hate myself for that).
Again it
looks like the schedulerinside the guest just does not want to schedule
anymore
(or takes coffee breaksinbetween), because tasks are piling up in the queue.


Info from guest (load at that time 28.88):

procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
id wa
43  0      0 129684     56 25702740    0    0   399    58    9  167 16
11 72  1
52  0      0 115632     56 25706964    0    0    85  1118 1091 1225 74
26  0  0
59  0      0 150380     56 25692648    0    0   122   681 1135 1283 73
27  0  0
47  0      0 135424     56 25699504    0    0    79   415 1076 1064 74
26  0  0
49  0      0 132420     56 25705604    0    0   130   551 1113 1083 74
26  0  0


Again, there is no apparent reason why the load should be that high. It
just spikes
up until we finally put the guest out of its misery (5 minutes before
the load went
up to 28.88 it was at 2.68 (normal for that time of day). We did not
wait until we
ended up at >70, because the system was actively used at that time. Shutdown
occured when load was at 35). There was no unusual workload and during
normal
operation the load on the system at that time of day is (reproducible)
between 2 and 4.


Info from host (same time, load on host 6.32 w/ 7 cores assigned to the
guest):

kvm_stat:

efer_reload                    0         0
exits                 2549659797    106703
fpu_reload              18554960       630
halt_exits              33754341       230
halt_wakeup             17080387       188
host_state_reload       91193250      2016
hypercalls                     0         0
insn_emulation          99324537      2569
insn_emulation_fail            0         0
invlpg                         0         0
io_exits                51338365      1097
irq_exits               82789640      6115
irq_injections          53016628      1494
irq_window               1240060        70
largepages                 10937       701
mmio_exits               4443644        72
mmu_cache_miss         409801870     21168
mmu_flooded                    0         0
mmu_pde_zapped                 0         0
mmu_pte_updated                0         0
mmu_pte_write             108123         0
mmu_recycled                   0         0
mmu_shadow_zapped      411841775     18928
mmu_unsync                     0         0
nmi_injections                 0         0
nmi_window                     0         0
pf_fixed              2248004983     95456
pf_guest                       0         0
remote_tlb_flush        35279973      4976
request_irq                    0         0
signal_exits                   0         0
tlb_flush                      0         0


vmstat 5 5:

procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
id wa st
 7  0      0 4442088 160968 2382804    0    0   251   115    3    1  9 
6 85  0  0     
 7  0      0 4459008 160968 2382804    0    0   497  1021 11254 3502  6
84 10  0  0    
 7  0      0 4431464 160968 2382804    0    0   372  1054 10286 2753  6
83 11  0  0    
 7  0      0 4437076 160972 2382804    0    0   552  1710 12890 4518  7
82 10  0  0    
 8  0      0 4438508 160972 2382804    0    0   174  1736 10398 3092  5
84 10  0  0   


Best regards,

Martin