Re: [User Question] Repeated severe performance problems on guest

[Martin Wawro <martin.wawro@gmail.com>]

On 04/16/2013 07:49 AM, Stefan Hajnoczi wrote:

Hi Stefan,

> Besides the kvm_stat, general performance data from the host is useful
> when dealing with high load averages.
>
> Do you have vmstat or sar data for periods of time when the machine was
> slow?
>
> Stefan

We do have a rather exhaustive log on the guest. As for the host, we did
not find
anything suspicious except  for the kvm_stat output. So we did not log
any more
than that.

Here is the output of "vmstat 5 5" on the guest:

procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
id wa
84  0  19596 104404     60 21932616    0    0   232   110    9    2  7 
2 90  1
80  0  19596  98100     60 21933920    0    0   106   119  854  912 79
21  0  0
89  0  19596  94216     60 21932764    0    0   106   223  864  886 79
21  0  0
87  0  19596  95848     60 21927612    0    0    82    47  856  906 79
21  0  0

Load average at that time: 75 (1:20 AM)

The guest seems to have a hard time scheduling tasks. The log output, which
is triggered by a simple cronjob that executes commands like "vmstat",
"atop"
or simply appending some information from /proc and /sys to the log
(in a sequential manner) is a bit scrambled (i.e. the expected order
of the output is not kept, most likely because the cronjobs get piled up).
This can also be seen on the rather large numbers in the first column
(there is
no workload scheduled for that time and virtually no one is using the
system,
the whole thing just seems to happen out of thin air).

The huge number of runnable tasks fits the high load average, but we
could not
see the reason why the tasks are piling up. There is/was no apparent I/O
issue on
disk or network and no messages from the kernel at that time. Also,
there was no
swap activity on either guest or host (host has swap disabled).


For comparison, here is the output from 22:55 (10:55 PM), a couple of hours
before the output from above:

procs -----------memory---------- ---swap-- -----io---- -system--
----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy
id wa
 5  0     36  97848     60 24259656    0    0   227   111    7    1  7 
2 91  1
 6  0     36 121532     60 24220340    0    0   106   327  974  539 31
22 46  0
 9  0     36 126108     60 24212756    0    0     6     0 1019  517 19 
7 73  0
 2  0     36 125264     60 24212780    0    0     0    26 1007  481 25 
6 69  0
 6  0     36 147564     60 24212808    0    0     4   120 1345 1161 25
10 65  0


If you need any more info, just let me know.

Best regards,

Martin