Disk activity issue...

Disk activity issue...

[Erich Weiler]

I've got this issue that I've been banging my head against a wall for a 
while over and I think another pair of eyes may help, if anyone have a 
moment.  We have this new-ish KVM VM server (with the latest CentOS 5.5 
updates, kmod-kvm-83-164.el5_5.25) that houses 3 VMs.  It works mostly 
as expected except it has a very high load all the time, like 40-60, 
when the VMs are running.  I suspect it has to do with memory 
management, because when all 3 VMs are online, they should consume 5GB 
RAM on the VM server and they only consume like 2GB, so I think the rest 
of the RAM is swapping or something, because the disks are spinning at 
100% all the time (even when the VMs are doing nothing).  Although, the 
VM server does not report any swapping happening. When I shut down the 
VMs one by one, the load drops and so does the disk activity.  I don't 
think I set this server up with anything out of the ordinary...  I've 
tried rebooting, but the same thing happens immediately upon reboot. 
Google searches, for me at least, yielded nothing useful.

Anyone heard of this kind of thing happening?

-erich

Re: Disk activity issue...

[Stefan Hajnoczi]

On Thu, Dec 30, 2010 at 11:25 PM, Erich Weiler <bitscrubber@gmail.com> wrote:
> I've got this issue that I've been banging my head against a wall for a
> while over and I think another pair of eyes may help, if anyone have a
> moment.  We have this new-ish KVM VM server (with the latest CentOS 5.5
> updates, kmod-kvm-83-164.el5_5.25) that houses 3 VMs.  It works mostly as
> expected except it has a very high load all the time, like 40-60, when the
> VMs are running.  I suspect it has to do with memory management, because
> when all 3 VMs are online, they should consume 5GB RAM on the VM server and
> they only consume like 2GB, so I think the rest of the RAM is swapping or
> something, because the disks are spinning at 100% all the time (even when
> the VMs are doing nothing).  Although, the VM server does not report any
> swapping happening. When I shut down the VMs one by one, the load drops and
> so does the disk activity.  I don't think I set this server up with anything
> out of the ordinary...  I've tried rebooting, but the same thing happens
> immediately upon reboot. Google searches, for me at least, yielded nothing
> useful.

"very high load all the time, like 40-60"
Is this number the host CPU utilization or load average?

Which guest OS (and versions) are you running?

Can you paste the qemu-kvm command-line for the VMs?

Can you send a few lines of "vmstat 5" output on the host while
running the 3 VMs?

Stefan

Re: Disk activity issue...

[Erich Weiler]

Hi Stefan,

Thanks for replying!  I was able to figure it out - it was not the fault of KVM.  One of the guests was running ganglia gmetad which was updating 30,000+ rrd files every 15 seconds (thus generating load via disk I/O), I didn't spot that until shutting down the VMs one by one until I found the offending one.  It was just a needle in a haystack.  ;)

Thanks again!

On Jan 4, 2011, at 2:57 AM, Stefan Hajnoczi <stefanha@gmail.com> wrote:

> On Thu, Dec 30, 2010 at 11:25 PM, Erich Weiler <bitscrubber@gmail.com> wrote:
>> I've got this issue that I've been banging my head against a wall for a
>> while over and I think another pair of eyes may help, if anyone have a
>> moment.  We have this new-ish KVM VM server (with the latest CentOS 5.5
>> updates, kmod-kvm-83-164.el5_5.25) that houses 3 VMs.  It works mostly as
>> expected except it has a very high load all the time, like 40-60, when the
>> VMs are running.  I suspect it has to do with memory management, because
>> when all 3 VMs are online, they should consume 5GB RAM on the VM server and
>> they only consume like 2GB, so I think the rest of the RAM is swapping or
>> something, because the disks are spinning at 100% all the time (even when
>> the VMs are doing nothing).  Although, the VM server does not report any
>> swapping happening. When I shut down the VMs one by one, the load drops and
>> so does the disk activity.  I don't think I set this server up with anything
>> out of the ordinary...  I've tried rebooting, but the same thing happens
>> immediately upon reboot. Google searches, for me at least, yielded nothing
>> useful.
> 
> "very high load all the time, like 40-60"
> Is this number the host CPU utilization or load average?
> 
> Which guest OS (and versions) are you running?
> 
> Can you paste the qemu-kvm command-line for the VMs?
> 
> Can you send a few lines of "vmstat 5" output on the host while
> running the 3 VMs?
> 
> Stefan

Re: Disk activity issue...

[Javier Guerra Giraldez]

On Tue, Jan 4, 2011 at 11:19 AM, Erich Weiler <bitscrubber@gmail.com> wrote:
> Thanks for replying!  I was able to figure it out - it was not the fault of KVM.  One of the guests was running ganglia gmetad which was updating 30,000+ rrd files every 15 seconds (thus generating load via disk I/O), I didn't spot that until shutting down the VMs one by one until I found the offending one.  It was just a needle in a haystack.  ;)

i use iotop to check which VM is hitting the disk

-- 
Javier

Re: Disk activity issue...

[Erich Weiler]

Yeah, some other folks have suggested that too, but it looks like you need kernel version 2.6.18-199 or higher to run iotop, and CentOS 5.5 only has 2.6.18-192 or something...  Guess I need to wait for the kernel to be updated there if I want to use iotop, which sounds like a very cool utility.

-erich

On Jan 5, 2011, at 1:35 PM, Javier Guerra Giraldez <javier@guerrag.com> wrote:

> On Tue, Jan 4, 2011 at 11:19 AM, Erich Weiler <bitscrubber@gmail.com> wrote:
>> Thanks for replying!  I was able to figure it out - it was not the fault of KVM.  One of the guests was running ganglia gmetad which was updating 30,000+ rrd files every 15 seconds (thus generating load via disk I/O), I didn't spot that until shutting down the VMs one by one until I found the offending one.  It was just a needle in a haystack.  ;)
> 
> i use iotop to check which VM is hitting the disk
> 
> -- 
> Javier