RAM utilization issues

RAM utilization issues

[Jonathan Petersson]

Hi,

Throughout the last few months we've been slowly migrating off our old
VMware installation to KVM, most of it has been successful and people
has been happy with the end-result.

The new infrastructure obviously has a bit more of power which is
heavily utilized by the guests however compared to VMware the RAM
usage is way above what we've expected.

In our VMware environment we operated at 24GB with ballooning and we
were able to maintain usage at around 90% without any swapping.
However with the new environment we've 24x3 GB och RAM and we're
seeing a 80% utilization across the board with the same amount of
guests.

Please notice that we've both ballooning (on both the host and guest)
and KSM enabled, hugepages hasn't made any major difference so we've
chosen to disable this. If you've any input as to why we're seeing
this behavior or if it's expected we'd appreciate it. In addition to
this we're utilizing virtio_blk/net/pci/ring and vhost_net.

Some data:
root@kvm01-01:~# grep -i balloon /boot/config-$(uname -r)
CONFIG_VMWARE_BALLOON=m
CONFIG_XEN_BALLOON=y
CONFIG_VIRTIO_BALLOON=y

Here's our KSM history from last week on the servers:
http://oi43.tinypic.com/2s9d56v.jpg
http://oi41.tinypic.com/ru8lm9.jpg
http://oi43.tinypic.com/2s9d56v.jpg

--
Jonathan Petersson - Operations Engineer

op5 AB

Norgegatan 2
SE-164 32 Kista, Sweden

Direct  +46 8-58 83 01 21
Mob +46 733709021
E-mail: jpetersson@op5.com
www.op5.com

Re: RAM utilization issues

[Avi Kivity]

On 01/12/2012 11:27 AM, Jonathan Petersson wrote:
> Hi,
>
> Throughout the last few months we've been slowly migrating off our old
> VMware installation to KVM, most of it has been successful and people
> has been happy with the end-result.
>
> The new infrastructure obviously has a bit more of power which is
> heavily utilized by the guests however compared to VMware the RAM
> usage is way above what we've expected.
>
> In our VMware environment we operated at 24GB with ballooning and we
> were able to maintain usage at around 90% without any swapping.
> However with the new environment we've 24x3 GB och RAM and we're
> seeing a 80% utilization across the board with the same amount of
> guests.
>
> Please notice that we've both ballooning (on both the host and guest)
> and KSM enabled, hugepages hasn't made any major difference so we've
> chosen to disable this. If you've any input as to why we're seeing
> this behavior or if it's expected we'd appreciate it. In addition to
> this we're utilizing virtio_blk/net/pci/ring and vhost_net.
>
> Some data:
> root@kvm01-01:~# grep -i balloon /boot/config-$(uname -r)
> CONFIG_VMWARE_BALLOON=m
> CONFIG_XEN_BALLOON=y
> CONFIG_VIRTIO_BALLOON=y
>

What's the aggregate size of the guests (sum of all guest memory
sizes)?  How many guests?  Are you in fact using the balloon?  How?

-- 
error compiling committee.c: too many arguments to function

Re: RAM utilization issues

[Jonathan Petersson]

On Thu, Jan 12, 2012 at 11:36 AM, Avi Kivity <avi@redhat.com> wrote:
> On 01/12/2012 11:27 AM, Jonathan Petersson wrote:
>> Hi,
>>
>> Throughout the last few months we've been slowly migrating off our old
>> VMware installation to KVM, most of it has been successful and people
>> has been happy with the end-result.
>>
>> The new infrastructure obviously has a bit more of power which is
>> heavily utilized by the guests however compared to VMware the RAM
>> usage is way above what we've expected.
>>
>> In our VMware environment we operated at 24GB with ballooning and we
>> were able to maintain usage at around 90% without any swapping.
>> However with the new environment we've 24x3 GB och RAM and we're
>> seeing a 80% utilization across the board with the same amount of
>> guests.
>>
>> Please notice that we've both ballooning (on both the host and guest)
>> and KSM enabled, hugepages hasn't made any major difference so we've
>> chosen to disable this. If you've any input as to why we're seeing
>> this behavior or if it's expected we'd appreciate it. In addition to
>> this we're utilizing virtio_blk/net/pci/ring and vhost_net.
>>
>> Some data:
>> root@kvm01-01:~# grep -i balloon /boot/config-$(uname -r)
>> CONFIG_VMWARE_BALLOON=m
>> CONFIG_XEN_BALLOON=y
>> CONFIG_VIRTIO_BALLOON=y
>>
>
> What's the aggregate size of the guests (sum of all guest memory
> sizes)?  How many guests?  Are you in fact using the balloon?  How?
>
> --
> error compiling committee.c: too many arguments to function
>

This is a sample from one of the servers:

Aggregate size is 28672 MB wheras 20133 is utilized on the host.
Looking at each induvidual VM the actual utilization is rarely above
20% i.e. ~ 6GB.
On this particular host there's a total of 23 VMs.
Each VM is started with -balloon virtio, the host and the guests has
the virtio_balloon drivers loaded.


-- 
Jonathan Petersson - Operations Engineer

op5 AB

Norgegatan 2
SE-164 32 Kista, Sweden

Direct  +46 8-58 83 01 21
Mob +46 733709021
E-mail: jpetersson@op5.com
www.op5.com

Re: RAM utilization issues

[Avi Kivity]

On 01/12/2012 12:47 PM, Jonathan Petersson wrote:
> > What's the aggregate size of the guests (sum of all guest memory
> > sizes)?  How many guests?  Are you in fact using the balloon?  How?
> >
> > --
> > error compiling committee.c: too many arguments to function
> >
>
> This is a sample from one of the servers:
>
> Aggregate size is 28672 MB wheras 20133 is utilized on the host.
> Looking at each induvidual VM the actual utilization is rarely above
> 20% i.e. ~ 6GB.
> On this particular host there's a total of 23 VMs.
> Each VM is started with -balloon virtio, the host and the guests has
> the virtio_balloon drivers loaded.

Loading the balloon drivers does nothing.  The management agent must
actively inflate or deflate he balloon.

-- 
error compiling committee.c: too many arguments to function

Re: RAM utilization issues

[Jonathan Petersson]

It seams that our provider hasn't built a management engine for
ballooning, however I found this project
(https://github.com/aglitke/mom) which is compatible with libvirt
however as the mangement-engine we use doesn't utilize libvirt it's
not compatible, would you happen to know of a different project that
utilizes qemu-socket.

Thanks

On Thu, Jan 12, 2012 at 11:51 AM, Avi Kivity <avi@redhat.com> wrote:
> On 01/12/2012 12:47 PM, Jonathan Petersson wrote:
>> > What's the aggregate size of the guests (sum of all guest memory
>> > sizes)?  How many guests?  Are you in fact using the balloon?  How?
>> >
>> > --
>> > error compiling committee.c: too many arguments to function
>> >
>>
>> This is a sample from one of the servers:
>>
>> Aggregate size is 28672 MB wheras 20133 is utilized on the host.
>> Looking at each induvidual VM the actual utilization is rarely above
>> 20% i.e. ~ 6GB.
>> On this particular host there's a total of 23 VMs.
>> Each VM is started with -balloon virtio, the host and the guests has
>> the virtio_balloon drivers loaded.
>
> Loading the balloon drivers does nothing.  The management agent must
> actively inflate or deflate he balloon.
>
> --
> error compiling committee.c: too many arguments to function
>



-- 
Jonathan Petersson - Operations Engineer

op5 AB

Norgegatan 2
SE-164 32 Kista, Sweden

Direct  +46 8-58 83 01 21
Mob +46 733709021
E-mail: jpetersson@op5.com
www.op5.com

Re: RAM utilization issues

[Avi Kivity]

On 01/12/2012 02:56 PM, Jonathan Petersson wrote:
> It seams that our provider hasn't built a management engine for
> ballooning, however I found this project
> (https://github.com/aglitke/mom) which is compatible with libvirt
> however as the mangement-engine we use doesn't utilize libvirt it's
> not compatible, would you happen to know of a different project that
> utilizes qemu-socket.
>
>


No, sorry.  In general it's hard to have multiple management tools
talking to a single guest because of the need for synchronization.

-- 
error compiling committee.c: too many arguments to function