[User question] Huge buffer size on KVM host

[User question] Huge buffer size on KVM host

[Martin Wawro]

Hi all,

We have noticed a strange thing in one of our setups which uses KVM for
virtualization.
The size of the buffer cache turns out to be rather large, here is what
'top' on the host
has to say about it:

Mem:  12274084k total, 12202860k used,    71224k free,  2991728k buffers
Swap:  8191992k total,    20232k used,  8171760k free,    44392k cached

As the host machine itself does not run any data intensive services, we
assume that
this is due to the guest VM which receives/sends about 200 GB of data
per day of which
about 35G have to be stored on disk (disk array connected to the host
and used inside
the guest via virtio). We did not run into problems with this setup, it
basically runs stable
for more than a year now (the buffer size never went past 30% of the
total memory).
However, we are not sure if the huge size of the buffer cache is
something one would
expect here. If this is the case, I would really love to know the reason
behind it and if
it has indeed something to do with KVM/virtio. Any enlightenment here
would make
my day.

Our setup:
--------------

Host:
  IBM BladeCenter HS22, 12 GB RAM, Xeon QC CPU
  CentOS 5.5 (64-bit) running 2.6.18-194.11.1.el5
  kvm-83-164.el5_5.25 (qemu 0.9.1, modules 2.6.18.194.17.1)
  libvirt 0.6.3

Guest:
  Ubuntu 10.04 (64-bit) running 2.6.32-27
  virtio for storage and network


Best regards,

-- 
--------------------------------------------------------------------
Martin Wawro           |                         Digital Medics GmbH 
Managing Director      |  Otto-Hahn-Str. 15, 44227 Dortmund, Germany
Tel. +49-231-9742-6622 |                      Fax: +49-231-9742-6623
Key: 0xB0A225BD        |        Registered at AG Dortmund, HRB 19360
--------------------------------------------------------------------

Re: [User question] Huge buffer size on KVM host

[Avi Kivity]

On 08/15/2012 11:06 AM, Martin Wawro wrote:
> Hi all,
> 
> We have noticed a strange thing in one of our setups which uses KVM for
> virtualization.
> The size of the buffer cache turns out to be rather large, here is what
> 'top' on the host
> has to say about it:
> 
> Mem:  12274084k total, 12202860k used,    71224k free,  2991728k buffers
> Swap:  8191992k total,    20232k used,  8171760k free,    44392k cached
> 
> As the host machine itself does not run any data intensive services, we
> assume that
> this is due to the guest VM which receives/sends about 200 GB of data
> per day of which
> about 35G have to be stored on disk (disk array connected to the host
> and used inside
> the guest via virtio). We did not run into problems with this setup, it
> basically runs stable
> for more than a year now (the buffer size never went past 30% of the
> total memory).
> However, we are not sure if the huge size of the buffer cache is
> something one would
> expect here. If this is the case, I would really love to know the reason
> behind it and if
> it has indeed something to do with KVM/virtio. Any enlightenment here
> would make
> my day.
> 
> Our setup:
> --------------
> 
> Host:
>   IBM BladeCenter HS22, 12 GB RAM, Xeon QC CPU
>   CentOS 5.5 (64-bit) running 2.6.18-194.11.1.el5
>   kvm-83-164.el5_5.25 (qemu 0.9.1, modules 2.6.18.194.17.1)
>   libvirt 0.6.3
> 

How is you storage set up?  Files (which format?) or logical volumes?
what's the cache= setting?


-- 
error compiling committee.c: too many arguments to function

Re: [User question] Huge buffer size on KVM host

[Martin Wawro]

On Aug 15, 2012, at 1:30 PM, Avi Kivity wrote:

>> We have noticed a strange thing in one of our setups which uses KVM for
>> virtualization.
>> The size of the buffer cache turns out to be rather large
>> [,,,]
>> 
> 
> How is you storage set up?  Files (which format?) or logical volumes?
> what's the cache= setting?

We are using logical volumes and the cache is set to 'none'.



With best regards,
-- 

--------------------------------------------------------------------
Martin Wawro           |                         Digital Medics GmbH 
Managing Director      |  Otto-Hahn-Str. 15, 44227 Dortmund, Germany
Tel. +49-231-9742-6622 |                      Fax: +49-231-9742-6623
Key: 0xB0A225BD        |        Registered at AG Dortmund, HRB 19360
--------------------------------------------------------------------

Re: [User question] Huge buffer size on KVM host

[Avi Kivity]

On 08/15/2012 03:05 PM, Martin Wawro wrote:
> 
> On Aug 15, 2012, at 1:30 PM, Avi Kivity wrote:
> 
>>> We have noticed a strange thing in one of our setups which uses KVM for
>>> virtualization.
>>> The size of the buffer cache turns out to be rather large
>>> [,,,]
>>> 
>> 
>> How is you storage set up?  Files (which format?) or logical volumes?
>> what's the cache= setting?
> 
> We are using logical volumes and the cache is set to 'none'.

Strange, that should work without any buffering.

What the contents of

  /sys/block/sda/queue/hw_sector_size

and

  /sys/block/sda/queue/logical_block_size

?

Replace sda by the disk underlying your physical volume.


-- 
error compiling committee.c: too many arguments to function

Re: [User question] Huge buffer size on KVM host

[Martin Wawro]

On Aug 15, 2012, at 2:57 PM, Avi Kivity wrote:

>> 
>> We are using logical volumes and the cache is set to 'none'.
> 
> Strange, that should work without any buffering.
> 
> What the contents of
> 
>  /sys/block/sda/queue/hw_sector_size
> 
> and
> 
>  /sys/block/sda/queue/logical_block_size
> 
> ?
> 

Hi Avi,

It seems that the kernel on that particular machine is too old, those entries are
not featured. We checked on a comparable setup with a newer kernel and both entries
were set to 512.

We also did have a third more thorough look on the caching. It turns out that the
virt-manager does not seem to honor the caching adjusted in the GUI correctly.
We disabled caching on all virtual devices for this particular VM and checking
with "ps -fxal" revealed, that only one of those devices (and a rather small one too)
had this set. We corrected this in the XML file directly and the buffer size
currently resides at around 1.8 GB after rebooting the VM (the only virtio device
not having the cache=none option set is now the (non-mounted) cdrom).


Best regards,
-- 

--------------------------------------------------------------------
Martin Wawro           |                         Digital Medics GmbH 
Managing Director      |  Otto-Hahn-Str. 15, 44227 Dortmund, Germany
Tel. +49-231-9742-6622 |                      Fax: +49-231-9742-6623
Key: 0xB0A225BD        |        Registered at AG Dortmund, HRB 19360
--------------------------------------------------------------------

Re: [User question] Huge buffer size on KVM host

[Avi Kivity]

On 08/16/2012 05:54 PM, Martin Wawro wrote:
> 
> On Aug 15, 2012, at 2:57 PM, Avi Kivity wrote:
> 
>>> 
>>> We are using logical volumes and the cache is set to 'none'.
>> 
>> Strange, that should work without any buffering.
>> 
>> What the contents of
>> 
>>  /sys/block/sda/queue/hw_sector_size
>> 
>> and
>> 
>>  /sys/block/sda/queue/logical_block_size
>> 
>> ?
>> 
> 
> Hi Avi,
> 
> It seems that the kernel on that particular machine is too old, those entries are
> not featured. We checked on a comparable setup with a newer kernel and both entries
> were set to 512.
> 
> We also did have a third more thorough look on the caching. It turns out that the
> virt-manager does not seem to honor the caching adjusted in the GUI correctly.
> We disabled caching on all virtual devices for this particular VM and checking
> with "ps -fxal" revealed, that only one of those devices (and a rather small one too)
> had this set. We corrected this in the XML file directly and the buffer size
> currently resides at around 1.8 GB after rebooting the VM (the only virtio device
> not having the cache=none option set is now the (non-mounted) cdrom).
> 

cc += libvirt-list

Is there a reason that cdroms don't get cache=none?


-- 
error compiling committee.c: too many arguments to function

Re: [User question] Huge buffer size on KVM host

[Martin Wawro]

On 08/16/2012 04:57 PM, Avi Kivity wrote:

Hi Avi,

No, there was no reason and we disabled it there too. Interestingly, the
buffer
size did not go down significantly, even when manually flushing the pages
using /proc/sys/vm/drop_caches (3), the buffer size did not go down.
Finally,
after rebooting the host, buffer size was back to normal again and when all
caches were disabled, we were also able to manually drop the buffered pages.

However, after 2 weeks or so, we encountered significant instabilities
in the system.
Inside the guest OS, the load went up like crazy (way past 30) and the 
guest
was virtually unusable. Checking the host: buffer size, load, memory,
I/O was all in
a well defined range.

When rebooting the guest OS, the problem manifested again after
about 20-30mins (reproducible a couple of times). The only thing that
worked to
stop this (I am afraid only for a couple of weeks or so), was rebooting
the host system.
On the guest and the host there was nothing suspicious in the logs or
dmesg output.


Best regards,

Martin

> Hi Avi,
>
> It seems that the kernel on that particular machine is too old, those entries are
> not featured. We checked on a comparable setup with a newer kernel and both entries
> were set to 512.
>
> We also did have a third more thorough look on the caching. It turns out that the
> virt-manager does not seem to honor the caching adjusted in the GUI correctly.
> We disabled caching on all virtual devices for this particular VM and checking
> with "ps -fxal" revealed, that only one of those devices (and a rather small one too)
> had this set. We corrected this in the XML file directly and the buffer size
> currently resides at around 1.8 GB after rebooting the VM (the only virtio device
> not having the cache=none option set is now the (non-mounted) cdrom).
>
> cc += libvirt-list
>
> Is there a reason that cdroms don't get cache=none?
>
>


-- 
--------------------------------------------------------------------
Martin Wawro           |                         Digital Medics GmbH 
Managing Director      |  Otto-Hahn-Str. 15, 44227 Dortmund, Germany
Tel. +49-231-9742-6622 |                      Fax: +49-231-9742-6623
Key: 0xB0A225BD        |        Registered at AG Dortmund, HRB 19360
--------------------------------------------------------------------