Memory leaks in virtio drivers?

Memory leaks in virtio drivers?

[Freddie Cash]

Are there still known memory leaks in the virtio block and/or net
drivers?  If not, have we found one?

This is our host setup:
  - Tyan h2000M motherboard
  - 2x dual-core AMD Opteron 2220 CPUs
  - 16 GB RAM
  - quad-port NIC bonded together into 1 kvmbr0 interface
  - 3Ware 9550SXU-12ML RAID controller
  - 12x 400 GB SATA drives (4-disk RAID5, 4-disk RAID5, 3-disk RAID5,
spare disk)
  - LVM used to stitch the 3 arrays into one volume group

Running 64-bit Debian 5.0 (Lenny), using qemu-kvm0.12.4 and kernel
2.6.32 from lenny-backports.

Guests are Ubuntu Server 8.04 LTS, Debian Lenny, Windows XP, and Windows 2003.

All of the Linux VMs use virtio for both network and disk.
All the Windows VMs use virtio for network and IDE emulation for disk.

Within 2 weeks of booting, the host machine is using 2 GB of swap, and
disk I/O wait is through the roof.  Restarting all of the VMs will
free up RAM, but restarting the whole box is the only way to get
performance back up.

A guest configured to use 8 GB of RAM will have 9 GB virt and 7.5 GB
res shown in top.  In fact, every single VM shows virt above the limit
set for the VM.  Usually by close to 25%.

Going back through the mailing list archives and bug archives, there's
been mention of several memory leaks in the virtio drivers back to
KVM-72.  Last bug report I read shows them all being fixed in 0.12.4,
which we're running.

Perhaps there's something wrong with our setup?  Perhaps there's still
a leak to be found?  Perhaps this is fixed in 0.12.5 or newer?

Open to suggetions.

Sample KVM command-line:
/usr/bin/kvm -name riker -smp 1 -m 1024 -vnc :5 -daemonize -localtime
-usb -usbdevice tablet -net nic,macaddr=00:16:3e:00:00:5,model=virtio
-net tap,ifname=tap5 -pidfile /var/run/kvm/riker.pid -boot c -drive
index=0,media=disk,if=virtio,boot=on,file=/dev/mapper/vol0-riker
-drive index=1,media=disk,if=virtio,file=/dev/mapper/vol0-riker--var

We're in the process of migrating from our custom scripts (kvmctl) to
libvirt, if that makes any difference.

-- 
Freddie Cash
fjwcash@gmail.com

Re: Memory leaks in virtio drivers?

[Stefan Hajnoczi]

On Fri, Nov 26, 2010 at 7:19 PM, Freddie Cash <fjwcash@gmail.com> wrote:
> Within 2 weeks of booting, the host machine is using 2 GB of swap, and
> disk I/O wait is through the roof.  Restarting all of the VMs will
> free up RAM, but restarting the whole box is the only way to get
> performance back up.
>
> A guest configured to use 8 GB of RAM will have 9 GB virt and 7.5 GB
> res shown in top.  In fact, every single VM shows virt above the limit
> set for the VM.  Usually by close to 25%.

Not sure about specific known issues with those Debian package versions, but...

Virtual memory does not mean much.  For example, a 64-bit process can
map in 32 GB and never touch it.  The virt number will be >32 GB but
actually no RAM is being used.  Or it could be a memory mapped file,
which is backed by the disk and can pages can dropped if physical
memory runs low.  Looking at the virtual memory figure is not that
useful.

Also remember that qemu-kvm itself requires memory to perform the
device emulation and virtualization.  If you have an 8 GB VM, plan for
more than 8 GB to be used.  Clearly this memory overhead should be
kept low, is your 25% virtual memory overhead figure from a small VM
because 9 GB virtual / 8 GB VM is 12.5% not 25%?

What is the sum of all VMs' RAM?  I'm guessing you may have
overcommitted resources (e.g. 2 x 8 GB VM on a 16 GB machine).  If you
don't leave host Linux system some resources you will get bad VM
performance.

Stefan

Re: Memory leaks in virtio drivers?

[Freddie Cash]

On Fri, Nov 26, 2010 at 12:04 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Fri, Nov 26, 2010 at 7:19 PM, Freddie Cash <fjwcash@gmail.com> wrote:
>> Within 2 weeks of booting, the host machine is using 2 GB of swap, and
>> disk I/O wait is through the roof.  Restarting all of the VMs will
>> free up RAM, but restarting the whole box is the only way to get
>> performance back up.
>>
>> A guest configured to use 8 GB of RAM will have 9 GB virt and 7.5 GB
>> res shown in top.  In fact, every single VM shows virt above the limit
>> set for the VM.  Usually by close to 25%.
>
> Not sure about specific known issues with those Debian package versions, but...
>
> Virtual memory does not mean much.  For example, a 64-bit process can
> map in 32 GB and never touch it.  The virt number will be >32 GB but
> actually no RAM is being used.  Or it could be a memory mapped file,
> which is backed by the disk and can pages can dropped if physical
> memory runs low.  Looking at the virtual memory figure is not that
> useful.
>
> Also remember that qemu-kvm itself requires memory to perform the
> device emulation and virtualization.  If you have an 8 GB VM, plan for
> more than 8 GB to be used.  Clearly this memory overhead should be
> kept low, is your 25% virtual memory overhead figure from a small VM
> because 9 GB virtual / 8 GB VM is 12.5% not 25%?
>
> What is the sum of all VMs' RAM?  I'm guessing you may have
> overcommitted resources (e.g. 2 x 8 GB VM on a 16 GB machine).  If you
> don't leave host Linux system some resources you will get bad VM
> performance.

Nope, not overcommitted.  Sum of RAM for all VMs (in MB):
512 + 768 + 1024 + 512 + 512 + 1024 +  1024 + 768 + 8192 = 14226
Leaving a little under 2 GB for the host.

Doing further googling, could it be a caching issue in the host?  We
currently have no cache= settings for any of our virtual disks.  I
believe the default is still write-through? so the host is trying to
cache everything.

From top on the host:
Mem:  16539696k total, 16467616k used,    72080k free,  3153416k buffers
Swap:  3253120k total,  2072976k used,  1180144k free,   194832k cached

Over 2 GB of swap used, and 3 GB in buffers.

One of our VMs is a Zimbra install, so it's doing a lot of
reads/writes to disk (~2100 users) (the 8 GB VM).  Another is our main
web server, running Apache + Drupal + MySQL (1 GB VM), so also doing a
fair bit of read/write to disk.  The others are lightly-used web
servers and Windows VMs used for remote desktop.

I'm going to restart the VMs over the weekend using cache=none for the
virtio disks and see if that makes a difference.

Anyone know how to force libvirt to use cache='none' in the <driver>
block?  libvirt-bin 0.8.3 and virt-manager 0.8.4 ignore it if I edit
the domain.xml file directly, and there's nowhere to set it in the
virt-manager GUI.  (Only 1 of the VMs is managed via libvirt
currently.)

-- 
Freddie Cash
fjwcash@gmail.com

Re: Memory leaks in virtio drivers?

[Stefan Hajnoczi]

On Fri, Nov 26, 2010 at 8:16 PM, Freddie Cash <fjwcash@gmail.com> wrote:
> On Fri, Nov 26, 2010 at 12:04 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
>> On Fri, Nov 26, 2010 at 7:19 PM, Freddie Cash <fjwcash@gmail.com> wrote:
>>> Within 2 weeks of booting, the host machine is using 2 GB of swap, and
>>> disk I/O wait is through the roof.  Restarting all of the VMs will
>>> free up RAM, but restarting the whole box is the only way to get
>>> performance back up.
>>>
>>> A guest configured to use 8 GB of RAM will have 9 GB virt and 7.5 GB
>>> res shown in top.  In fact, every single VM shows virt above the limit
>>> set for the VM.  Usually by close to 25%.
>>
>> Not sure about specific known issues with those Debian package versions, but...
>>
>> Virtual memory does not mean much.  For example, a 64-bit process can
>> map in 32 GB and never touch it.  The virt number will be >32 GB but
>> actually no RAM is being used.  Or it could be a memory mapped file,
>> which is backed by the disk and can pages can dropped if physical
>> memory runs low.  Looking at the virtual memory figure is not that
>> useful.
>>
>> Also remember that qemu-kvm itself requires memory to perform the
>> device emulation and virtualization.  If you have an 8 GB VM, plan for
>> more than 8 GB to be used.  Clearly this memory overhead should be
>> kept low, is your 25% virtual memory overhead figure from a small VM
>> because 9 GB virtual / 8 GB VM is 12.5% not 25%?
>>
>> What is the sum of all VMs' RAM?  I'm guessing you may have
>> overcommitted resources (e.g. 2 x 8 GB VM on a 16 GB machine).  If you
>> don't leave host Linux system some resources you will get bad VM
>> performance.
>
> Nope, not overcommitted.  Sum of RAM for all VMs (in MB):
> 512 + 768 + 1024 + 512 + 512 + 1024 +  1024 + 768 + 8192 = 14226
> Leaving a little under 2 GB for the host.

How do those VM RAM numbers stack up with ps -eo rss,args | grep kvm?

If the rss reveals the qemu-kvm processes are >15 GB RAM then it might
be worth giving them more breathing room.

> Doing further googling, could it be a caching issue in the host?  We
> currently have no cache= settings for any of our virtual disks.  I
> believe the default is still write-through? so the host is trying to
> cache everything.

Yes, the default is writethrough.  cache=none would reduce buffered
file pages so it's worth a shot.

> Anyone know how to force libvirt to use cache='none' in the <driver>
> block?  libvirt-bin 0.8.3 and virt-manager 0.8.4 ignore it if I edit
> the domain.xml file directly, and there's nowhere to set it in the
> virt-manager GUI.  (Only 1 of the VMs is managed via libvirt
> currently.)

A hack you can do if your libvirt does not support the <driver
cache="none"/> attribute is to move /usr/bin/qemu-kvm out of the way
and replace it with a shell script that does
s/if=virtio/if=virtio,cache=none/ on its arguments before invoking the
real /usr/bin/qemu-kvm.  (Perhaps the cleaner way is editing the
domain XML for <emulator>/usr/bin/kvm_cache_none.sh</emulator> but I
haven't tested it.)

Stefan

Re: Memory leaks in virtio drivers?

[Nikola Ciprich]

> How do those VM RAM numbers stack up with ps -eo rss,args | grep kvm?
> 
> If the rss reveals the qemu-kvm processes are >15 GB RAM then it might
> be worth giving them more breathing room.
you might also consider using KSM, it usually saves a LOT of memory
cheers
n.

 

-- 
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava

tel.:   +420 596 603 142
fax:    +420 596 621 273
mobil:  +420 777 093 799

www.linuxbox.cz

mobil servis: +420 737 238 656
email servis: servis@linuxbox.cz
-------------------------------------

Re: Memory leaks in virtio drivers?

[Freddie Cash]

On Fri, Nov 26, 2010 at 12:54 PM, Stefan Hajnoczi <stefanha@gmail.com> wrote:
> On Fri, Nov 26, 2010 at 8:16 PM, Freddie Cash <fjwcash@gmail.com> wrote:
>> Nope, not overcommitted.  Sum of RAM for all VMs (in MB):
>> 512 + 768 + 1024 + 512 + 512 + 1024 +  1024 + 768 + 8192 = 14226
>> Leaving a little under 2 GB for the host.
>
> How do those VM RAM numbers stack up with ps -eo rss,args | grep kvm?
>
> If the rss reveals the qemu-kvm processes are >15 GB RAM then it might
> be worth giving them more breathing room.

That's a little under 12 GB (11,843,888 KB).

-- 
Freddie Cash
fjwcash@gmail.com

Re: Memory leaks in virtio drivers?

[Michael Tokarev]

26.11.2010 22:19, Freddie Cash wrote:
> Are there still known memory leaks in the virtio block and/or net
> drivers?  If not, have we found one?
> 
> This is our host setup:
>   - Tyan h2000M motherboard
>   - 2x dual-core AMD Opteron 2220 CPUs
>   - 16 GB RAM
>   - quad-port NIC bonded together into 1 kvmbr0 interface
>   - 3Ware 9550SXU-12ML RAID controller
>   - 12x 400 GB SATA drives (4-disk RAID5, 4-disk RAID5, 3-disk RAID5,
> spare disk)
>   - LVM used to stitch the 3 arrays into one volume group
> 
> Running 64-bit Debian 5.0 (Lenny), using qemu-kvm0.12.4 and kernel
> 2.6.32 from lenny-backports.

There were some memleaks in i/o paths fixed in 0.12 series of kvm.
In particular, 0.12.4 upstream version had a memory leak in virtio_blk,
but that bug isn't present in 0.12.4+dfsg-1 - first 0.12.4 version as
appeared in debian (the fix included upstream in 0.12.5).  I know no
other memleaks in later 0.12 series which aren't fixed in debian.

> Guests are Ubuntu Server 8.04 LTS, Debian Lenny, Windows XP, and Windows 2003.

For debian lenny guests using virtio, consider switching to
backports 2.6.32 kernel too - lenny's 2.6.26 has several
defects in virtio implementation.

> All of the Linux VMs use virtio for both network and disk.
> All the Windows VMs use virtio for network and IDE emulation for disk.
> 
> Within 2 weeks of booting, the host machine is using 2 GB of swap, and
> disk I/O wait is through the roof.  Restarting all of the VMs will
> free up RAM, but restarting the whole box is the only way to get
> performance back up.
> 
> A guest configured to use 8 GB of RAM will have 9 GB virt and 7.5 GB
> res shown in top.  In fact, every single VM shows virt above the limit
> set for the VM.  Usually by close to 25%.
> 
> Going back through the mailing list archives and bug archives, there's
> been mention of several memory leaks in the virtio drivers back to
> KVM-72.  Last bug report I read shows them all being fixed in 0.12.4,
> which we're running.

Yes, 0.12.4 had that bug, but it's fixed in 0.12.4+dfsg-1.
The current version in backports is based on 0.12.5.

> Perhaps there's something wrong with our setup?  Perhaps there's still
> a leak to be found?  Perhaps this is fixed in 0.12.5 or newer?

Well, how about reading the changelog first, before asking? They're
there for a reason, right?

Speaking of your setup, in addition to what Stephen said already,
I'd strongly suggest considering hugepages - for this amount of
memory hugepages should help quite alot, and this way you'll eliminate
lots of pagetable entries.

/mjt

Re: Memory leaks in virtio drivers?

[Freddie Cash]

On Fri, Nov 26, 2010 at 1:00 PM, Michael Tokarev <mjt@tls.msk.ru> wrote:
> 26.11.2010 22:19, Freddie Cash wrote:
>> Are there still known memory leaks in the virtio block and/or net
>> drivers?  If not, have we found one?
>>
>> This is our host setup:
>>   - Tyan h2000M motherboard
>>   - 2x dual-core AMD Opteron 2220 CPUs
>>   - 16 GB RAM
>>   - quad-port NIC bonded together into 1 kvmbr0 interface
>>   - 3Ware 9550SXU-12ML RAID controller
>>   - 12x 400 GB SATA drives (4-disk RAID5, 4-disk RAID5, 3-disk RAID5,
>> spare disk)
>>   - LVM used to stitch the 3 arrays into one volume group
>>
>> Running 64-bit Debian 5.0 (Lenny), using qemu-kvm0.12.4 and kernel
>> 2.6.32 from lenny-backports.
>
> There were some memleaks in i/o paths fixed in 0.12 series of kvm.
> In particular, 0.12.4 upstream version had a memory leak in virtio_blk,
> but that bug isn't present in 0.12.4+dfsg-1 - first 0.12.4 version as
> appeared in debian (the fix included upstream in 0.12.5).  I know no
> other memleaks in later 0.12 series which aren't fixed in debian.

Okay, that's what I've read via the bug reports and changelogs, but
wanted to be sure.

>> Guests are Ubuntu Server 8.04 LTS, Debian Lenny, Windows XP, and Windows 2003.
>
> For debian lenny guests using virtio, consider switching to
> backports 2.6.32 kernel too - lenny's 2.6.26 has several
> defects in virtio implementation.

Can do.  There's only one that isn't already running the 2.6.32-bpo kernel.

>> All of the Linux VMs use virtio for both network and disk.
>> All the Windows VMs use virtio for network and IDE emulation for disk.
>>
>> Within 2 weeks of booting, the host machine is using 2 GB of swap, and
>> disk I/O wait is through the roof.  Restarting all of the VMs will
>> free up RAM, but restarting the whole box is the only way to get
>> performance back up.
>>
>> A guest configured to use 8 GB of RAM will have 9 GB virt and 7.5 GB
>> res shown in top.  In fact, every single VM shows virt above the limit
>> set for the VM.  Usually by close to 25%.
>>
>> Going back through the mailing list archives and bug archives, there's
>> been mention of several memory leaks in the virtio drivers back to
>> KVM-72.  Last bug report I read shows them all being fixed in 0.12.4,
>> which we're running.
>
> Yes, 0.12.4 had that bug, but it's fixed in 0.12.4+dfsg-1.
> The current version in backports is based on 0.12.5.

That would be this version (which we have installed)?
Version: 0.12.4+dfsg-1~bpo50+2

Hrm, no, there's a newer one, just not in our mirror yet:
0.12.5+dfsg-3~bpo50+2

I'll see if we can upgrade to that one over the weekend as well.

> Well, how about reading the changelog first, before asking? They're
> there for a reason, right?

I read everything under /usr/share/doc/[qemu|kvm|libvirt], along with
the release notes on the qemu/kvm websites, and ended up with more
questions than answers, hence why I posted.  :)  I also read through a
bunch of old threads on the kvm mailing list, several bug reports for
qemu, qemu-kvm, libvirt, rhel, and so forth.

The repeating theme through them all was that mem leaks in virtio
cropped up every other release or so, including some major ones in the
0.12.x series (0.12.3 and 0.12.4 especially).  The symptoms we're
having now are the same ones we originally had with KVM-72 when we
first switched from SCSI-emulated disks to virtio.

> Speaking of your setup, in addition to what Stephen said already,
> I'd strongly suggest considering hugepages - for this amount of
> memory hugepages should help quite alot, and this way you'll eliminate
> lots of pagetable entries.

Hrm, I'll have to do some more googling on this.  There's nothing
about it in the man pages or --help output, but I've seen it
references on the mailing list.  Same for KSM (as suggested in another
message), although would that be helpful with such a variety of VM
guest OSes (Debian Etch, Debian Lenny, Ubuntu Server, Windows XP,
Windows 2003)?

-- 
Freddie Cash
fjwcash@gmail.com

Re: Memory leaks in virtio drivers?

[Michael Tokarev]

27.11.2010 00:38, Freddie Cash wrote:
> On Fri, Nov 26, 2010 at 1:00 PM, Michael Tokarev <mjt@tls.msk.ru> wrote:
[]
>> Well, how about reading the changelog first, before asking? They're
>> there for a reason, right?
> 
> I read everything under /usr/share/doc/[qemu|kvm|libvirt], along with
> the release notes on the qemu/kvm websites, and ended up with more
> questions than answers, hence why I posted.  :)  I also read through a
> bunch of old threads on the kvm mailing list, several bug reports for
> qemu, qemu-kvm, libvirt, rhel, and so forth.

No, I mean the changelog for the more recent debian packages.
But it was more of a joke actually, please don't treat it too
seriously.

> The repeating theme through them all was that mem leaks in virtio
> cropped up every other release or so, including some major ones in the
> 0.12.x series (0.12.3 and 0.12.4 especially).  The symptoms we're
> having now are the same ones we originally had with KVM-72 when we
> first switched from SCSI-emulated disks to virtio.

qemu-kvm has a long history of bugs, that's true.
That's why I kept 0.12 in Debian stable, instead
of switching to 0.13.  Lots of code, lots of new
code too, difficult to have it bug-free from the
beginning.

Besides, maybe you hit yet another, still unknown
bug of this theme - which is also a possibility
ofcourse.  No, kvm-0.12.5 (+ a few patches) is
used to run guests in production, with quite
some uptime too, and unlike with <=0.12.4, there
were nothing wrong so far.

>> Speaking of your setup, in addition to what Stephen said already,
>> I'd strongly suggest considering hugepages - for this amount of
>> memory hugepages should help quite alot, and this way you'll eliminate
>> lots of pagetable entries.
> 
> Hrm, I'll have to do some more googling on this.  There's nothing
> about it in the man pages or --help output, but I've seen it
> references on the mailing list.  Same for KSM (as suggested in another
> message), although would that be helpful with such a variety of VM
> guest OSes (Debian Etch, Debian Lenny, Ubuntu Server, Windows XP,
> Windows 2003)?

KSM may help, but it wont work with hugepages (since
the more the page size is, the less chances to find
another page with same contents).

Hugepages - see -mem-path parameter.  It's not widely
known feature.  The idea is that you allocate hugepages
at boot, mount hugetlbfs, and give path to it in -mem-path,
so that instead of malloc() kvm uses mmap() there.  Or
something of that theme.

Hugepages are unswappable and unsharable (KSM), but they're
much lighter in terms of pagetables and tlb misses.

/mjt

Re: Memory leaks in virtio drivers?

[Freddie Cash]

On Fri, Nov 26, 2010 at 2:16 PM, Michael Tokarev <mjt@tls.msk.ru> wrote:
>> Hrm, I'll have to do some more googling on this.  There's nothing
>> about it in the man pages or --help output, but I've seen it
>> references on the mailing list.  Same for KSM (as suggested in another
>> message), although would that be helpful with such a variety of VM
>> guest OSes (Debian Etch, Debian Lenny, Ubuntu Server, Windows XP,
>> Windows 2003)?
>
> KSM may help, but it wont work with hugepages (since
> the more the page size is, the less chances to find
> another page with same contents).
>
> Hugepages - see -mem-path parameter.  It's not widely
> known feature.  The idea is that you allocate hugepages
> at boot, mount hugetlbfs, and give path to it in -mem-path,
> so that instead of malloc() kvm uses mmap() there.  Or
> something of that theme.
>
> Hugepages are unswappable and unsharable (KSM), but they're
> much lighter in terms of pagetables and tlb misses.

After even more googling, and experimenting, I think we may just be
running a very poorly optimised setup, and running ourselves out of
memory because of it.  I think, and hopefully testing over the weekend
will dis/prove it, a combination of the following should resolve our
issues:
  - update Linux guest kernels to 2.6.32 or newer
  - bring guest VMs (including Windows) up-to-date and match
versions/patch-levels (maximise possibility of sharing pages)
  - enable KSM in the host
  - add "cache=none" to the disk config on the KVM command-line

Enabling KSM with the way things are currently freed up 1.2 GB of RAM
in the host.  Which was then gobbled up as buffers (now over 4 GB).
I'm guessing that's from the host caching all the writes from the
Zimbra and web servers.

Thanks,everyone, for the helpful nudges and suggestions.  If there's
anything else that may be useful, or things I should be reading up on,
let me know.  :)
-- 
Freddie Cash
fjwcash@gmail.com

Re: Memory leaks in virtio drivers?

[Michael Tokarev]

27.11.2010 02:09, Freddie Cash wrote:
[]
> Enabling KSM with the way things are currently freed up 1.2 GB of RAM
> in the host.  Which was then gobbled up as buffers (now over 4 GB).

This may go away after some time.  When you start windows there are
lots of zero pages, easy to share.  Later on win actually starts using
these pages, and it shares alot less.  So it might be false goodie,
watch it closely.  It'd be interesting to see how it really performs
on your case.

For cache=off,  I highly recommend using raw partitions (or LVM logical
volumes) instead of files for disk images - if you don't do that already.
And speaking of storage, if you use some sort of raid (except raid1,
but _especially_ for raid[456]), ensure proper alignment of everythg.

/mjt

Re: Memory leaks in virtio drivers?

[Freddie Cash]

Just an update on this.  We made the change over the weekend to enable
"cache=off" for all the VMs, including the libvirt managed ones (turns
out, libvirtd only reads the .xml files at startup); and enabeld KSM
on the host.

5 days later, we have only 700 MB of swap used, and 15.2 GB of VM
committed.  This appears to be the steady-state for the host, as it
hasn't changed (cache, free, and buffer change a bit throughout the
day).

Unfortunately, this has exposed just how horribly unoptimised our
storage array underneath it.  :(  It's a single 12-drive RAID6,
auto-carved into 2 TB chunks, and then stitched back together via LVM
into a single volume group.  With each VM getting it's own logical
volume.  We have plans over the Christmas break to  re-do this as a
RAID50 or possible a RAID10 + RAID50.

Thanks for all the tips and pointers.  I'm starting to get all this
figured out and understood.  There's been a lot of changes since
KVM-72.  :)

-- 
Freddie Cash
fjwcash@gmail.com

Re: Memory leaks in virtio drivers?

[Freddie Cash]

On Wed, Dec 1, 2010 at 10:16 AM, Freddie Cash <fjwcash@gmail.com> wrote:
> Just an update on this.  We made the change over the weekend to enable
> "cache=off" for all the VMs, including the libvirt managed ones (turns
> out, libvirtd only reads the .xml files at startup); and enabeld KSM
> on the host.
>
> 5 days later, we have only 700 MB of swap used, and 15.2 GB of VM
> committed.  This appears to be the steady-state for the host, as it
> hasn't changed (cache, free, and buffer change a bit throughout the
> day).
>
> Unfortunately, this has exposed just how horribly unoptimised our
> storage array underneath it.  :(  It's a single 12-drive RAID6,
> auto-carved into 2 TB chunks, and then stitched back together via LVM
> into a single volume group.  With each VM getting it's own logical
> volume.  We have plans over the Christmas break to  re-do this as a
> RAID50 or possible a RAID10 + RAID50.
>
> Thanks for all the tips and pointers.  I'm starting to get all this
> figured out and understood.  There's been a lot of changes since
> KVM-72.  :)

One final update for the archives.

Further research showed that our I/O setup was non-optimal for the
RAID controllers were are using (3Ware 9650SE).  Modifying the
following sysfs variables dropper our I/O wait down to near 0,
allowing the controller to keepup with the requests, and eliminating
our use of swap (even without using cache=none):

(for each block device)
block/sda/queue/scheduler       = deadline
block/sda/queue/nr_requests     = 512
block/sda/queue/max_sectors_kb  = 128
block/sda/queue/read_ahead_kb   = 16384

We're in the process of adding a separate 6-disk RAID10 array for our
most I/O-heavy filesystems, which should help even more.

So, it looks like our I/O setup was so horrible originally that the
controller couldn't keep up, and more RAM in the host was used for
buffering, until the host started swapping, and things would grind to
a halt.

Over a week later, and we're still sitting with 0 bytes swap used in
the host, with several 100 MB of free RAM (fluctuates), and I/O usage
rarely above 30%.

-- 
Freddie Cash
fjwcash@gmail.com