KVM and the OOM-Killer

[James Stevens <James.Stevens@jrcs.co.uk>]

This is *NOT* a KVM issue, but may be worth adding into the FAQ...


We have a KVM host with 48Gb of RAM and run about 20 KVM clients on it. 
After some time - different time depending on the kernel version - the 
VM host kernel will start OOM-Killing the VM clients, even when there is 
lots of free RAM (>10Gb) and free SWAP (>34Gb).

This seems to be caused by the kernel running out of LOWMEM (memory 
below 1Gb) - because of the large amount of RAM a lot of LOWMEM (~400Mb) 
is used by the memory map (32 bytes per 4Kb page), add in the kernel 
itself and that leaves "only" about 460Mb of LOWMEM for kernel alloc.

This may not have been a problem, except Linux may also put cache blocks 
and user processes in LOWMEM - it seems this can then lead to a LOWMEM 
exhaust situation which triggers OOM-Killing even when there is LOADS of 
SWAP and HIGHMEM free.

Sadly, killing userland processes is not a good way to try and free 
LOWMEM, so what happens is a killing spree where by every process on the 
VM host gets killed (inc all the VMs, sysklogd, klogd, sshd, udevd etc).

This is very bad in 2.6.32.6, quite bad in 2.6.32.9,  better (but still 
bad in) 2.6.31.12 - currently testing 2.6.33.3

See https://bugzilla.kernel.org/show_bug.cgi?id=15058


General advice seems to be, if you have more than 16Gb RAM then you 
should run the VM host 64bit.

We didn't see this issue on a server with 32Gb running the same set of VMs.



James