Re: KVM and the OOM-Killer

[Avi Kivity <avi@redhat.com>]

On 05/13/2010 03:20 PM, James Stevens wrote:
> This is *NOT* a KVM issue, but may be worth adding into the FAQ...
>
>
> We have a KVM host with 48Gb of RAM and run about 20 KVM clients on 
> it. After some time - different time depending on the kernel version - 
> the VM host kernel will start OOM-Killing the VM clients, even when 
> there is lots of free RAM (>10Gb) and free SWAP (>34Gb).
>
> This seems to be caused by the kernel running out of LOWMEM (memory 
> below 1Gb) - because of the large amount of RAM a lot of LOWMEM 
> (~400Mb) is used by the memory map (32 bytes per 4Kb page), add in the 
> kernel itself and that leaves "only" about 460Mb of LOWMEM for kernel 
> alloc.
>
> This may not have been a problem, except Linux may also put cache 
> blocks and user processes in LOWMEM - it seems this can then lead to a 
> LOWMEM exhaust situation which triggers OOM-Killing even when there is 
> LOADS of SWAP and HIGHMEM free.
>
> Sadly, killing userland processes is not a good way to try and free 
> LOWMEM, so what happens is a killing spree where by every process on 
> the VM host gets killed (inc all the VMs, sysklogd, klogd, sshd, udevd 
> etc).
>
> This is very bad in 2.6.32.6, quite bad in 2.6.32.9,  better (but 
> still bad in) 2.6.31.12 - currently testing 2.6.33.3
>
> See https://bugzilla.kernel.org/show_bug.cgi?id=15058
>
>
> General advice seems to be, if you have more than 16Gb RAM then you 
> should run the VM host 64bit.
>
> We didn't see this issue on a server with 32Gb running the same set of 
> VMs.
>

I'd go with 64-bit at 2GB and above.  It's both faster and safer.

The lowmem load is about 0.5% of guest memory, so  48GB means 240MB 
lowmem allocated.  Thin ice.

Since you can run a 64-bit kernel with your existing userspace, at least 
you have a simple upgrade path.

-- 
error compiling committee.c: too many arguments to function