Re: OOM problems

[John Weekes <lists.xen@nuclearfallout.net>]

> They are throttled, but the single control I'm aware of
> is /proc/sys/vm/dirty_ratio (or dirty_bytes, nowadays). Which is only
> per process, not a global limit. Could well be that's part of the
> problem -- outwitting mm with just too many writers on too many cores?
>
> We had a bit of trouble when switching dom0 to 2.6.32, buffered writes
> made it much easier than with e.g. 2.6.27 to drive everybody else into
> costly reclaims.
>
> The Oom shown here reports about ~650M in dirty pages. The fact alone
> that this counts as on oom condition doesn't sound quite right in
> itself. That qemu might just have dared to ask at the wrong point in
> time.
>
> Just to get an idea -- how many guests did this box carry?

It carries about two dozen guests, with a mix of mostly HVMs (all 
stubdom-based, some with PV-on-HVM drivers) and some PV.

This problem occurred more often for me under 2.6.32 than 2.6.31, I 
noticed. Since I made the switch to aio, I haven't seen a crash, but it 
hasn't been long enough for that to mean much.

Having extra caching in the dom0 is nice because it allows for domUs to 
get away with having small amounts of free memory, while still having 
very good (much faster than hardware) write performance. If you have a 
large number of domUs that are all memory-constrained and use the disk 
in infrequent, large bursts, this can work out pretty well, since the 
big communal pool provides a better value proposition than giving each 
domU a few more megabytes of RAM.

If the OOM problem isn't something that can be fixed, it might be a good 
idea to print out a warning to the user when a domain using "file:" is 
started. Or, to go a step further and automatically run "file" based 
domains as though "aio" was specified, possibly with a warning and a way 
to override that behavior. It's not really intuitive that "file" would 
cause crashes.

-John