From mboxrd@z Thu Jan 1 00:00:00 1970 From: James Dingwall Subject: Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning Date: Thu, 16 Jan 2014 10:52:01 +0000 Message-ID: <52D7B9D1.5080907@zynstra.com> References: <52A602E5.3080300@zynstra.com> <20131209214816.GA3000@phenom.dumpdata.com> <52A72AB8.9060707@zynstra.com> <20131210152746.GF3184@phenom.dumpdata.com> <52A812B0.6060607@oracle.com> <52A89334.3090007@zynstra.com> <52B18F44.2030500@oracle.com> <52B3443F.5060704@zynstra.com> <52B3B6D7.50606@oracle.com> <52BBEBEF.8040509@zynstra.com> <52C50661.7060900@oracle.com> <52CBC700.1060602@zynstra.com> <52CE7E67.5080708@oracle.com> <52D64B87.6000400@zynstra.com> <52D69E0B.5020006@oracle.com> <52D6B8B6.5070302@zynstra.com> <52D7346A.3000300@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <52D7346A.3000300@oracle.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Bob Liu Cc: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org Bob Liu wrote: > On 01/16/2014 12:35 AM, James Dingwall wrote: >> Bob Liu wrote: >>> On 01/15/2014 04:49 PM, James Dingwall wrote: >>>> Bob Liu wrote: >>>>> On 01/07/2014 05:21 PM, James Dingwall wrote: >>>>>> Bob Liu wrote: >>>>>>> Could you confirm that this problem doesn't exist if loading tmem >>>>>>> with >>>>>>> selfshrinking=0 during compile gcc? It seems that you are compiling >>>>>>> difference packages during your testing. >>>>>>> This will help to figure out whether selfshrinking is the root cause. >>>>>> Got an oom with selfshrinking=0, again during a gcc compile. >>>>>> Unfortunately I don't have a single test case which demonstrates the >>>>>> problem but as I mentioned before it will generally show up under >>>>>> compiles of large packages such as glibc, kdelibs, gcc etc. >>>>>> >>>>> So the root cause is not because enabled selfshrinking. >>>>> Then what I can think of is that the xen-selfballoon driver was too >>>>> aggressive, too many pages were ballooned out which causeed heavy >>>>> memory >>>>> pressure to guest OS. >>>>> And kswapd started to reclaim page until most of pages were >>>>> unreclaimable(all_unreclaimable=yes for all zones), then OOM Killer was >>>>> triggered. >>>>> In theory the balloon driver should give back ballooned out pages to >>>>> guest OS, but I'm afraid this procedure is not fast enough. >>>>> >>>>> My suggestion is reserve a min memory for your guest OS so that the >>>>> xen-selfballoon won't be so aggressive. >>>>> You can do it through parameters selfballoon_reserved_mb or >>>>> selfballoon_min_usable_mb. >>>> I am still getting OOM errors with both of these set to 32 so I'll try >>>> another bump to 64. I think that if I do find values which prevent it >>>> though then it is more of a work around than a fix because it still >>>> suggests that swap is not being used when ballooning is no longer >>> Yes, it's like a work around. But I don't think there is a better way. >>> >>> From the recent OOM log your reported: >>> [ 8212.940769] Free swap = 1925576kB >>> [ 8212.940770] Total swap = 2097148kB >>> >>> [504638.442136] Free swap = 1868108kB >>> [504638.442137] Total swap = 2097148kB >>> >>> 171572KB and 229040KB data are swapped out to swap disk, I think there >>> are already significantly values for guest OS has only ~300M usable >>> memory. >>> guest OS can't find out pages suitable for swap any more after so many >>> pages are swapped out, although at that time the swap device still have >>> enough space. >>> >>> The OOM may not be triggered if the balloon driver can give back memory >>> to guest OS fast enough but I think it's unrealistic. >>> So the best way is reserve more memory to guest OS. >>> >>>> capable of satisfying the request. I've also got an Ubuntu Saucy (3.11 >>>> kernel) guest running on the dom0 with tmem activated so I'm going to >>>> see if I can find a comparable workload to see if I get the same issue >>>> with a different kernel configuration. >>>> >> I've done a bit more testing and seem to have found an extra condition >> which is affecting the OOM behaviour in my guests. All my Gentoo guests >> have swap space backed by a dm-crypt volume and if I remove this layer >> then things seem to be behaving much more reliably. In my Ubuntu guests >> I have plain swap space and so far I haven't been able to trigger an OOM >> condition. Is it possible that it is the dm-crypt layer failing to get >> working memory when swapping something in/out and causing the error? >> > One possible reason is the dm layer and related dm target driver occupy > a significant mount of memory and there is no way for xenselfballoon to > know this. So selfballoon driver ballooned out more memory than the > system really requires. > > I have made a patch by reserving extra 10% of original total memory, by > this way I think we can make the system much more reliably in all cases. > Could you please have a test? You don't need to set > selfballoon_reserved_mb by yourself any more. I'm running a 3.12.7 kernel with all 3 patches and original swap configuration, so far so good. I shall keep testing and let you know how things go in a few days. Thanks, James /sys/module/tmem/parameters/cleancache Y /sys/module/tmem/parameters/frontswap Y /sys/module/tmem/parameters/selfballooning Y /sys/module/tmem/parameters/selfshrinking Y /sys/devices/system/xen_memory/xen_memory0/selfballoon/frontswap_hysteresis 20 /sys/devices/system/xen_memory/xen_memory0/selfballoon/frontswap_inertia 6 /sys/devices/system/xen_memory/xen_memory0/selfballoon/frontswap_selfshrinking 1 /sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_downhysteresis 8 /sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballooning 1 /sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_interval 5 /sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_min_usable_mb 0 /sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_reserved_mb 74 /sys/devices/system/xen_memory/xen_memory0/selfballoon/selfballoon_uphysteresis 1 > > xen_selfballoon_deaggressive.patch > > > diff --git a/drivers/xen/xen-selfballoon.c b/drivers/xen/xen-selfballoon.c > index 21e18c1..8f33254 100644 > --- a/drivers/xen/xen-selfballoon.c > +++ b/drivers/xen/xen-selfballoon.c > @@ -175,6 +175,7 @@ static void frontswap_selfshrink(void) > #endif /* CONFIG_FRONTSWAP */ > > #define MB2PAGES(mb) ((mb) << (20 - PAGE_SHIFT)) > +#define PAGES2MB(pages) ((pages) >> (20 - PAGE_SHIFT)) > > /* > * Use current balloon size, the goal (vm_committed_as), and hysteresis > @@ -525,6 +526,7 @@ EXPORT_SYMBOL(register_xen_selfballooning); > int xen_selfballoon_init(bool use_selfballooning, bool use_frontswap_selfshrink) > { > bool enable = false; > + unsigned long reserve_pages; > > if (!xen_domain()) > return -ENODEV; > @@ -549,6 +551,26 @@ int xen_selfballoon_init(bool use_selfballooning, bool use_frontswap_selfshrink) > if (!enable) > return -ENODEV; > > + /* > + * Give selfballoon_reserved_mb a default value(10% of total ram pages) > + * to make selfballoon not so aggressive. > + * > + * There are two reasons: > + * 1) The goal_page doesn't contain some pages used by kernel space, > + * like slab cache and pages used by device drivers. > + * > + * 2) The balloon driver may not give back memory to guest OS fast > + * enough when the workload suddenly aquries a lot of memory. > + * > + * In both cases, the guest OS will suffer from memory pressure and > + * OOM killer may be triggered. > + * By reserving extra 10% of total ram pages, we can keep the system > + * much more reliably and response faster in some cases. > + */ > + if (!selfballoon_reserved_mb) { > + reserve_pages = totalram_pages / 10; > + selfballoon_reserved_mb = PAGES2MB(reserve_pages); > + } > schedule_delayed_work(&selfballoon_worker, selfballoon_interval * HZ); > > return 0; >