From mboxrd@z Thu Jan 1 00:00:00 1970 From: James Dingwall Subject: Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning Date: Wed, 11 Dec 2013 09:25:09 +0000 Message-ID: <52A82F75.3000609@zynstra.com> References: <52A602E5.3080300@zynstra.com> <20131209214816.GA3000@phenom.dumpdata.com> <52A72AB8.9060707@zynstra.com> <20131210152746.GF3184@phenom.dumpdata.com> <52A812B0.6060607@oracle.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <52A812B0.6060607@oracle.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Bob Liu , Konrad Rzeszutek Wilk Cc: xen-devel@lists.xen.org List-Id: xen-devel@lists.xenproject.org Bob Liu wrote: > On 12/10/2013 11:27 PM, Konrad Rzeszutek Wilk wrote: >> On Tue, Dec 10, 2013 at 02:52:40PM +0000, James Dingwall wrote: >>> Konrad Rzeszutek Wilk wrote: >>>> On Mon, Dec 09, 2013 at 05:50:29PM +0000, James Dingwall wrote: >>>>> Hi, >>>>> >>>>> Since 3.11 I have noticed that the OOM killer quite frequently >>>>> triggers in my Xen guest domains which use ballooning to >>>>> increase/decrease their memory allocation according to their >>>>> requirements. One example domain I have has a maximum memory >>>>> setting of ~1.5Gb but it usually idles at ~300Mb, it is also >>>>> configured with 2Gb swap which is almost 100% free. >>>>> >>>>> # free >>>>> total used free shared buffers cached >>>>> Mem: 272080 248108 23972 0 1448 63064 >>>>> -/+ buffers/cache: 183596 88484 >>>>> Swap: 2097148 8 2097140 >>>>> >>>>> There is plenty of available free memory in the hypervisor to >>>>> balloon to the maximum size: >>>>> # xl info | grep free_mem >>>>> free_memory : 14923 >>>>> >>>>> An example trace (they are always the same) from the oom killer in >>>>> 3.12 is added below. So far I have not been able to reproduce this >>>>> at will so it is difficult to start bisecting it to see if a >>>>> particular change introduced this. However it does seem that the >>>>> behaviour is wrong because a) ballooning could give the guest more >>>>> memory, b) there is lots of swap available which could be used as a >>>>> fallback. >> Keep in mind that swap with tmem is actually no more swap. Heh, that >> sounds odd -but basically pages that are destined for swap end up >> going in the tmem code which pipes them up to the hypervisor. >> >>>>> If other information could help or there are more tests that I could >>>>> run then please let me know. >>>> I presume you have enabled 'tmem' both in the hypervisor and in >>>> the guest right? >>> Yes, domU and dom0 both have the tmem module loaded and tmem >>> tmem_dedup=on tmem_compress=on is given on the xen command line. >> Excellent. The odd thing is that your swap is not used that much, but >> it should be (as that is part of what the self-balloon is suppose to do). >> >> Bob, you had a patch for the logic of how self-balloon is suppose >> to account for the slab - would this be relevant to this problem? > Perhaps, I have attached the patch. > James, could you please apply it and try your application again? You > have to rebuild the guest kernel. > Oh, and also take a look at whether frontswap is in use, you can check > it by watching "cat /sys/kernel/debug/frontswap/*". I will test this patch later today and let you know how it goes. In the meantime I have found that loading tmem with selfshrinking=0 prevents the problem I originally reported. Frontswap is in use, these values are from shortly after a guest reboot: # for i in /sys/kernel/debug/frontswap/* ; do echo $i $(< $i) ; done /sys/kernel/debug/frontswap/failed_stores 0 /sys/kernel/debug/frontswap/invalidates 5 /sys/kernel/debug/frontswap/loads 83 /sys/kernel/debug/frontswap/succ_stores 107 # for i in /sys/module/tmem/parameters/* ; do echo $i $(< $i) ; done /sys/module/tmem/parameters/cleancache Y /sys/module/tmem/parameters/frontswap Y /sys/module/tmem/parameters/selfballooning Y /sys/module/tmem/parameters/selfshrinking N James > > balloon.patch > > > diff --git a/drivers/xen/xen-selfballoon.c b/drivers/xen/xen-selfballoon.c > index 21e18c1..4814759 100644 > --- a/drivers/xen/xen-selfballoon.c > +++ b/drivers/xen/xen-selfballoon.c > @@ -191,6 +191,8 @@ static void selfballoon_process(struct work_struct *work) > tgt_pages = cur_pages; /* default is no change */ > goal_pages = vm_memory_committed() + > totalreserve_pages + > + global_page_state(NR_SLAB_RECLAIMABLE) + > + global_page_state(NR_SLAB_UNRECLAIMABLE) + > MB2PAGES(selfballoon_reserved_mb); > #ifdef CONFIG_FRONTSWAP > /* allow space for frontswap pages to be repatriated */ > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 580a5f0..863b05c 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -16,6 +16,7 @@ > > #include > #include > +#include > #include > #include > #include > @@ -3075,7 +3076,7 @@ void show_free_areas(unsigned int filter) > " dirty:%lu writeback:%lu unstable:%lu\n" > " free:%lu slab_reclaimable:%lu slab_unreclaimable:%lu\n" > " mapped:%lu shmem:%lu pagetables:%lu bounce:%lu\n" > - " free_cma:%lu\n", > + " free_cma:%lu totalram:%lu balloontarget:%lu\n", > global_page_state(NR_ACTIVE_ANON), > global_page_state(NR_INACTIVE_ANON), > global_page_state(NR_ISOLATED_ANON), > @@ -3093,7 +3094,9 @@ void show_free_areas(unsigned int filter) > global_page_state(NR_SHMEM), > global_page_state(NR_PAGETABLE), > global_page_state(NR_BOUNCE), > - global_page_state(NR_FREE_CMA_PAGES)); > + global_page_state(NR_FREE_CMA_PAGES), > + totalram_pages, > + vm_memory_committed() + totalreserve_pages); > > for_each_populated_zone(zone) { > int i;