From mboxrd@z Thu Jan 1 00:00:00 1970 From: James Dingwall Subject: Re: Kernel 3.11 / 3.12 OOM killer and Xen ballooning Date: Tue, 10 Dec 2013 14:52:22 +0000 Message-ID: <52A72AA6.6010401@zynstra.com> References: <52A602E5.3080300@zynstra.com> <52A6DBEF020000780010BAC9@nat28.tlf.novell.com> <52A71ED5.5080709@zynstra.com> <52A73283020000780010BE04@nat28.tlf.novell.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; Format="flowed" Content-Transfer-Encoding: 7bit Return-path: Received: from mail6.bemta4.messagelabs.com ([85.158.143.247]) by lists.xen.org with esmtp (Exim 4.72) (envelope-from ) id 1VqOfp-0004go-7Q for xen-devel@lists.xenproject.org; Tue, 10 Dec 2013 14:52:33 +0000 In-Reply-To: <52A73283020000780010BE04@nat28.tlf.novell.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: Jan Beulich Cc: xen-devel List-Id: xen-devel@lists.xenproject.org Jan Beulich wrote: >>>> On 10.12.13 at 15:01, James Dingwall wrote: >> Jan Beulich wrote: >>>>>> On 09.12.13 at 18:50, James Dingwall wrote: >>>> Since 3.11 I have noticed that the OOM killer quite frequently triggers >>>> in my Xen guest domains which use ballooning to increase/decrease their >>>> memory allocation according to their requirements. One example domain I >>>> have has a maximum memory setting of ~1.5Gb but it usually idles at >>>> ~300Mb, it is also configured with 2Gb swap which is almost 100% free. >>>> >>>> # free >>>> total used free shared buffers cached >>>> Mem: 272080 248108 23972 0 1448 63064 >>>> -/+ buffers/cache: 183596 88484 >>>> Swap: 2097148 8 2097140 >>>> >>>> There is plenty of available free memory in the hypervisor to balloon to >>>> the maximum size: >>>> # xl info | grep free_mem >>>> free_memory : 14923 >>> But you don't tell us how you trigger the process of re-filling >>> memory. Yet that's the crucial point here; the fact that there is >>> enough memory available in the hypervisor is only a necessary >>> prerequisite. >> My guest systems are Gentoo based and most often the problems happen >> while running emerge (Python script). The example trace was taken from >> an emerge command launched via a cron script which runs emerge --sync ; >> emerge --deep --newuse -p -u world. Almost all the other times I have >> seen the behaviour have been during emerge --deep --newuse -u world >> runs, most often during the build of large packages such as kdelibs, >> seamonkey or libreoffice. However occasionally I have seen it with >> smaller builds or during the package merge phase where files are indexed >> and copied in to the live filesystem. > I don't think I understand what you're trying to tell me, or in what > way this answers the question. > >> I have done some testing with the program below which successfully fills >> all memory and swap before being killed. One thought I had was that >> perhaps there was some issue around a balloon down/up when the rate at >> which memory is being requested and released is high. I will try >> another program with a random pattern of malloc/free to see if I can >> make a test case to help with a bisect. > Again - the question is not how to drive your system out of > memory, but what entity (if any) it is that is supposed to react on > memory becoming tight and triggering the balloon driver to re- > populate pages. One such thing could be xenballoond (which is > gone in the unstable tree, i.e. what is to become 4.4). > > Jan Sorry, I misunderstood the question. I'm just relying on the built in kernel behaviour to balloon up/down as and when memory is required I wasn't aware there were multiple ways that this could be achieved. I'm not running anything in user space, so does tmem answer the question? When I modprobe tmem the kernel logs: [ 18.518714] xen:tmem: frontswap enabled, RAM provided by Xen Transcendent Memory [ 18.518739] xen:tmem: cleancache enabled, RAM provided by Xen Transcendent Memory [ 18.518741] xen_selfballoon: Initializing Xen selfballooning driver [ 18.518742] xen_selfballoon: Initializing frontswap selfshrinking driver The kernel command line for dom0 and domU contains tmem, the command line for xen contains tmem tmem_dedup=on tmem_compress=on James > -- *James Dingwall* Script Monkey zynstra-signature-logo twitter-black linkedin-black Zynstra is a private limited company registered in England and Wales (registered number 07864369). Our registered office is 5 New Street Square, London, EC4A 3TW and our headquarters are at Bath Ventures, Broad Quay, Bath, BA1 1UD.