From mboxrd@z Thu Jan 1 00:00:00 1970 From: Kai Meyer Subject: Re: [Xen-users] xend (12, 'Cannot allocate memory') Date: Wed, 26 Mar 2008 09:25:30 -0600 Message-ID: <47EA6AEA.9030700@unixlords.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: xen-devel@lists.xensource.com List-Id: xen-devel@lists.xenproject.org Well, that's no so great. It thinks I have barely a fraction of what I'm supposed to have available. Free shows 10GB available, xm info shows 1 MB available. If I stop a virtual machine, the memory in xminfo increases by the amount of memory used by that xen machine, which is to be expected. Is there a way to force Xen to refresh that number, or let me force an override? -Kai Meyer Keir Fraser wrote: > Given you are running dom0 with plenty of swap, it's most likely that Xen > has run out of memory to allocate to your new domU, rather than the > allocation failure coming from within dom0. Probably auto-ballooning is > screwing up somehow (that's the bit of xend that gives away memory back to > Xen to make sure it has enough memory to create your new domU). > > Sorry I don't have a precise diagnosis for you, but I think looking at Xen's > allocator is at least the right direction for you to take. 'xm info' tells > you how much memory Xen thinks it has. > > -- Keir > > On 26/3/08 14:54, "Kai Meyer" wrote: > > >> I'm getting the error: >> xend (12, 'Cannot allocate memory') >> From the command line after the Dom-0 goes through a period of heavy >> memory usage. I don't have any hard numbers on how to duplicate it, but >> I'll give you the steps I've gone through to duplicate the problem. >> >> First, some information about my setup. It's a new setup, and has been >> in heavy development for the past few months. I'm new to Xen as of this >> project we started in January. As far as I can tell, this problem has >> existed since we started, and may have even happened before but we were >> too new to think it wasn't our fault (which it may still be....) I've >> found the same error addressed here: >> http://archive.netbsd.se/?ml=xen-devel&a=2007-04&t=3887921 >> The 'hack' to fix this problem is already built into my >> XendDomainInfo.py script already, so I'm a little more than lost. >> >> [root@xen1 ~]# xm info >> host : xen1.fiber.net >> release : 2.6.18-53.1.13.el5xen >> version : #1 SMP Tue Feb 12 13:33:07 EST 2008 >> machine : x86_64 >> nr_cpus : 8 >> nr_nodes : 1 >> sockets_per_node : 2 >> cores_per_socket : 4 >> threads_per_core : 1 >> cpu_mhz : 1866 >> hw_caps : >> bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001 >> total_memory : 16382 >> free_memory : 2 >> xen_major : 3 >> xen_minor : 1 >> xen_extra : .0-53.1.13.el5 >> xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 >> hvm-3.0-x86_32p hvm-3.0-x86_64 >> xen_pagesize : 4096 >> platform_params : virt_start=0xffff800000000000 >> xen_changeset : unavailable >> cc_compiler : gcc version 4.1.2 20070626 (Red Hat 4.1.2-14) >> cc_compile_by : mockbuild >> cc_compile_domain : >> cc_compile_date : Tue Feb 12 12:55:35 EST 2008 >> xend_config_format : 2 >> >> >> Here's the error in action: >> [root@xen1 centos-5.1_core_image]# xm create centos-5.1_core_image >> Using config file "./centos-5.1_core_image". >> Error: (12, 'Cannot allocate memory') >> [root@xen1 centos-5.1_core_image]# >> >> >> Here is the output from xend.log that occurs when I get this error: >> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >> (XendDomainInfo:200) XendDomainInfo.create(['vm', ['name', >> 'centos-5.1_core_image'], ['memory', 4024], ['maxmem', 4024], >> ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', >> 'restart'], ['vcpus', 6], ['image', ['hvm', ['kernel', >> '/usr/lib/xen/boot/hvmloader'], ['device_model', >> '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot', >> 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority', >> '/root/.Xauthority'], ['keymap', 'en-us']]], ['device', ['vbd', >> ['uname', 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'], >> ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname', >> 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'], >> ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr1'], ['mac', >> '00:16:3e:4c:c0:01'], ['type', 'ioemu']]]]) >> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >> (XendDomainInfo:306) parseConfig: config is ['vm', ['name', >> 'centos-5.1_core_image'], ['memory', 4024], ['maxmem', 4024], >> ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', >> 'restart'], ['vcpus', 6], ['image', ['hvm', ['kernel', >> '/usr/lib/xen/boot/hvmloader'], ['device_model', >> '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot', >> 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority', >> '/root/.Xauthority'], ['keymap', 'en-us']]], ['device', ['vbd', >> ['uname', 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'], >> ['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname', >> 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'], >> ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr1'], ['mac', >> '00:16:3e:4c:c0:01'], ['type', 'ioemu']]]] >> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >> (XendDomainInfo:411) parseConfig: result is {'shadow_memory': None, >> 'start_time': None, 'uuid': None, 'on_crash': 'restart', 'on_reboot': >> 'restart', 'localtime': None, 'image': ['hvm', ['kernel', >> '/usr/lib/xen/boot/hvmloader'], ['device_model', >> '/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot', >> 'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority', >> '/root/.Xauthority'], ['keymap', 'en-us']], 'on_poweroff': 'destroy', >> 'bootloader_args': None, 'cpus': None, 'name': 'centos-5.1_core_image', >> 'backend': [], 'vcpus': 6, 'cpu_weight': None, 'features': None, >> 'vcpu_avail': None, 'memory': 4024, 'device': [('vbd', ['vbd', ['uname', >> 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'], ['dev', >> 'hda'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', >> 'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'], >> ['mode', 'w']]), ('vif', ['vif', ['bridge', 'xenbr1'], ['mac', >> '00:16:3e:4c:c0:01'], ['type', 'ioemu']])], 'bootloader': None, 'cpu': >> None, 'maxmem': 4024} >> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG >> (XendDomainInfo:1349) XendDomainInfo.construct: None >> [2008-03-19 14:07:43 xend 3069] DEBUG (balloon:127) Balloon: 2276 KiB >> free; need 2048; done. >> [2008-03-19 14:07:43 xend.XendDomainInfo 3069] ERROR >> (XendDomainInfo:212) Domain construction failed >> Traceback (most recent call last): >> File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", >> line 204, in create >> vm.construct() >> File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", >> line 1369, in construct >> hvm = hvm) >> Error: (12, 'Cannot allocate memory') >> >> >> Here's the configuration file being used: >> name = "centos-5.1_core_image" >> maxmem = 4024 >> memory = 4024 >> vcpus = 6 >> builder = "hvm" >> kernel = "/usr/lib/xen/boot/hvmloader" >> boot = "cda" >> pae = 1 >> acpi = 0 >> apic = 0 >> on_poweroff = "destroy" >> on_reboot = "restart" >> on_crash = "restart" >> device_model = "/usr/lib64/xen/bin/qemu-dm" >> sdl = 0 >> vnc = 1 >> vncunused = 1 >> keymap = "en-us" >> disk = [ 'file:/xen/domains2/centos-5.1_core_image/disk.img,hda,w' , >> 'file:/xen/domains2/centos-5.1_core_image/swap.img,hdb,w' ] >> vif = [ "mac=00:16:3e:4c:c0:01,bridge=xenbr1,type=ioemu" ] >> serial = "pty" >> >> At the time the error occured, my memory looked like this: >> [root@xen1 centos-5.1_core_image]# free >> total used free shared buffers cached >> Mem: 15845376 15821080 24296 0 113596 15081468 >> -/+ buffers/cache: 626016 15219360 >> Swap: 4192956 4 4192952 >> >> The only way I've been able to duplicate this error is by making >> multiple copies of the 6GB disk images (created with dd, so ls shows 6GB >> but du shows 2GB) for different virtual machines and booting them up. I >> can get up to about 8 new virtual machines roughly, before I start >> seeing my memory error. If you're familiar with IBM's 'xen_deploy.pl' >> script, I've modified it slightly to meet our needs, and launch our >> systems with that tool. It's basic functionality is to read in a xen >> configuration file you want to use as a template, and output a modified >> one based on arguments you pass to the scripts, and then create a floppy >> disk image and copy the image from the template config file, and then >> boot the new system. >> >> I've read in previous emails that this error occurs because of a bug in >> the hvmloader, and look for an update. That email response was for a 3.0 >> version of xen, and I'm on 3.1. >> >> What other information can I provide to help diagnose this problem? >> >> -Kai Meyer >> >> _______________________________________________ >> Xen-devel mailing list >> Xen-devel@lists.xensource.com >> http://lists.xensource.com/xen-devel >> > > > > _______________________________________________ > Xen-devel mailing list > Xen-devel@lists.xensource.com > http://lists.xensource.com/xen-devel >