[Xen-users] xend (12, 'Cannot allocate memory')

[Kai Meyer <kai@unixlords.com>]

I'm getting the error:
xend (12, 'Cannot allocate memory')
 From the command line after the Dom-0 goes through a period of heavy 
memory usage. I don't have any hard numbers on how to duplicate it, but 
I'll give you the steps I've gone through to duplicate the problem.

First, some information about my setup. It's a new setup, and has been 
in heavy development for the past few months. I'm new to Xen as of this 
project we started in January. As far as I can tell, this problem has 
existed since we started, and may have even happened before but we were 
too new to think it wasn't our fault (which it may still be....)  I've 
found the same error addressed here:
http://archive.netbsd.se/?ml=xen-devel&a=2007-04&t=3887921
The 'hack' to fix this problem is already built into my 
XendDomainInfo.py script already, so I'm a little more than lost.

[root@xen1 ~]# xm info
host                   : xen1.fiber.net
release                : 2.6.18-53.1.13.el5xen
version                : #1 SMP Tue Feb 12 13:33:07 EST 2008
machine                : x86_64
nr_cpus                : 8
nr_nodes               : 1
sockets_per_node       : 2
cores_per_socket       : 4
threads_per_core       : 1
cpu_mhz                : 1866
hw_caps                : 
bfebfbff:20100800:00000000:00000140:0004e3bd:00000000:00000001
total_memory           : 16382
free_memory            : 2
xen_major              : 3
xen_minor              : 1
xen_extra              : .0-53.1.13.el5
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 
hvm-3.0-x86_32p hvm-3.0-x86_64
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20070626 (Red Hat 4.1.2-14)
cc_compile_by          : mockbuild
cc_compile_domain      :
cc_compile_date        : Tue Feb 12 12:55:35 EST 2008
xend_config_format     : 2


Here's the error in action:
[root@xen1 centos-5.1_core_image]# xm create centos-5.1_core_image
Using config file "./centos-5.1_core_image".
Error: (12, 'Cannot allocate memory')
[root@xen1 centos-5.1_core_image]#


Here is the output from xend.log that occurs when I get this error:
[2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG 
(XendDomainInfo:200) XendDomainInfo.create(['vm', ['name', 
'centos-5.1_core_image'], ['memory', 4024], ['maxmem', 4024], 
['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 
'restart'], ['vcpus', 6], ['image', ['hvm', ['kernel', 
'/usr/lib/xen/boot/hvmloader'], ['device_model', 
'/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot', 
'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority', 
'/root/.Xauthority'], ['keymap', 'en-us']]], ['device', ['vbd', 
['uname', 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'], 
['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname', 
'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'], 
['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr1'], ['mac', 
'00:16:3e:4c:c0:01'], ['type', 'ioemu']]]])
[2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG 
(XendDomainInfo:306) parseConfig: config is ['vm', ['name', 
'centos-5.1_core_image'], ['memory', 4024], ['maxmem', 4024], 
['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['on_crash', 
'restart'], ['vcpus', 6], ['image', ['hvm', ['kernel', 
'/usr/lib/xen/boot/hvmloader'], ['device_model', 
'/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot', 
'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority', 
'/root/.Xauthority'], ['keymap', 'en-us']]], ['device', ['vbd', 
['uname', 'file:/xen/domains2/centos-5.1_core_image/new_disk.img'], 
['dev', 'hda'], ['mode', 'w']]], ['device', ['vbd', ['uname', 
'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'], 
['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr1'], ['mac', 
'00:16:3e:4c:c0:01'], ['type', 'ioemu']]]]
[2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG 
(XendDomainInfo:411) parseConfig: result is {'shadow_memory': None, 
'start_time': None, 'uuid': None, 'on_crash': 'restart', 'on_reboot': 
'restart', 'localtime': None, 'image': ['hvm', ['kernel', 
'/usr/lib/xen/boot/hvmloader'], ['device_model', 
'/usr/lib64/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 6], ['boot', 
'cda'], ['serial', 'pty'], ['vnc', 1], ['vncunused', 1], ['xauthority', 
'/root/.Xauthority'], ['keymap', 'en-us']], 'on_poweroff': 'destroy', 
'bootloader_args': None, 'cpus': None, 'name': 'centos-5.1_core_image', 
'backend': [], 'vcpus': 6, 'cpu_weight': None, 'features': None, 
'vcpu_avail': None, 'memory': 4024, 'device': [('vbd', ['vbd', ['uname', 
'file:/xen/domains2/centos-5.1_core_image/new_disk.img'], ['dev', 
'hda'], ['mode', 'w']]), ('vbd', ['vbd', ['uname', 
'file:/xen/domains2/centos-5.1_core_image/swap.img'], ['dev', 'hdb'], 
['mode', 'w']]), ('vif', ['vif', ['bridge', 'xenbr1'], ['mac', 
'00:16:3e:4c:c0:01'], ['type', 'ioemu']])], 'bootloader': None, 'cpu': 
None, 'maxmem': 4024}
[2008-03-19 14:07:43 xend.XendDomainInfo 3069] DEBUG 
(XendDomainInfo:1349) XendDomainInfo.construct: None
[2008-03-19 14:07:43 xend 3069] DEBUG (balloon:127) Balloon: 2276 KiB 
free; need 2048; done.
[2008-03-19 14:07:43 xend.XendDomainInfo 3069] ERROR 
(XendDomainInfo:212) Domain construction failed
Traceback (most recent call last):
 File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", 
line 204, in create
   vm.construct()
 File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py", 
line 1369, in construct
   hvm = hvm)
Error: (12, 'Cannot allocate memory')


Here's the configuration file being used:
name = "centos-5.1_core_image"
maxmem = 4024
memory = 4024
vcpus = 6
builder = "hvm"
kernel = "/usr/lib/xen/boot/hvmloader"
boot = "cda"
pae = 1
acpi = 0
apic = 0
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
device_model = "/usr/lib64/xen/bin/qemu-dm"
sdl = 0
vnc = 1
vncunused = 1
keymap = "en-us"
disk = [ 'file:/xen/domains2/centos-5.1_core_image/disk.img,hda,w' , 
'file:/xen/domains2/centos-5.1_core_image/swap.img,hdb,w' ]
vif = [ "mac=00:16:3e:4c:c0:01,bridge=xenbr1,type=ioemu" ]
serial = "pty"

At the time the error occured, my memory looked like this:
[root@xen1 centos-5.1_core_image]# free
            total       used       free     shared    buffers     cached
Mem:      15845376   15821080      24296          0     113596   15081468
-/+ buffers/cache:     626016   15219360
Swap:      4192956          4    4192952

The only way I've been able to duplicate this error is by making 
multiple copies of the 6GB disk images (created with dd, so ls shows 6GB 
but du shows 2GB) for different virtual machines and booting them up. I 
can get up to about 8 new virtual machines roughly, before I start 
seeing my memory error. If you're familiar with IBM's 'xen_deploy.pl' 
script, I've modified it slightly to meet our needs, and launch our 
systems with that tool. It's basic functionality is to read in a xen 
configuration file you want to use as a template, and output a modified 
one based on arguments you pass to the scripts, and then create a floppy 
disk image and copy the image from the template config file, and then 
boot the new system.

I've read in previous emails that this error occurs because of a bug in 
the hvmloader, and look for an update. That email response was for a 3.0 
version of xen, and I'm on 3.1.

What other information can I provide to help diagnose this problem?

-Kai Meyer