Xen 3.0.1 Error creating domain: (12, 'Cannot allocate memory')

Xen 3.0.1 Error creating domain: (12, 'Cannot allocate memory')

[Peter]

Hi.  I have a domU that refuses to start up.  It reports an error 
allocating memory.  There appears to be sufficient memory in dom0 as 
well as on the host.  I can restart other domUs no problem.

To resolve the issue I'm having to restart the whole server.  Which is a 
bit disruptive.

Details follow.

Regards, Peter

xm create -f /home/users/fullserviceasians2/params  extra="init=/bin/bash"
Using config file "/home/users/fullserviceasians2/params".
Warning: The nics option is deprecated.  Please use an empty vif entry 
instead:

   vif = [ '' ]

Error: Error creating domain: (12, 'Cannot allocate memory')
[root@host73 fullserviceasians2]# free -m
              total       used       free     shared    buffers     cached
Mem:           192        173         19          0         17         56
-/+ buffers/cache:         99         92
Swap:         2996          6       2989
[root@host73 fullserviceasians2]# xm info
system                 : Linux
release                : 2.6.12.6-xen0
version                : #2 Fri Feb 3 22:40:46 UTC 2006
machine                : i686
nr_cpus                : 2
nr_nodes               : 1
sockets_per_node       : 2
cores_per_socket       : 1
threads_per_core       : 1
cpu_mhz                : 1994
hw_caps                : 078bfbff:e1d3fbff:00000000:00000010
total_memory           : 8192
free_memory            : 2902
xen_major              : 3
xen_minor              : 0
xen_extra              : .1
xen_caps               : xen-3.0-x86_32p
platform_params        : virt_start=0xf5800000
xen_changeset          : Tue Jan 31 11:04:12 2006 +0100 8738:eff96462fde8
cc_compiler            : gcc version 3.4.4 20050721 (Red Hat 3.4.4-2)
cc_compile_by          : root
cc_compile_date        : Sat Feb  4 01:05:28 UTC 2006

Re: Xen 3.0.1 Error creating domain: (12, 'Cannot allocate memory')

[Keir Fraser]

On 27 Feb 2006, at 09:53, Peter wrote:

> Hi.  I have a domU that refuses to start up.  It reports an error 
> allocating memory.  There appears to be sufficient memory in dom0 as 
> well as on the host.  I can restart other domUs no problem.
>
> To resolve the issue I'm having to restart the whole server.  Which is 
> a bit disruptive.

How many domains are already running? Xen's own private heap can be 
exhausted even when there is guest memory available.

  -- Keir

Re: Xen 3.0.1 Error creating domain: (12, 'Cannot allocate memory')

[Peter]

Keir Fraser wrote:
> 
> On 27 Feb 2006, at 09:53, Peter wrote:
> 
>> Hi.  I have a domU that refuses to start up.  It reports an error 
>> allocating memory.  There appears to be sufficient memory in dom0 as 
>> well as on the host.  I can restart other domUs no problem.
>>
>> To resolve the issue I'm having to restart the whole server.  Which is 
>> a bit disruptive.
> 
> How many domains are already running? Xen's own private heap can be 
> exhausted even when there is guest memory available.
> 
>  -- Keir
> 
I have 41 domains running (including dom0).

Is there a way to help avoid Xen's private heap getting exhausted?  e.g. 
increase the memory allocated to dom0?  Or a setting change?  or?

-- Peter

Re: Xen 3.0.1 Error creating domain: (12, 'Cannot allocate memory')

[Keir Fraser]

On 27 Feb 2006, at 20:58, Peter wrote:

>> How many domains are already running? Xen's own private heap can be 
>> exhausted even when there is guest memory available.
>>  -- Keir
> I have 41 domains running (including dom0).
>
> Is there a way to help avoid Xen's private heap getting exhausted?  
> e.g. increase the memory allocated to dom0?  Or a setting change?  or?

Unfortunately not, but I'd expect you to be able to create more than 41 
domains. When we've tested this we've been able to create about double 
that number.

  -- Keir

Re: Xen 3.0.1 Error creating domain: (12, 'Cannot allocate memory')

[Peter]

Keir Fraser wrote:
>> Is there a way to help avoid Xen's private heap getting exhausted?  
>> e.g. increase the memory allocated to dom0?  Or a setting change?  or?
> 
> Unfortunately not, but I'd expect you to be able to create more than 41 
> domains. When we've tested this we've been able to create about double 
> that number.
We've hit this before.  I'm not sure if it was on the same host server 
or not.  Is there any other information we can collect that would be 
useful resolving this?

Or are there any actions we can try to see if we can fix the issue 
(other than a reboot of the host server which seems to do the job, but 
is obviously a bit disruptive to other domUs).

-- Peter

Re: Xen 3.0.1 Error creating domain: (12, 'Cannot allocate memory')

[Keir Fraser]

On 28 Feb 2006, at 09:53, Peter wrote:

>> Unfortunately not, but I'd expect you to be able to create more than 
>> 41 domains. When we've tested this we've been able to create about 
>> double that number.
> We've hit this before.  I'm not sure if it was on the same host server 
> or not.  Is there any other information we can collect that would be 
> useful resolving this?
>
> Or are there any actions we can try to see if we can fix the issue 
> (other than a reboot of the host server which seems to do the job, but 
> is obviously a bit disruptive to other domUs).

I just added a 'm' debug key to the unstable repository. If you have a 
serial line on the offending machine and apply that patch to your tree, 
you can get free-memory infomation when the failure happens and confirm 
that the Xen heap is depleted.

  -- Keir