3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory

3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory

[Daniel P. Berrange]

I'm testing on the Xen 3.0.5 rc3 (#14934) hypervisor and userspace. On the
whole it looks pretty good - the only significant problem I seeing is that
when booting a fullyvirt guest, the Domain-0 is not being ballooned down
to make room for its memory.

[2007-04-26 18:03:31 2654] ERROR (xmlrpclib2:178) Internal error handling xend.domain.create
Traceback (most recent call last):
  File "/usr/lib64/python2.5/site-packages/xen/util/xmlrpclib2.py", line 131, in _marshaled_dispatch
    response = self._dispatch(method, params)
  File "/usr/lib64/python2.5/SimpleXMLRPCServer.py", line 415, in _dispatch
    return func(*params)
  File "/usr/lib64/python2.5/site-packages/xen/xend/server/XMLRPCServer.py", line 72, in domain_create
    info = XendDomain.instance().domain_create(config)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomain.py", line 914, in domain_create
    dominfo = XendDomainInfo.create(config)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 81, in create
    vm.start()
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 395, in start
    XendTask.log_progress(0, 30, self._constructDomain)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendTask.py", line 209, in log_progress
    retval = func(*args, **kwds)
  File "/usr/lib64/python2.5/site-packages/xen/xend/XendDomainInfo.py", line 1430, in _constructDomain
    hvm = int(hvm))
Error: (12, 'Cannot allocate memory')

What is interesting is that the start() method does:

             XendTask.log_progress(0, 30, self._constructDomain)
             XendTask.log_progress(31, 60, self._initDomain)

The _constructDomain call is failing in xc.domain_create() - the code which
deals with ballooning down Domain-0 is not even run yet - that lives in
the _initDomain method. So HVM domain creation is failing before XenD has
even had a chance to do ballooning. If I manually use 'xm mem-set' to
balloon down Domain-0 ahead of time, everything works just fine.

I'm at a lose to figure out how to solve this without major re-arranging
of the order of _constructDomain & _initDomain work. Perhaps someone can
figure out an easier fix ?

This is on the x86_64 architecture and the guest config is:

name = "rhel3i386"
builder = "hvm"
memory = "500"
disk = [ 'file:/xen/rhel3i386.img,hda,w', ]
vif = [ 'type=ioemu, mac=00:16:3e:0c:e4:e3, bridge=virbr0', ]
uuid = "06017e0c-4f74-d1d0-017c-def49abd8c9c"
device_model = "/usr/lib64/xen/bin/qemu-dm"
kernel = "/usr/lib/xen/boot/hvmloader"
vnc=1
vncunused=1
apic=1
acpi=0
pae=1
vcpus=1
serial = "pty"
on_reboot   = 'restart'
on_crash    = 'restart'

The domain-0 host info is

# xm info
host                   : celery.virt.boston.redhat.com
release                : 2.6.20-2925.5.fc7ehabkosttestbuild2xen
version                : #1 SMP Thu Apr 26 16:43:56 EDT 2007
machine                : x86_64
nr_cpus                : 2
nr_nodes               : 1
sockets_per_node       : 1
cores_per_socket       : 2
threads_per_core       : 1
cpu_mhz                : 2211
hw_caps                : 178bfbff:ebd3fbff:00000000:00000010:00002001:00000000:0000001f
total_memory           : 4095
free_memory            : 0
xen_major              : 3
xen_minor              : 0
xen_extra              : .5-rc3-2925.5.f
xen_caps               : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64 
xen_scheduler          : credit
xen_pagesize           : 4096
platform_params        : virt_start=0xffff800000000000
xen_changeset          : unavailable
cc_compiler            : gcc version 4.1.2 20070424 (Red Hat 4.1.2-11)
cc_compile_by          : brewbuilder
cc_compile_domain      : build.redhat.com
cc_compile_date        : Thu Apr 26 16:40:09 EDT 2007
xend_config_format     : 4

And there is nothing being reported by 'xm dmesg' when the failures
occur.

I've not had a chance to try & reproduce this on i386 yet. Hopefully will
do that tomorrrow.

Regards,
Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory

[Keir Fraser]

On 27/4/07 00:12, "Daniel P. Berrange" <berrange@redhat.com> wrote:

> The _constructDomain call is failing in xc.domain_create() - the code which
> deals with ballooning down Domain-0 is not even run yet - that lives in
> the _initDomain method. So HVM domain creation is failing before XenD has
> even had a chance to do ballooning. If I manually use 'xm mem-set' to
> balloon down Domain-0 ahead of time, everything works just fine.
> 
> I'm at a lose to figure out how to solve this without major re-arranging
> of the order of _constructDomain & _initDomain work. Perhaps someone can
> figure out an easier fix ?

I don't think domain_create() allocates very many pages, although it's
possible it's making a default allocation of shadow memory. Assuming not, it
may be sufficient to make auto-balloon leave a bit of slack memory rather
than try to account for all available memory down to the last page. A few
pages might turn out to suffice. Does this problem occur with the very first
domain you try to create after booting the machine?

 -- Keir

Re: 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory

[Daniel P. Berrange]

On Fri, Apr 27, 2007 at 07:53:24AM +0100, Keir Fraser wrote:
> On 27/4/07 00:12, "Daniel P. Berrange" <berrange@redhat.com> wrote:
> 
> > The _constructDomain call is failing in xc.domain_create() - the code which
> > deals with ballooning down Domain-0 is not even run yet - that lives in
> > the _initDomain method. So HVM domain creation is failing before XenD has
> > even had a chance to do ballooning. If I manually use 'xm mem-set' to
> > balloon down Domain-0 ahead of time, everything works just fine.
> > 
> > I'm at a lose to figure out how to solve this without major re-arranging
> > of the order of _constructDomain & _initDomain work. Perhaps someone can
> > figure out an easier fix ?
> 
> I don't think domain_create() allocates very many pages, although it's
> possible it's making a default allocation of shadow memory. Assuming not, it
> may be sufficient to make auto-balloon leave a bit of slack memory rather
> than try to account for all available memory down to the last page. A few
> pages might turn out to suffice. Does this problem occur with the very first
> domain you try to create after booting the machine?

It seems to occurr with any HVM guest I try to create, unless there was already
some free mem (from starting & stopping a previous dom). I'm testing with this
very simple hack which seems to make it work - just ensure the balloon driver
frees a 10 MB chunk right at the start. Hardcoding is fine because when we
later get to the _initDomain() method we'll definitely be freeing up much
more than 10 MB for the HVM guests's actual mem requirements.

--- xen-3.0.5-testing.hg-rc3-14934/tools/python/xen/xend/XendDomainInfo.py      2007-04-26 19:30:11.000000000 -0400
+++ xen-3.0.5-testing.hg-rc3-14934.new/tools/python/xen/xend/XendDomainInfo.py  2007-04-26 19:36:06.000000000 -0400
@@ -1422,6 +1422,8 @@ class XendDomainInfo:
                 raise VmError("HVM guest support is unavailable: is VT/AMD-V "
                               "supported by your CPU and enabled in your "
                               "BIOS?")
+            # Hack to pre-reserve some mem for basic HVM setup
+            balloon.free(10*1024)
 
         self.domid = xc.domain_create(
             domid = 0,


Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory

[Keir Fraser]

On 27/4/07 14:26, "Daniel P. Berrange" <berrange@redhat.com> wrote:

> It seems to occurr with any HVM guest I try to create, unless there was
> already
> some free mem (from starting & stopping a previous dom). I'm testing with this
> very simple hack which seems to make it work - just ensure the balloon driver
> frees a 10 MB chunk right at the start. Hardcoding is fine because when we
> later get to the _initDomain() method we'll definitely be freeing up much
> more than 10 MB for the HVM guests's actual mem requirements.

We do most testing with a sensible 'dom0_mem=' so we wouldn't see this.

The hack looks fine.

 -- Keir

Re: 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory

[Daniel P. Berrange]

On Fri, Apr 27, 2007 at 02:56:48PM +0100, Keir Fraser wrote:
> On 27/4/07 14:26, "Daniel P. Berrange" <berrange@redhat.com> wrote:
> 
> > It seems to occurr with any HVM guest I try to create, unless there was
> > already
> > some free mem (from starting & stopping a previous dom). I'm testing with this
> > very simple hack which seems to make it work - just ensure the balloon driver
> > frees a 10 MB chunk right at the start. Hardcoding is fine because when we
> > later get to the _initDomain() method we'll definitely be freeing up much
> > more than 10 MB for the HVM guests's actual mem requirements.
> 
> We do most testing with a sensible 'dom0_mem=' so we wouldn't see this.

Well 'sensible' depends on your target audience :-) In the desktop/laptop
arena users like their Dom0 to have all the memory which isn't used by
active guests - even to the extent we've had bugs reported asking us to
make XenD automatically balloon dom0 back up after a guest shuts down !

Dan.
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory

[Keir Fraser]

On 27/4/07 15:09, "Daniel P. Berrange" <berrange@redhat.com> wrote:

>> We do most testing with a sensible 'dom0_mem=' so we wouldn't see this.
> 
> Well 'sensible' depends on your target audience :-) In the desktop/laptop
> arena users like their Dom0 to have all the memory which isn't used by
> active guests - even to the extent we've had bugs reported asking us to
> make XenD automatically balloon dom0 back up after a guest shuts down !

Oh yes. If dom0 is your client OS and virtualisation is a secondary activity
on your box then this makes perfect sense!

 -- Keir

Re: 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory

[Charles Coffing]

On Fri, Apr 27, 2007 at  7:26 AM, "Daniel P. Berrange" <berrange@redhat.com> wrote: 
> On Fri, Apr 27, 2007 at 07:53:24AM +0100, Keir Fraser wrote:
>> On 27/4/07 00:12, "Daniel P. Berrange" <berrange@redhat.com> wrote:
>> 
>> > The _constructDomain call is failing in xc.domain_create() - the code which
>> > deals with ballooning down Domain-0 is not even run yet - that lives in
>> > the _initDomain method. So HVM domain creation is failing before XenD has
>> > even had a chance to do ballooning. If I manually use 'xm mem-set' to
>> > balloon down Domain-0 ahead of time, everything works just fine.
>> > 
>> > I'm at a lose to figure out how to solve this without major re-arranging
>> > of the order of _constructDomain & _initDomain work. Perhaps someone can
>> > figure out an easier fix ?
>> 
>> I don't think domain_create() allocates very many pages, although it's
>> possible it's making a default allocation of shadow memory. Assuming not, it
>> may be sufficient to make auto-balloon leave a bit of slack memory rather
>> than try to account for all available memory down to the last page. A few
>> pages might turn out to suffice. Does this problem occur with the very first
>> domain you try to create after booting the machine?
> 
> It seems to occurr with any HVM guest I try to create, unless there was 
> already
> some free mem (from starting & stopping a previous dom). I'm testing with 
> this
> very simple hack which seems to make it work - just ensure the balloon driver
> frees a 10 MB chunk right at the start. Hardcoding is fine because when we
> later get to the _initDomain() method we'll definitely be freeing up much
> more than 10 MB for the HVM guests's actual mem requirements.


Keir,

The problem Daniel fixed for fully virtual affects paravirtual, also.

We have received bug reports from ISVs of this problem in both the PV and FV cases.  I traced code paths and discovered that FV attempts to allocate the most memory during the initial domain creation; 64-bit PV allocates the next most; 32-bit PV allocates the least.  So failures for PV are less common, but not impossible.

Would you please de-indent the added lines, so both PV and FV guests have the 2 MB slack?

I've been testing such a patch for about a month in our SLES 10 SP1 betas (based on xen-3.0.4), and it works well.

Thanks.

 
> --- xen-3.0.5-testing.hg-rc3-14934/tools/python/xen/xend/XendDomainInfo.py      
> 2007-04-26 19:30:11.000000000 -0400
> +++ xen-3.0.5-testing.hg-rc3-14934.new/tools/python/xen/xend/XendDomainInfo.py  
> 2007-04-26 19:36:06.000000000 -0400
> @@ -1422,6 +1422,8 @@ class XendDomainInfo:
>                  raise VmError("HVM guest support is unavailable: is VT/AMD-V 
> "
>                                "supported by your CPU and enabled in your "
>                                "BIOS?")
> +            # Hack to pre-reserve some mem for basic HVM setup
> +            balloon.free(10*1024)
>  
>          self.domid = xc.domain_create(
>              domid = 0,
> 
> 
> Dan.

Re: 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory

[Daniel P. Berrange]

On Mon, Apr 30, 2007 at 02:59:08PM -0400, Charles Coffing wrote:
> On Fri, Apr 27, 2007 at  7:26 AM, "Daniel P. Berrange" <berrange@redhat.com> wrote: 
> > On Fri, Apr 27, 2007 at 07:53:24AM +0100, Keir Fraser wrote:
> >> On 27/4/07 00:12, "Daniel P. Berrange" <berrange@redhat.com> wrote:
> >> 
> >> > The _constructDomain call is failing in xc.domain_create() - the code which
> >> > deals with ballooning down Domain-0 is not even run yet - that lives in
> >> > the _initDomain method. So HVM domain creation is failing before XenD has
> >> > even had a chance to do ballooning. If I manually use 'xm mem-set' to
> >> > balloon down Domain-0 ahead of time, everything works just fine.
> >> > 
> >> > I'm at a lose to figure out how to solve this without major re-arranging
> >> > of the order of _constructDomain & _initDomain work. Perhaps someone can
> >> > figure out an easier fix ?
> >> 
> >> I don't think domain_create() allocates very many pages, although it's
> >> possible it's making a default allocation of shadow memory. Assuming not, it
> >> may be sufficient to make auto-balloon leave a bit of slack memory rather
> >> than try to account for all available memory down to the last page. A few
> >> pages might turn out to suffice. Does this problem occur with the very first
> >> domain you try to create after booting the machine?
> > 
> > It seems to occurr with any HVM guest I try to create, unless there was 
> > already
> > some free mem (from starting & stopping a previous dom). I'm testing with 
> > this
> > very simple hack which seems to make it work - just ensure the balloon driver
> > frees a 10 MB chunk right at the start. Hardcoding is fine because when we
> > later get to the _initDomain() method we'll definitely be freeing up much
> > more than 10 MB for the HVM guests's actual mem requirements.
> 
> 
> Keir,
> 
> The problem Daniel fixed for fully virtual affects paravirtual, also.
> 
> We have received bug reports from ISVs of this problem in both the PV 
> and FV cases.  I traced code paths and discovered that FV attempts to 
> allocate the most memory during the initial domain creation; 64-bit PV 
> allocates the next most; 32-bit PV allocates the least.  So failures 
> for PV are less common, but not impossible.

Indeed - in testing on 64-bit PV over the weekend I saw perhaps 1 in 5
PV creation attempts fail with the not enough memory bug. So it does look
like the patch needs to be de-indented to fix PV too.

Regards,
Dan
-- 
|=- Red Hat, Engineering, Emerging Technologies, Boston.  +1 978 392 2496 -=|
|=-           Perl modules: http://search.cpan.org/~danberr/              -=|
|=-               Projects: http://freshmeat.net/~danielpb/               -=|
|=-  GnuPG: 7D3B9505   F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505  -=| 

Re: 3.0.5-rc3: Creating HVM guest fails with: Cannot allocate memory

[Keir Fraser]

On 30/4/07 20:11, "Daniel P. Berrange" <berrange@redhat.com> wrote:

>> We have received bug reports from ISVs of this problem in both the PV
>> and FV cases.  I traced code paths and discovered that FV attempts to
>> allocate the most memory during the initial domain creation; 64-bit PV
>> allocates the next most; 32-bit PV allocates the least.  So failures
>> for PV are less common, but not impossible.
> 
> Indeed - in testing on 64-bit PV over the weekend I saw perhaps 1 in 5
> PV creation attempts fail with the not enough memory bug. So it does look
> like the patch needs to be de-indented to fix PV too.

It's weird that this is only cropping up now. Still, the workaround is
pretty harmless.

 -- Keir