domU reboot failure

domU reboot failure

[Alex Williamson]

   FYI, latest unstable tip (14462) fails to reboot pv domains (issuing
reboot from within the domain).  Xend.log shows the following:

[2007-03-21 14:54:26 3097] DEBUG (__init__:1072) features       = 
[2007-03-21 14:54:26 3097] ERROR (__init__:1072) XendDomainInfo.initDomain: exception occurred
Traceback (most recent call last):
  File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 1499, in _initDomain
    channel_details = self.image.createImage()
  File "//usr/lib64/python/xen/xend/image.py", line 112, in createImage
    return self.createDomain()
  File "//usr/lib64/python/xen/xend/image.py", line 131, in createDomain
    result = self.buildDomain()
  File "//usr/lib64/python/xen/xend/image.py", line 611, in buildDomain
    return LinuxImageHandler.buildDomain(self)
  File "//usr/lib64/python/xen/xend/image.py", line 215, in buildDomain
    features       = self.vm.getFeatures())
Error: (11, 'Resource temporarily unavailable')
[2007-03-21 14:54:26 3097] ERROR (__init__:1072) VM start failed
...

Seeing this both on ia64 and x86_64 (didn't test x86_32).  Domain was
started with 'xm create'.  The breakage seems to have occurred within
the last ~50 csets.  Thanks,

	Alex

-- 
Alex Williamson                             HP Open Source & Linux Org.

Re: domU reboot failure

[Al Stone]

On Wed, 2007-03-21 at 15:08 -0600, Alex Williamson wrote:
>    FYI, latest unstable tip (14462) fails to reboot pv domains (issuing
> reboot from within the domain).

I can recreate this with 'xm reboot' also -- and get the same
exact errors.

>   Xend.log shows the following:
> 
> [2007-03-21 14:54:26 3097] DEBUG (__init__:1072) features       = 
> [2007-03-21 14:54:26 3097] ERROR (__init__:1072) XendDomainInfo.initDomain: exception occurred
> Traceback (most recent call last):
>   File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 1499, in _initDomain
>     channel_details = self.image.createImage()
>   File "//usr/lib64/python/xen/xend/image.py", line 112, in createImage
>     return self.createDomain()
>   File "//usr/lib64/python/xen/xend/image.py", line 131, in createDomain
>     result = self.buildDomain()
>   File "//usr/lib64/python/xen/xend/image.py", line 611, in buildDomain
>     return LinuxImageHandler.buildDomain(self)
>   File "//usr/lib64/python/xen/xend/image.py", line 215, in buildDomain
>     features       = self.vm.getFeatures())
> Error: (11, 'Resource temporarily unavailable')
> [2007-03-21 14:54:26 3097] ERROR (__init__:1072) VM start failed
> ...
> 
> Seeing this both on ia64 and x86_64 (didn't test x86_32).  Domain was
> started with 'xm create'.  The breakage seems to have occurred within
> the last ~50 csets.  Thanks,

Digging through the logs some more, it looks like my domU
is initially started with 800MB RAM as it should be:

...
[2007-03-21 17:42:00 4248] INFO (__init__:1072) buildDomain os=linux dom=9 vcpus=1
[2007-03-21 17:42:00 4248] DEBUG (__init__:1072) domid          = 9
[2007-03-21 17:42:00 4248] DEBUG (__init__:1072) memsize        = 800
[2007-03-21 17:42:00 4248] DEBUG (__init__:1072) image          = /boot/vmlinuz-2.6.18-xen
...

However, after the domU has shut down, it gets
restarted by XenDomainInfo.create_from_dict, and for
some reason I haven't figured out yet, memsize is
now set to zero:

...
[2007-03-21 17:42:50 4248] INFO (__init__:1072) buildDomain os=linux dom=10 vcpus=1
[2007-03-21 17:42:50 4248] DEBUG (__init__:1072) domid          = 10
[2007-03-21 17:42:50 4248] DEBUG (__init__:1072) memsize        = 0
[2007-03-21 17:42:50 4248] DEBUG (__init__:1072) image          = /boot/vmlinuz-2.6.18-xen
...

This, of course, leads the domain builder to complain
for obvious reasons:

...
xc_dom_mem_init: mem 0 MB, pages 0x0 pages, 16k each
xc_dom_mem_init: 0x0 pages
xc_dom_boot_mem_init: called
arch_setup_meminit: doing nothing
xc_dom_build_image: called
xc_dom_alloc_segment:   kernel       : 0x4000000 -> 0x519c000  (pfn 0x1000 + 0x467 pages)
xc_dom_pfn_to_ptr: pfn out of range (0x1000 > 0x0)
xc_dom_release: called
...

Is this a red herring, or did the reboot code somehow forget
to get the right amount of memory?

-- 
Ciao,
al
----------------------------------------------------------------------
Al Stone                                      Alter Ego:
Open Source and Linux R&D                     Debian Developer
Hewlett-Packard Company                       http://www.debian.org
E-mail: ahs3@fc.hp.com                        ahs3@debian.org
----------------------------------------------------------------------

Re: domU reboot failure

[Ewan Mellor]

On Thu, Mar 22, 2007 at 03:59:20PM +0000, Al Stone wrote:

> On Wed, 2007-03-21 at 15:08 -0600, Alex Williamson wrote:
> >    FYI, latest unstable tip (14462) fails to reboot pv domains (issuing
> > reboot from within the domain).
> 
> I can recreate this with 'xm reboot' also -- and get the same
> exact errors.
> 
> >   Xend.log shows the following:
> > 
> > [2007-03-21 14:54:26 3097] DEBUG (__init__:1072) features       = 
> > [2007-03-21 14:54:26 3097] ERROR (__init__:1072) XendDomainInfo.initDomain: exception occurred
> > Traceback (most recent call last):
> >   File "//usr/lib64/python/xen/xend/XendDomainInfo.py", line 1499, in _initDomain
> >     channel_details = self.image.createImage()
> >   File "//usr/lib64/python/xen/xend/image.py", line 112, in createImage
> >     return self.createDomain()
> >   File "//usr/lib64/python/xen/xend/image.py", line 131, in createDomain
> >     result = self.buildDomain()
> >   File "//usr/lib64/python/xen/xend/image.py", line 611, in buildDomain
> >     return LinuxImageHandler.buildDomain(self)
> >   File "//usr/lib64/python/xen/xend/image.py", line 215, in buildDomain
> >     features       = self.vm.getFeatures())
> > Error: (11, 'Resource temporarily unavailable')
> > [2007-03-21 14:54:26 3097] ERROR (__init__:1072) VM start failed
> > ...
> > 
> > Seeing this both on ia64 and x86_64 (didn't test x86_32).  Domain was
> > started with 'xm create'.  The breakage seems to have occurred within
> > the last ~50 csets.  Thanks,
> 
> Digging through the logs some more, it looks like my domU
> is initially started with 800MB RAM as it should be:
> 
> ...
> [2007-03-21 17:42:00 4248] INFO (__init__:1072) buildDomain os=linux dom=9 vcpus=1
> [2007-03-21 17:42:00 4248] DEBUG (__init__:1072) domid          = 9
> [2007-03-21 17:42:00 4248] DEBUG (__init__:1072) memsize        = 800
> [2007-03-21 17:42:00 4248] DEBUG (__init__:1072) image          = /boot/vmlinuz-2.6.18-xen
> ...
> 
> However, after the domU has shut down, it gets
> restarted by XenDomainInfo.create_from_dict, and for
> some reason I haven't figured out yet, memsize is
> now set to zero:
> 
> ...
> [2007-03-21 17:42:50 4248] INFO (__init__:1072) buildDomain os=linux dom=10 vcpus=1
> [2007-03-21 17:42:50 4248] DEBUG (__init__:1072) domid          = 10
> [2007-03-21 17:42:50 4248] DEBUG (__init__:1072) memsize        = 0
> [2007-03-21 17:42:50 4248] DEBUG (__init__:1072) image          = /boot/vmlinuz-2.6.18-xen
> ...
> 
> This, of course, leads the domain builder to complain
> for obvious reasons:
> 
> ...
> xc_dom_mem_init: mem 0 MB, pages 0x0 pages, 16k each
> xc_dom_mem_init: 0x0 pages
> xc_dom_boot_mem_init: called
> arch_setup_meminit: doing nothing
> xc_dom_build_image: called
> xc_dom_alloc_segment:   kernel       : 0x4000000 -> 0x519c000  (pfn 0x1000 + 0x467 pages)
> xc_dom_pfn_to_ptr: pfn out of range (0x1000 > 0x0)
> xc_dom_release: called
> ...
> 
> Is this a red herring, or did the reboot code somehow forget
> to get the right amount of memory?

This is a real bug.  I'll fix it.

Ewan.