PV domain save/restore break

PV domain save/restore break

[Zhai, Edwin]

I encounter PV domain restore failure on r14770. Are you guys aware of this?


========================================================================
[2007-04-10 09:57:24 4664] DEBUG (balloon:113) Balloon: 754076 KiB free; need 
65536; done.
[2007-04-10 09:57:24 4664] DEBUG (XendCheckpoint:220) [xc_restore]: /usr/lib/xen/bin/xc_restore 24 4 1 2 0 0 0
[2007-04-10 09:57:24 4664] INFO (XendCheckpoint:336) xc_domain_restore start: p2m_size = 4800
[2007-04-10 09:57:24 4664] INFO (XendCheckpoint:336) Reloading memory pages:   0%
[2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336) Failed allocation for dom 4: 125 extents of order 0
[2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336) ERROR Internal error: Failed to allocate memory for batch.!
[2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336)
[2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336) PT L1 race on pfn=000003ec mfn=0002e475
[2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336) Failed allocation for dom 4: 49 extents of order 0
[2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336) ERROR Internal error: Failed to allocate memory for batch.!
[2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336)
[2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336) Restore exit with rc=1
[2007-04-10 09:57:25 4664] DEBUG (XendDomainInfo:1652) XendDomainInfo.destroy: domid=4
[2007-04-10 09:57:25 4664] DEBUG (XendDomainInfo:1669) XendDomainInfo.destroyDomain(4)
[2007-04-10 09:57:25 4664] ERROR (XendDomainInfo:1681) XendDomainInfo.destroy: xc.domain_destroy failed.
Traceback (most recent call last):
  File "/home/gzhai/srcs/hg/tip/dist/install/usr/lib/python/xen/xend/XendDomainInfo.py", line 1675, in destroyDomain
Error: (3, 'No such process')
[2007-04-10 09:57:25 4664] ERROR (XendDomain:1086) Restore failed
Traceback (most recent call last):
  File "/home/gzhai/srcs/hg/tip/dist/install/usr/lib/python/xen/xend/XendDomain.py", line 1081, in domain_restore_fd
  File "/home/gzhai/srcs/hg/tip/dist/install/usr/lib/python/xen/xend/XendCheckpoint.py", line 224, in restore
  File "/home/gzhai/srcs/hg/tip/dist/install/usr/lib/python/xen/xend/XendCheckpoint.py", line 324, in forkHelper
XendError: /usr/lib/xen/bin/xc_restore 24 4 1 2 0 0 0 failed


-- 
best rgds,
edwin

Re: PV domain save/restore break

[Keir Fraser]

Our regression tests ran fine on r14773, and our tests include lots of PV
save/restores.

 -- Keir

On 10/4/07 03:05, "Zhai, Edwin" <edwin.zhai@intel.com> wrote:

> I encounter PV domain restore failure on r14770. Are you guys aware of this?
> 
> 
> ========================================================================
> [2007-04-10 09:57:24 4664] DEBUG (balloon:113) Balloon: 754076 KiB free; need
> 65536; done.
> [2007-04-10 09:57:24 4664] DEBUG (XendCheckpoint:220) [xc_restore]:
> /usr/lib/xen/bin/xc_restore 24 4 1 2 0 0 0
> [2007-04-10 09:57:24 4664] INFO (XendCheckpoint:336) xc_domain_restore start:
> p2m_size = 4800
> [2007-04-10 09:57:24 4664] INFO (XendCheckpoint:336) Reloading memory pages:
> 0%
> [2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336) Failed allocation for dom
> 4: 125 extents of order 0
> [2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336) ERROR Internal error:
> Failed to allocate memory for batch.!
> [2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336)
> [2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336) PT L1 race on
> pfn=000003ec mfn=0002e475
> [2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336) Failed allocation for dom
> 4: 49 extents of order 0
> [2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336) ERROR Internal error:
> Failed to allocate memory for batch.!
> [2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336)
> [2007-04-10 09:57:25 4664] INFO (XendCheckpoint:336) Restore exit with rc=1
> [2007-04-10 09:57:25 4664] DEBUG (XendDomainInfo:1652) XendDomainInfo.destroy:
> domid=4
> [2007-04-10 09:57:25 4664] DEBUG (XendDomainInfo:1669)
> XendDomainInfo.destroyDomain(4)
> [2007-04-10 09:57:25 4664] ERROR (XendDomainInfo:1681) XendDomainInfo.destroy:
> xc.domain_destroy failed.
> Traceback (most recent call last):
>   File 
> "/home/gzhai/srcs/hg/tip/dist/install/usr/lib/python/xen/xend/XendDomainInfo.p
> y", line 1675, in destroyDomain
> Error: (3, 'No such process')
> [2007-04-10 09:57:25 4664] ERROR (XendDomain:1086) Restore failed
> Traceback (most recent call last):
>   File 
> "/home/gzhai/srcs/hg/tip/dist/install/usr/lib/python/xen/xend/XendDomain.py",
> line 1081, in domain_restore_fd
>   File 
> "/home/gzhai/srcs/hg/tip/dist/install/usr/lib/python/xen/xend/XendCheckpoint.p
> y", line 224, in restore
>   File 
> "/home/gzhai/srcs/hg/tip/dist/install/usr/lib/python/xen/xend/XendCheckpoint.p
> y", line 324, in forkHelper
> XendError: /usr/lib/xen/bin/xc_restore 24 4 1 2 0 0 0 failed
> 

Re: Re: PV domain save/restore break

[Stephan Creutz]

On Tue, 10 Apr 2007 10:13:14 +0100
Keir Fraser <keir@xensource.com> wrote:

> Our regression tests ran fine on r14773, and our tests include lots of
> PV save/restores.

I have the same problem running r14774. The output of "xm log" is very
similar to the one from Edwin. Additionally "xm dmesg" gives me the
following messages:

(XEN) page_alloc.c:770:d0 Over-allocation for domain 3: 16385 > 16384
(XEN) memory.c:124:d0 Could not allocate order=0 extent: id=3 memflags=0
(439 of 694)
(XEN) page_alloc.c:770:d0 Over-allocation for domain 3: 16385 > 16384
(XEN) memory.c:124:d0 Could not allocate order=0 extent: id=3 memflags=0
(0 of 44)

"xm log":

[2007-04-10 11:19:07 3648] DEBUG (__init__:1072) restore:shadow=0x0, _static_max=0x4000000, _static_min=0x0, 
[2007-04-10 11:19:07 3648] DEBUG (__init__:1072) Balloon: 66172 KiB free; need 65536; done.
[2007-04-10 11:19:07 3648] DEBUG (__init__:1072) [xc_restore]: /usr/lib/xen/bin/xc_restore 28 3 1 2 0 0 0
[2007-04-10 11:19:07 3648] INFO (__init__:1072) xc_domain_restore start: p2m_size = 4800
[2007-04-10 11:19:07 3648] INFO (__init__:1072) Reloading memory pages:   0%
[2007-04-10 11:19:07 3648] INFO (__init__:1072) Failed allocation for dom 3: 694 extents of order 0
[2007-04-10 11:19:07 3648] INFO (__init__:1072) ERROR Internal error: Failed to allocate memory for batch.!
[2007-04-10 11:19:07 3648] INFO (__init__:1072) 
[2007-04-10 11:19:07 3648] INFO (__init__:1072) PT L1 race on pfn=0000077a mfn=00001403
[2007-04-10 11:19:07 3648] INFO (__init__:1072) Failed allocation for dom 3: 44 extents of order 0
[2007-04-10 11:19:07 3648] INFO (__init__:1072) ERROR Internal error: Failed to allocate memory for batch.!
[2007-04-10 11:19:07 3648] INFO (__init__:1072) 
[2007-04-10 11:19:08 3648] INFO (__init__:1072) Restore exit with rc=1
[2007-04-10 11:19:08 3648] DEBUG (__init__:1072) XendDomainInfo.destroy: domid=3
[2007-04-10 11:19:08 3648] DEBUG (__init__:1072) XendDomainInfo.destroyDomain(3)
[2007-04-10 11:19:08 3648] ERROR (__init__:1072) XendDomainInfo.destroy: xc.domain_destroy failed.
Traceback (most recent call last):
  File "//usr/lib/python/xen/xend/XendDomainInfo.py", line 1675, in destroyDomain
    xc.domain_destroy(self.domid)
Error: (3, \047No such process\047)
[2007-04-10 11:19:08 3648] ERROR (__init__:1072) Restore failed
Traceback (most recent call last):
  File "//usr/lib/python/xen/xend/XendDomain.py", line 1081, in domain_restore_fd
    return XendCheckpoint.restore(self, fd, paused=paused)
  File "//usr/lib/python/xen/xend/XendCheckpoint.py", line 224, in restore
    forkHelper(cmd, fd, handler.handler, True)
  File "//usr/lib/python/xen/xend/XendCheckpoint.py", line 324, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib/xen/bin/xc_restore 28 3 1 2 0 0 0 failed

Stephan

Re: Re: PV domain save/restore break

[Keir Fraser]

Is this a 64MB domain? Have you used the balloon driver in the domain (or xm
mem-set) to change the domain's memory usage after it has been created?

 -- Keir

On 10/4/07 10:40, "Stephan Creutz" <stephan.creutz@inf.tu-dresden.de> wrote:

> I have the same problem running r14774. The output of "xm log" is very
> similar to the one from Edwin. Additionally "xm dmesg" gives me the
> following messages:
> 
> (XEN) page_alloc.c:770:d0 Over-allocation for domain 3: 16385 > 16384
> (XEN) memory.c:124:d0 Could not allocate order=0 extent: id=3 memflags=0
> (439 of 694)
> (XEN) page_alloc.c:770:d0 Over-allocation for domain 3: 16385 > 16384
> (XEN) memory.c:124:d0 Could not allocate order=0 extent: id=3 memflags=0
> (0 of 44)

Re: Re: PV domain save/restore break

[Stephan Creutz]

On Tue, 10 Apr 2007 10:48:18 +0100
Keir Fraser <keir@xensource.com> wrote:

> Is this a 64MB domain? Have you used the balloon driver in the domain
> (or xm mem-set) to change the domain's memory usage after it has been
> created?

Yes, this is a 64MB domain and I haven't changed the size of memory
after creating it.

Stephan

Re: Re: PV domain save/restore break

[Keir Fraser]

On 10/4/07 10:58, "Stephan Creutz" <stephan.creutz@inf.tu-dresden.de> wrote:

> On Tue, 10 Apr 2007 10:48:18 +0100
> Keir Fraser <keir@xensource.com> wrote:
> 
>> Is this a 64MB domain? Have you used the balloon driver in the domain
>> (or xm mem-set) to change the domain's memory usage after it has been
>> created?
> 
> Yes, this is a 64MB domain and I haven't changed the size of memory
> after creating it.

I can repro this on unstable tip. I'll look into it.

 -- Keir

Re: Re: PV domain save/restore break

[Keir Fraser]

On 10/4/07 11:07, "Keir Fraser" <keir@xensource.com> wrote:

>>> Is this a 64MB domain? Have you used the balloon driver in the domain
>>> (or xm mem-set) to change the domain's memory usage after it has been
>>> created?
>> 
>> Yes, this is a 64MB domain and I haven't changed the size of memory
>> after creating it.
> 
> I can repro this on unstable tip. I'll look into it.

Fixed by changeset 14781:099593da38cb.

 -- Keir

Re: Re: PV domain save/restore break

[Stephan Creutz]

On Tue, 10 Apr 2007 13:46:48 +0100
Keir Fraser <keir@xensource.com> wrote:
> Fixed by changeset 14781:099593da38cb.

This morning your patch appeared in xen-unstable and it works for me.
Thank you very much Keir.

Stephan