Error restoring DomU when using GPLPV

Error restoring DomU when using GPLPV

[James Harper]

A user (Joshua) is reporting that 'xm restore' isn't working when GPLPV
is involved. I've checked the logs generated by GPLPV and there are no
problems on the save side of things that I can see. Is there anything
extra that the suspend or restore needs to do since 3.4.x? 

Joshua has captured the following:

On the dom0 I initiated a "xm save" of the VM.  No problems here, but
when I initiate an "xm restore", I receive the following error:

Error: /usr/lib64/xen/bin/xc_restore 56 103 2 3 1 1 1 failed

And in /var/log/xen/xend.log, I see (pertaining to this event):

[2009-08-02 15:12:44 4839] INFO (image:745) Need to create platform
device.[domid:103]
[2009-08-02 15:12:44 4839] DEBUG (XendCheckpoint:261)
restore:shadow=0x9, _static_max=0x40000000, _static_min=0x0,
[2009-08-02 15:12:44 4839] DEBUG (balloon:166) Balloon: 31589116 KiB
free; need 1061888; done.
[2009-08-02 15:12:44 4839] DEBUG (XendCheckpoint:278) [xc_restore]:
/usr/lib64/xen/bin/xc_restore 56 103 2 3 1 1 1
[2009-08-02 15:12:44 4839] INFO (XendCheckpoint:417) xc_domain_restore
start: p2m_size = 100000
[2009-08-02 15:12:44 4839] INFO (XendCheckpoint:417) Reloading memory
pages:   0%
[2009-08-02 15:12:52 4839] INFO (XendCheckpoint:417) Failed allocation
for dom 103: 1024 extents of order 0
[2009-08-02 15:12:52 4839] INFO (XendCheckpoint:417) ERROR Internal
error: Failed to allocate memory for batch.!
[2009-08-02 15:12:52 4839] INFO (XendCheckpoint:417)
[2009-08-02 15:12:52 4839] INFO (XendCheckpoint:417) Restore exit with
rc=1
[2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:2724)
XendDomainInfo.destroy: domid=103
[2009-08-02 15:12:52 4839] ERROR (XendDomainInfo:2738)
XendDomainInfo.destroy: domain destruction failed.
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
line 2731, in destroy
    xc.domain_pause(self.domid)
Error: (3, 'No such process')
[2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:2204) No device model
[2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:2206) Releasing devices
[2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:2219) Removing vbd/768
[2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:1134)
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/768
[2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:2219) Removing vfb/0
[2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:1134)
XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0
[2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:2219) Removing
console/0
[2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:1134)
XendDomainInfo.destroyDevice: deviceClass = console, device = console/0
[2009-08-02 15:12:52 4839] ERROR (XendDomain:1149) Restore failed
Traceback (most recent call last):
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py", line
1147, in domain_restore_fd
    return XendCheckpoint.restore(self, fd, paused=paused,
relocating=relocating)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
line 282, in restore
    forkHelper(cmd, fd, handler.handler, True)
  File "/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
line 405, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib64/xen/bin/xc_restore 56 103 2 3 1 1 1 failed

RE: Error restoring DomU when using GPLPV

[James Harper]

It seems that somewhere along the line Xen started using an event
channel to trigger a suspend, as opposed to the 'shutdown' xenstore
value. Is there anything else there I need to know about?

Thanks

James

> -----Original Message-----
> From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-
> bounces@lists.xensource.com] On Behalf Of James Harper
> Sent: Tuesday, 4 August 2009 11:23
> To: xen-devel@lists.xensource.com
> Cc: Joshua West
> Subject: [Xen-devel] Error restoring DomU when using GPLPV
> 
> A user (Joshua) is reporting that 'xm restore' isn't working when
GPLPV
> is involved. I've checked the logs generated by GPLPV and there are no
> problems on the save side of things that I can see. Is there anything
> extra that the suspend or restore needs to do since 3.4.x?
> 
> Joshua has captured the following:
> 
> On the dom0 I initiated a "xm save" of the VM.  No problems here, but
> when I initiate an "xm restore", I receive the following error:
> 
> Error: /usr/lib64/xen/bin/xc_restore 56 103 2 3 1 1 1 failed
> 
> And in /var/log/xen/xend.log, I see (pertaining to this event):
> 
> [2009-08-02 15:12:44 4839] INFO (image:745) Need to create platform
> device.[domid:103]
> [2009-08-02 15:12:44 4839] DEBUG (XendCheckpoint:261)
> restore:shadow=0x9, _static_max=0x40000000, _static_min=0x0,
> [2009-08-02 15:12:44 4839] DEBUG (balloon:166) Balloon: 31589116 KiB
> free; need 1061888; done.
> [2009-08-02 15:12:44 4839] DEBUG (XendCheckpoint:278) [xc_restore]:
> /usr/lib64/xen/bin/xc_restore 56 103 2 3 1 1 1
> [2009-08-02 15:12:44 4839] INFO (XendCheckpoint:417) xc_domain_restore
> start: p2m_size = 100000
> [2009-08-02 15:12:44 4839] INFO (XendCheckpoint:417) Reloading memory
> pages:   0%
> [2009-08-02 15:12:52 4839] INFO (XendCheckpoint:417) Failed allocation
> for dom 103: 1024 extents of order 0
> [2009-08-02 15:12:52 4839] INFO (XendCheckpoint:417) ERROR Internal
> error: Failed to allocate memory for batch.!
> [2009-08-02 15:12:52 4839] INFO (XendCheckpoint:417)
> [2009-08-02 15:12:52 4839] INFO (XendCheckpoint:417) Restore exit with
> rc=1
> [2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:2724)
> XendDomainInfo.destroy: domid=103
> [2009-08-02 15:12:52 4839] ERROR (XendDomainInfo:2738)
> XendDomainInfo.destroy: domain destruction failed.
> Traceback (most recent call last):
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendDomainInfo.py",
> line 2731, in destroy
>     xc.domain_pause(self.domid)
> Error: (3, 'No such process')
> [2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:2204) No device model
> [2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:2206) Releasing
devices
> [2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:2219) Removing
vbd/768
> [2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:1134)
> XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/768
> [2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:2219) Removing vfb/0
> [2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:1134)
> XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0
> [2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:2219) Removing
> console/0
> [2009-08-02 15:12:52 4839] DEBUG (XendDomainInfo:1134)
> XendDomainInfo.destroyDevice: deviceClass = console, device =
console/0
> [2009-08-02 15:12:52 4839] ERROR (XendDomain:1149) Restore failed
> Traceback (most recent call last):
>   File "/usr/lib64/python2.4/site-packages/xen/xend/XendDomain.py",
line
> 1147, in domain_restore_fd
>     return XendCheckpoint.restore(self, fd, paused=paused,
> relocating=relocating)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
> line 282, in restore
>     forkHelper(cmd, fd, handler.handler, True)
>   File
"/usr/lib64/python2.4/site-packages/xen/xend/XendCheckpoint.py",
> line 405, in forkHelper
>     raise XendError("%s failed" % string.join(cmd))
> XendError: /usr/lib64/xen/bin/xc_restore 56 103 2 3 1 1 1 failed
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

RE: Error restoring DomU when using GPLPV

[James Harper]

> 
> It seems that somewhere along the line Xen started using an event
> channel to trigger a suspend, as opposed to the 'shutdown' xenstore
> value. Is there anything else there I need to know about?
> 

Actually that seems to be unrelated the problem (I found this out after
adding suspend evtchn support to gplpv...)

The actual error is that the call to xc_memory_op passes 33 as
nr_extents, but the return value is 32, which is an error condition. Is
it not counting an already allocated page in the PVonHVM case or
something?

XendCheckpoint:417 calls xc_domain_restore which calls
xc_domain_memory_increase_reservation.

Does XENMEM_increase_reservation mean "increase reservation _by_ X" or
"increase reservation _to_ X"?

James

RE: Error restoring DomU when using GPLPV

[James Harper]

> 
> >
> > It seems that somewhere along the line Xen started using an event
> > channel to trigger a suspend, as opposed to the 'shutdown' xenstore
> > value. Is there anything else there I need to know about?
> >
> 
> Actually that seems to be unrelated the problem (I found this out
after adding
> suspend evtchn support to gplpv...)
> 
> The actual error is that the call to xc_memory_op passes 33 as
nr_extents, but
> the return value is 32, which is an error condition. Is it not
counting an
> already allocated page in the PVonHVM case or something?
> 
> XendCheckpoint:417 calls xc_domain_restore which calls
> xc_domain_memory_increase_reservation.
> 
> Does XENMEM_increase_reservation mean "increase reservation _by_ X" or
> "increase reservation _to_ X"?
> 

Actually I was looking at the wrong thing. The error is actually in
xc_domain_memory_populate_physmap

James

RE: Error restoring DomU when using GPLPV

[James Harper]

When the DomU is running, 'xm debug q' looks like:

(XEN) General information for domain 23:
(XEN)     refcnt=3 dying=0 nr_pages=197611 xenheap_pages=33
dirty_cpus={1} max_pages=197632

During restore, it looks like this:
(XEN) General information for domain 22:
(XEN)     refcnt=3 dying=0 nr_pages=196576 xenheap_pages=5 dirty_cpus={}
max_pages=197632

(last capture before the domain ran out of pages)

I added some debugging to libxc and it allocates bunches of 1024 pages
over and over quite successfully but then fails at 33. Presumably
something is counting pages incorrectly somewhere?

James

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 04/08/2009 08:58, "James Harper" <james.harper@bendigoit.com.au> wrote:

> When the DomU is running, 'xm debug q' looks like:
> 
> (XEN) General information for domain 23:
> (XEN)     refcnt=3 dying=0 nr_pages=197611 xenheap_pages=33
> dirty_cpus={1} max_pages=197632
> 
> During restore, it looks like this:
> (XEN) General information for domain 22:
> (XEN)     refcnt=3 dying=0 nr_pages=196576 xenheap_pages=5 dirty_cpus={}
> max_pages=197632

Is the host simply out of memory? If dom22 above has 196576 pages and
max_pages=197632 then an allocation of 33 order-0 extents should not fail
due to over-commitment to the guest. The only reason for such a failure is
inadequate memory available in the host free pools. Perhaps xend
auto-ballooning is involved? I'd turn it off if so, as it blows. It could
have freed up one-page-too-few or somesuch.

 -- Keir

RE: Error restoring DomU when using GPLPV

[James Harper]

> 
> On 04/08/2009 08:58, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> > When the DomU is running, 'xm debug q' looks like:
> >
> > (XEN) General information for domain 23:
> > (XEN)     refcnt=3 dying=0 nr_pages=197611 xenheap_pages=33
> > dirty_cpus={1} max_pages=197632
> >
> > During restore, it looks like this:
> > (XEN) General information for domain 22:
> > (XEN)     refcnt=3 dying=0 nr_pages=196576 xenheap_pages=5
dirty_cpus={}
> > max_pages=197632
> 
> Is the host simply out of memory?

No. 5G physical memory free and there is only 768MB assigned to the
DomU. I can start the guest again, I just can't restore it.

> If dom22 above has 196576 pages and
> max_pages=197632 then an allocation of 33 order-0 extents should not
fail
> due to over-commitment to the guest.

196576 is just where it happened to be when I took the last 'xm debug
q', before 'xm restore' failed and deleted it. The allocation of '33'
returns '32' so it does appear to be an off-by-one error.

> The only reason for such a failure is
> inadequate memory available in the host free pools. Perhaps xend
> auto-ballooning is involved? I'd turn it off if so, as it blows. It
could
> have freed up one-page-too-few or somesuch.

I assume that what happens is that the memory continues to grow until it
hits max_pages, for some reason.  Is there a way to tell 'xm restore'
not to delete the domain when the restore fails so I can see if nr_pages
really does equal max_pages at the time that it dies?

The curious thing is that this only happens when GPLPV is running. A PV
domU or a pure HVM DomU doesn't have this problem (presumably that would
have been noticed during regression testing). It would be interesting to
try a PVonHVM Linux DomU and see how that goes... hopefully someone who
having the problem with GPLPV also has PVonHVM domains they could test.

James

RE: Error restoring DomU when using GPLPV

[James Harper]

> 
> Is the host simply out of memory? If dom22 above has 196576 pages and
> max_pages=197632 then an allocation of 33 order-0 extents should not
fail
> due to over-commitment to the guest.

I added some more debugging...

batch 1024 [1]
Allocating 1024 mfns [2]
197600 allocated [3]
batch 1024
Allocating 33 mfns
Failed allocation for dom 24: 33 extents of order 0 (err = 32) [4]

[1] is just after 'j' is read in xc_domain_restore
[2] is just before the call to populate_physmap
[3] is just after the call to populate_physmap
[4] is the error message in the memory_op function in libxc modified to
give the value of err

According to the a total of 197632 are being allocated and the last page
cannot be (could be more pages required the next time around the loop
too...)

James

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 04/08/2009 10:01, "James Harper" <james.harper@bendigoit.com.au> wrote:

> I assume that what happens is that the memory continues to grow until it
> hits max_pages, for some reason.  Is there a way to tell 'xm restore'
> not to delete the domain when the restore fails so I can see if nr_pages
> really does equal max_pages at the time that it dies?
> 
> The curious thing is that this only happens when GPLPV is running. A PV
> domU or a pure HVM DomU doesn't have this problem (presumably that would
> have been noticed during regression testing). It would be interesting to
> try a PVonHVM Linux DomU and see how that goes... hopefully someone who
> having the problem with GPLPV also has PVonHVM domains they could test.

Okay, also this is a normal save/restore (no live migration of pages)?

Could the grant-table/shinfo Xenheap pages be confusing matters I wonder.
The save process may save those pages out - since dom0 can map them it will
also save them - and then they get mistakenly restored as domheap pages at
the far end. All would work out okay in the end when you remap those special
pages during GPLPV restore as the domheap pages would get implciitly freed.
But maybe there is not allocation headroom for the guest in the meantime so
the restore fails.

Just a theory. Maybe you could try unmapping grant/shinfo pages in the
suspend callback? This may not help for live migration though, where pages
get transmitted before the callback. It may be necessary to allow dom0 to
specify 'map me a page but not if it's special' and plumb that up to
xc_domain_save. It'd be good to have the theory proved first.

 Cheers,
 Keir

RE: Error restoring DomU when using GPLPV

[James Harper]

> 
> Just a theory. Maybe you could try unmapping grant/shinfo pages in the
> suspend callback? This may not help for live migration though, where
pages
> get transmitted before the callback. It may be necessary to allow dom0
to
> specify 'map me a page but not if it's special' and plumb that up to
> xc_domain_save. It'd be good to have the theory proved first.
> 

Can do. What is the opposite of 'XENMEM_add_to_physmap' which I assume
I'll need to unmap the grant pages? Is it XENMEM_decrease_reservation?

James

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 04/08/2009 10:34, "James Harper" <james.harper@bendigoit.com.au> wrote:

>> Just a theory. Maybe you could try unmapping grant/shinfo pages in the
>> suspend callback? This may not help for live migration though, where
> pages
>> get transmitted before the callback. It may be necessary to allow dom0
> to
>> specify 'map me a page but not if it's special' and plumb that up to
>> xc_domain_save. It'd be good to have the theory proved first.
>> 
> 
> Can do. What is the opposite of 'XENMEM_add_to_physmap' which I assume
> I'll need to unmap the grant pages? Is it XENMEM_decrease_reservation?

Oh yes, there is no direct opposite of add_to_physmap... But I think
decrease_reservation will work okay in this case, fortunately.

 -- Keir

RE: Error restoring DomU when using GPLPV

[James Harper]

> 
> Could the grant-table/shinfo Xenheap pages be confusing matters I
wonder.
> The save process may save those pages out - since dom0 can map them it
will
> also save them - and then they get mistakenly restored as domheap
pages at
> the far end. All would work out okay in the end when you remap those
special
> pages during GPLPV restore as the domheap pages would get implciitly
freed.
> But maybe there is not allocation headroom for the guest in the
meantime so
> the restore fails.
> 
> Just a theory. Maybe you could try unmapping grant/shinfo pages in the
> suspend callback? This may not help for live migration though, where
pages
> get transmitted before the callback. It may be necessary to allow dom0
to
> specify 'map me a page but not if it's special' and plumb that up to
> xc_domain_save. It'd be good to have the theory proved first.
> 

I took the easier path and told my grant code to map 2 less pages, and
sure enough it tries to allocate 31 pages (which succeeds) but then
tries to allocate 7 (which fails). So the 31 (was 33) must contain the
grant table pages etc. I'll attempt to add the unmap code you requested
and see if it makes a difference...

So why doesn't PV have this problem? Does it not send the pages first?
And do you think that a Linux HVM domain with PV drivers would suffer
the same fate?

Thanks

James

RE: Error restoring DomU when using GPLPV

[James Harper]

> 
> On 04/08/2009 10:34, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> >> Just a theory. Maybe you could try unmapping grant/shinfo pages in
the
> >> suspend callback? This may not help for live migration though,
where
> > pages
> >> get transmitted before the callback. It may be necessary to allow
dom0
> > to
> >> specify 'map me a page but not if it's special' and plumb that up
to
> >> xc_domain_save. It'd be good to have the theory proved first.
> >>
> >
> > Can do. What is the opposite of 'XENMEM_add_to_physmap' which I
assume
> > I'll need to unmap the grant pages? Is it
XENMEM_decrease_reservation?
> 
> Oh yes, there is no direct opposite of add_to_physmap... But I think
> decrease_reservation will work okay in this case, fortunately.
> 

Given that I'm not going to use the grant table subsequent to unmapping
them I'll probably get away with it, but does
XENMEM_decrease_reservation actually tell xen that the pages are no
longer actually part of the grant table?

James

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 04/08/2009 11:40, "James Harper" <james.harper@bendigoit.com.au> wrote:

>> Oh yes, there is no direct opposite of add_to_physmap... But I think
>> decrease_reservation will work okay in this case, fortunately.
>> 
> 
> Given that I'm not going to use the grant table subsequent to unmapping
> them I'll probably get away with it, but does
> XENMEM_decrease_reservation actually tell xen that the pages are no
> longer actually part of the grant table?

No, for a xenheap page the page won't actually get freed. Xen keeps a
reference to them until the domain is finally destroyed.

Regarding the Linux PV-on-HVM drivers - they may have the same issue. Full
PV guests do not as they have a gnttab_suspend() function called during
suspend callback (and for subtle reasons xc_domain_save can detect and not
save Xenheap pages for a full PV guests anyway - because it can see the P2M
table in that case).

Like I said before -- unmapping the gnttab pages I think will not help you
for live migration, but I suppose it is a reasonable thing to do anyway. For
live migration I think xc_domain_save needs t get a bit smarter about
Xenheap pages in HVM guests.

 -- Keir

RE: Error restoring DomU when using GPLPV

[James Harper]

> On 04/08/2009 11:40, "James Harper" <james.harper@bendigoit.com.au>
wrote:
> 
> >> Oh yes, there is no direct opposite of add_to_physmap... But I
think
> >> decrease_reservation will work okay in this case, fortunately.
> >>
> >
> > Given that I'm not going to use the grant table subsequent to
unmapping
> > them I'll probably get away with it, but does
> > XENMEM_decrease_reservation actually tell xen that the pages are no
> > longer actually part of the grant table?
> 
> No, for a xenheap page the page won't actually get freed. Xen keeps a
> reference to them until the domain is finally destroyed.
> 
> Regarding the Linux PV-on-HVM drivers - they may have the same issue.
Full
> PV guests do not as they have a gnttab_suspend() function called
during
> suspend callback (and for subtle reasons xc_domain_save can detect and
not
> save Xenheap pages for a full PV guests anyway - because it can see
the P2M
> table in that case).
> 
> Like I said before -- unmapping the gnttab pages I think will not help
you
> for live migration, but I suppose it is a reasonable thing to do
anyway. For
> live migration I think xc_domain_save needs t get a bit smarter about
> Xenheap pages in HVM guests.
> 

Understood. Do you have any idea about why it worked fine under 3.3.x
but not 3.4.x?

Thanks

James

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 04/08/2009 12:34, "James Harper" <james.harper@bendigoit.com.au> wrote:

>> Like I said before -- unmapping the gnttab pages I think will not help
> you
>> for live migration, but I suppose it is a reasonable thing to do
> anyway. For
>> live migration I think xc_domain_save needs t get a bit smarter about
>> Xenheap pages in HVM guests.
> 
> Understood. Do you have any idea about why it worked fine under 3.3.x
> but not 3.4.x?

The bit of code in 3.3's xc_domain_save.c that is commented "Skip PFNs that
aren't really there" is removed in 3.4. That will be the reason.

 -- Keir

Re: Error restoring DomU when using GPLPV

[Pasi Kärkkäinen]

On Tue, Aug 04, 2009 at 02:12:48PM +0100, Keir Fraser wrote:
> On 04/08/2009 12:34, "James Harper" <james.harper@bendigoit.com.au> wrote:
> 
> >> Like I said before -- unmapping the gnttab pages I think will not help
> > you
> >> for live migration, but I suppose it is a reasonable thing to do
> > anyway. For
> >> live migration I think xc_domain_save needs t get a bit smarter about
> >> Xenheap pages in HVM guests.
> > 
> > Understood. Do you have any idea about why it worked fine under 3.3.x
> > but not 3.4.x?
> 
> The bit of code in 3.3's xc_domain_save.c that is commented "Skip PFNs that
> aren't really there" is removed in 3.4. That will be the reason.
> 

James: Did you figure out how to fix this problem in gplpv drivers? Just
asking because it seems people hit this save/restore/migration problem pretty often..

-- Pasi

RE: Error restoring DomU when using GPLPV

[James Harper]

> > The bit of code in 3.3's xc_domain_save.c that is commented "Skip PFNs that
> > aren't really there" is removed in 3.4. That will be the reason.
> >
> 
> James: Did you figure out how to fix this problem in gplpv drivers? Just
> asking because it seems people hit this save/restore/migration problem pretty
> often..
> 

It's a problem that affects any PVonHVM domain afaict, so I'd rather defer to the person who made the change originally.

James

Re: Error restoring DomU when using GPLPV

[ANNIE LI]

[-- Attachment #1.1: Type: text/plain, Size: 1541 bytes --]

Hi James,
>
> It's a problem that affects any PVonHVM domain afaict, so I'd rather defer to the person who made the change originally.
I did some test. Linux PVM live migration works well on Xen3.4. As you 
said above, PVHVM fails live migration and got error: "Error: 
/usr/lib/xen/bin/xc_save 32 8 0 0 5 failed", but save/restore is OK.
So this problem should be fixed on xen side instead of pv driver side?

Following is the log of linux PVHVM live migration:

[2009-08-19 10:52:08 2832] DEBUG (XendCheckpoint:110) [xc_save]: 
/usr/lib/xen/bin/xc_save 32 8 0 0 5
[2009-08-19 10:52:08 2832] INFO (XendCheckpoint:418) xc_save: failed to 
get the suspend evtchn port
[2009-08-19 10:52:08 2832] INFO (XendCheckpoint:418)
[2009-08-19 10:52:08 2832] INFO (XendCheckpoint:418) Saving memory 
pages: iter 1   0%ERROR Internal error: Error when writing to state file 
(4a) (errno 104)
[2009-08-19 10:52:08 2832] INFO (XendCheckpoint:418) Save exit rc=1
[2009-08-19 10:52:08 2832] ERROR (XendCheckpoint:164) Save failed on 
domain OVM_EL5U3_X86_PVHVM_4GB (8) - resuming.
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/xen/xend/XendCheckpoint.py", 
line 132, in save
    forkHelper(cmd, fd, saveInputHandler, False)
  File "/usr/lib/python2.4/site-packages/xen/xend/XendCheckpoint.py", 
line 406, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib/xen/bin/xc_save 32 8 0 0 5 failed
[2009-08-19 10:52:08 2832] DEBUG (XendDomainInfo:2806) 
XendDomainInfo.resumeDomain(8)


Thanks
Annie.

[-- Attachment #1.2: Type: text/html, Size: 2015 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 19/08/2009 08:39, "ANNIE LI" <annie.li@oracle.com> wrote:

> Following is the log of linux PVHVM live migration:
> 
> [2009-08-19 10:52:08 2832] DEBUG (XendCheckpoint:110) [xc_save]:
> /usr/lib/xen/bin/xc_save 32 8 0 0 5
> [2009-08-19 10:52:08 2832] INFO (XendCheckpoint:418) xc_save: failed to get
> the suspend evtchn port
> [2009-08-19 10:52:08 2832] INFO (XendCheckpoint:418)
> [2009-08-19 10:52:08 2832] INFO (XendCheckpoint:418) Saving memory pages: iter
> 1   0%ERROR Internal error: Error when writing to state file (4a) (errno 104)

The original error will be on the receive side. The sender failed because
the receiver closed down.

 -- Keir

Re: Error restoring DomU when using GPLPV

[ANNIE LI]

[-- Attachment #1.1: Type: text/plain, Size: 1023 bytes --]

Hi

Keir Fraser wrote:
> On 19/08/2009 08:39, "ANNIE LI" <annie.li@oracle.com> wrote:
>
>   
>> Following is the log of linux PVHVM live migration:
>>
>> [2009-08-19 10:52:08 2832] DEBUG (XendCheckpoint:110) [xc_save]:
>> /usr/lib/xen/bin/xc_save 32 8 0 0 5
>> [2009-08-19 10:52:08 2832] INFO (XendCheckpoint:418) xc_save: failed to get
>> the suspend evtchn port
>> [2009-08-19 10:52:08 2832] INFO (XendCheckpoint:418)
>> [2009-08-19 10:52:08 2832] INFO (XendCheckpoint:418) Saving memory pages: iter
>> 1   0%ERROR Internal error: Error when writing to state file (4a) (errno 104)
>>     
>
> The original error will be on the receive side. The sender failed because
> the receiver closed down.
>   
Please ignore my last post, it was wrong because of the kernel version 
problem.
I tested again after updating my test environment. Live migration/ save/ 
restore works well on pvhvm with Xen3.4. So this should not be problem 
for any PVonHVM domain, just windows domain with pv drivers has this 
problem.

Thanks
Annie.

[-- Attachment #1.2: Type: text/html, Size: 1509 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Error restoring DomU when using GPLPV

[ANNIE LI]

Hi
> Regarding the Linux PV-on-HVM drivers - they may have the same issue. Full
> PV guests do not as they have a gnttab_suspend() function called during
> suspend callback (and for subtle reasons xc_domain_save can detect and not
> save Xenheap pages for a full PV guests anyway - because it can see the P2M
> table in that case).
>   
Live migration of linux PVonHVM passed on  Xen3.4 successfully, but 
windows os with pv driver failed.
> Like I said before -- unmapping the gnttab pages I think will not help you
> for live migration, but I suppose it is a reasonable thing to do anyway. For
> live migration I think xc_domain_save needs t get a bit smarter about
> Xenheap pages in HVM guests.
Yes. The live migration of windows domu with pv driver failed again 
after we added unmapping gnttab and shinfo pages in suspending process. 
The save process is OK, but the restore hit the similar problem. So what 
should we do in windows pv driver to avoid this problem? Any suggestion 
for this issue?

Any suggestion is appreciated.

Following is the log of restore process:

[2009-08-21 00:17:34 2918] DEBUG (XendCheckpoint:278) [xc_restore]: 
/usr/lib/xen/bin/xc_restore 31 33 2 3 1 1 1
[2009-08-21 00:17:34 2918] INFO (XendCheckpoint:418) xc_domain_restore 
start: p2m_size = 100000
[2009-08-21 00:17:34 2918] INFO (XendCheckpoint:418) Reloading memory 
pages:   0%
[2009-08-21 00:17:44 2918] INFO (XendCheckpoint:418) Failed allocation 
for dom 33: 7 extents of order 0
[2009-08-21 00:17:44 2918] INFO (XendCheckpoint:418) ERROR Internal 
error: Failed to allocate memory for batch.!
[2009-08-21 00:17:44 2918] INFO (XendCheckpoint:418)
[2009-08-21 00:17:44 2918] INFO (XendCheckpoint:418) Restore exit with rc=1
[2009-08-21 00:17:44 2918] DEBUG (XendDomainInfo:2750) 
XendDomainInfo.destroy: domid=33
[2009-08-21 00:17:44 2918] ERROR (XendDomainInfo:2764) 
XendDomainInfo.destroy: domain destruction failed.
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/xen/xend/XendDomainInfo.py", 
line 2757, in destroy
    xc.domain_pause(self.domid)
Error: (3, 'No such process')
[2009-08-21 00:17:44 2918] DEBUG (XendDomainInfo:2225) No device model
[2009-08-21 00:17:44 2918] DEBUG (XendDomainInfo:2227) Releasing devices
[2009-08-21 00:17:44 2918] DEBUG (XendDomainInfo:2240) Removing vif/0
[2009-08-21 00:17:44 2918] DEBUG (XendDomainInfo:1142) 
XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2009-08-21 00:17:44 2918] DEBUG (XendDomainInfo:2240) Removing vbd/768
[2009-08-21 00:17:44 2918] DEBUG (XendDomainInfo:1142) 
XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/768
[2009-08-21 00:17:44 2918] DEBUG (XendDomainInfo:2240) Removing vfb/0
[2009-08-21 00:17:44 2918] DEBUG (XendDomainInfo:1142) 
XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0
[2009-08-21 00:17:44 2918] DEBUG (XendDomainInfo:2240) Removing console/0
[2009-08-21 00:17:44 2918] DEBUG (XendDomainInfo:1142) 
XendDomainInfo.destroyDevice: deviceClass = console, device = console/0
[2009-08-21 00:17:45 2918] ERROR (XendDomain:1149) Restore failed
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/xen/xend/XendDomain.py", line 
1147, in domain_restore_fd
    return XendCheckpoint.restore(self, fd, paused=paused, 
relocating=relocating)
  File "/usr/lib/python2.4/site-packages/xen/xend/XendCheckpoint.py", 
line 282, in restore
    forkHelper(cmd, fd, handler.handler, True)
  File "/usr/lib/python2.4/site-packages/xen/xend/XendCheckpoint.py", 
line 406, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib/xen/bin/xc_restore 31 33 2 3 1 1 1 failed


Thanks
Annie.

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 20/08/2009 09:17, "ANNIE LI" <annie.li@oracle.com> wrote:

>> Like I said before -- unmapping the gnttab pages I think will not help you
>> for live migration, but I suppose it is a reasonable thing to do anyway. For
>> live migration I think xc_domain_save needs t get a bit smarter about
>> Xenheap pages in HVM guests.
> Yes. The live migration of windows domu with pv driver failed again
> after we added unmapping gnttab and shinfo pages in suspending process.
> The save process is OK, but the restore hit the similar problem. So what
> should we do in windows pv driver to avoid this problem? Any suggestion
> for this issue?

Balloon down by (#gnttab+#shinfo) pages when PV drivers first load. You
could do this instead of unmapping gnttab+shinfo pages on suspend, or as
well as.

Ultimately the 'right' fix will need to be implemented in Xen and dom0
tools. But the above kludge would work perfectly well I believe.

 -- Keir

RE: Error restoring DomU when using GPLPV

[James Harper]

> 
> On 20/08/2009 09:17, "ANNIE LI" <annie.li@oracle.com> wrote:
> 
> >> Like I said before -- unmapping the gnttab pages I think will not
help you
> >> for live migration, but I suppose it is a reasonable thing to do
anyway.
> For
> >> live migration I think xc_domain_save needs t get a bit smarter
about
> >> Xenheap pages in HVM guests.
> > Yes. The live migration of windows domu with pv driver failed again
> > after we added unmapping gnttab and shinfo pages in suspending
process.
> > The save process is OK, but the restore hit the similar problem. So
what
> > should we do in windows pv driver to avoid this problem? Any
suggestion
> > for this issue?
> 
> Balloon down by (#gnttab+#shinfo) pages when PV drivers first load.
You
> could do this instead of unmapping gnttab+shinfo pages on suspend, or
as
> well as.
> 
> Ultimately the 'right' fix will need to be implemented in Xen and dom0
> tools. But the above kludge would work perfectly well I believe.
> 

Kludgy, as you say, but if it works then so be it.

Is there a future proof way for the drivers to know if they are running
under a future version of xen that isn't broken in this way? I'm
guessing not...

James

Re: Error restoring DomU when using GPLPV

[ANNIE LI]

[-- Attachment #1.1: Type: text/plain, Size: 319 bytes --]

Hi
> Kludgy, as you say, but if it works then so be it.
>   
Just finished test,  live migration works well on winpv domu after 
ballooning down those pages when pv driver first load.

Another question:
Why linux PVonHVM don't have this issue? does the linux pv driver have 
the same process like above?

Thanks
Annie.

[-- Attachment #1.2: Type: text/html, Size: 694 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 20/08/2009 10:42, "James Harper" <james.harper@bendigoit.com.au> wrote:

>> Ultimately the 'right' fix will need to be implemented in Xen and dom0
>> tools. But the above kludge would work perfectly well I believe.
>> 
> 
> Kludgy, as you say, but if it works then so be it.
> 
> Is there a future proof way for the drivers to know if they are running
> under a future version of xen that isn't broken in this way? I'm
> guessing not...

No, not really. We could add a xenstore flag or something I suppose. But
really losing a few memory pages is not the end of the world. I suppose the
major pain might be if it shatters a physical superpage, and hence makes
VT-d/EPT type stuff more expensive. If you could at least arrange for the
pages to come from the same aligned 2MB region, or even from the bottom 2MB
of memory (which can never be allocated as a superpage because of the VGA
area), that might be nice.

Thinking about how to fix this nicely in the tools, it seems pretty tricky
if I don't want to have to change the dom0 kernel too. The kernel is quite
involved in mapping foreign pages up to user space and gets in the way of
hacking in a flag between tools and Xen... And we do want to be able to map
foreign Xen-heap pages in some cases. It's only a nuisance for
xc_domain_save.

 -- Keir

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 20/08/2009 11:05, "ANNIE LI" <annie.li@oracle.com> wrote:

> Hi
>>  
>> Kludgy, as you say, but if it works then so be it.
>>   
> Just finished test,  live migration works well on winpv domu after ballooning
> down those pages when pv driver first load.
> 
> Another question:
> Why linux PVonHVM don't have this issue? does the linux pv driver have the
> same process like above?

It's working more by luck than design I fear.

 -- Keir

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 20/08/2009 11:19, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:

> No, not really. We could add a xenstore flag or something I suppose. But
> really losing a few memory pages is not the end of the world. I suppose the
> major pain might be if it shatters a physical superpage, and hence makes
> VT-d/EPT type stuff more expensive. If you could at least arrange for the
> pages to come from the same aligned 2MB region, or even from the bottom 2MB
> of memory (which can never be allocated as a superpage because of the VGA
> area), that might be nice.
> 
> Thinking about how to fix this nicely in the tools, it seems pretty tricky
> if I don't want to have to change the dom0 kernel too. The kernel is quite
> involved in mapping foreign pages up to user space and gets in the way of
> hacking in a flag between tools and Xen... And we do want to be able to map
> foreign Xen-heap pages in some cases. It's only a nuisance for
> xc_domain_save.

Another method would be for the PV-on-HVM drivers to map shinfo and gnttab
pages in a restricted guest-physical address range and then advertise the
range via, say, a new HVMPARAM. Tools could also indicate to PV-on-HVM that
this method is supported via the same HVMPARAM.

How does that sound? It would give the ability for a general exclusion range
for save/restore.

 -- Keir

Re: Error restoring DomU when using GPLPV

[ANNIE LI]

Hi
> Just finished test,  live migration works well on winpv domu after 
> ballooning down those pages when pv driver first load.
After more test, i find there are some problem with this method: 
ballooning down those pages when pv driver first load.

If i balloon down those pages when driver first load, save/restore can 
only work only once, and the migration can not work. It seems that a 
restored vm lost those pages ballooned down. For migration, destination 
does not have those pages which ballooned down on source.

But if i balloon down those pages every time(not driver first load),  i 
tested save/restore/migration for several times, and all work fine.  But 
the domu will waste lots of memory in this situation.

I will do more test about this and update here.

Thanks
Annie.

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 20/08/2009 12:55, "ANNIE LI" <annie.li@oracle.com> wrote:

>> Just finished test,  live migration works well on winpv domu after
>> ballooning down those pages when pv driver first load.
> After more test, i find there are some problem with this method:
> ballooning down those pages when pv driver first load.
> 
> If i balloon down those pages when driver first load, save/restore can
> only work only once, and the migration can not work.

Oh dear.

> It seems that a 
> restored vm lost those pages ballooned down. For migration, destination
> does not have those pages which ballooned down on source.

Right, that's the correct behaviour isn't it? Pages freed on source VM do
not magically reappear on destination VM?

> But if i balloon down those pages every time(not driver first load),  i
> tested save/restore/migration for several times, and all work fine.  But
> the domu will waste lots of memory in this situation.

Yes, that's weird. Do you know what condition causes guest memory allocation
failure on xc_domain_restore? Is it due to hitting the guest maxmem limit in
Xen? If so, is maxmem the same value across multiple iterations of
save/restore or migration?

> I will do more test about this and update here.

Thanks.

 -- Keir

Re: Error restoring DomU when using GPLPV

[ANNIE LI]

[-- Attachment #1.1: Type: text/plain, Size: 1013 bytes --]

Hi

>> It seems that a 
>> restored vm lost those pages ballooned down. For migration, destination
>> does not have those pages which ballooned down on source.
>>     
>
> Right, that's the correct behaviour isn't it? Pages freed on source VM do
> not magically reappear on destination VM?
>
>   
Yes,  so this method can not fix this problem.
>> But if i balloon down those pages every time(not driver first load),  i
>> tested save/restore/migration for several times, and all work fine.  But
>> the domu will waste lots of memory in this situation.
>>     
>
> Yes, that's weird. Do you know what condition causes guest memory allocation
> failure on xc_domain_restore? Is it due to hitting the guest maxmem limit in
> Xen? If so, is maxmem the same value across multiple iterations of
> save/restore or migration?
>   
Sorry, i have no idea about it. Maybe I need to print more log in 
for(;;) in xc_domain_restore to see what is the difference between 
without and with balooning down pages.

Thanks
Annie.


[-- Attachment #1.2: Type: text/html, Size: 1635 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Error restoring DomU when using GPLPV

[Wayne Gong]

[-- Attachment #1: Type: text/plain, Size: 2569 bytes --]

James Harper wrote:
>
> I added some more debugging...
>
> batch 1024 [1]
> Allocating 1024 mfns [2]
> 197600 allocated [3]
> batch 1024
> Allocating 33 mfns
> Failed allocation for dom 24: 33 extents of order 0 (err = 32) [4]
>
> [1] is just after 'j' is read in xc_domain_restore
> [2] is just before the call to populate_physmap
> [3] is just after the call to populate_physmap
> [4] is the error message in the memory_op function in libxc modified to
> give the value of err
>
> According to the a total of 197632 are being allocated and the last page
> cannot be (could be more pages required the next time around the loop
> too...)

I've also added some debugging in xc_domain_save.c and 
xc_domain_restore.c. Attach has those two files which I am using and 
xend.log. Those two files are modified based on Xen 3.4.0-xx.

What I've done is:
1, Create a Windows Server 2008 32bit with PV drivers which can support 
migration on Xen 3.1.x.
2, Save windows DomU then restore it. Error log is the same as the first 
mail of this thread.
3, Create a fresh install Windows 2008 32bit VM with almost the same 
vm.cfg file.
4, Save and restore windows DomU.

Here is some xend logs. I am not very sure if it's fine as I didn't know 
the whole process of save/restore in Xen. If you need any more 
information, please let me know.

    Line 100: [2009-08-26 01:34:24 2883] INFO (XendCheckpoint:418) 
shared_info_frame: 0x41ec
    Line 167: [2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) 
shared_info_frame: 0xffffffff
    Line 581: [2009-08-26 02:10:47 2883] INFO (XendCheckpoint:418) 
shared_info_frame: 0xfffff
    Line 649: [2009-08-26 02:11:01 2883] INFO (XendCheckpoint:418) 
shared_info_frame: 0xffffffff    <-- is this OK for a PVHVM when restoring.

   Line 169: [2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) 
Reloading memory pages:   0%
   Line 170: [2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
   Line 171: [2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) 
nr_mfns: 991      <----

   Line 651: [2009-08-26 02:11:01 2883] INFO (XendCheckpoint:418) 
Reloading memory pages:   0%
   Line 652: [2009-08-26 02:11:01 2883] INFO (XendCheckpoint:418) batch 1024
   Line 653: [2009-08-26 02:11:01 2883] INFO (XendCheckpoint:418) 
nr_mfns: 992      <---- nf_mfns of HVM is one more lager then PVHVM's

   Line 472: [2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
   Line 473: [2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) 
nr_mfns: 31   <---Restore a PVHVM will allocate 31 more nf_mfns then HVM.

thanks
wayne

[-- Attachment #2: xend.log --]
[-- Type: text/plain, Size: 94722 bytes --]

[2009-08-26 01:25:40 2907] INFO (SrvDaemon:220) Xend exited with status 0.
[2009-08-26 01:27:29 2883] INFO (SrvDaemon:332) Xend Daemon started
[2009-08-26 01:27:29 2883] INFO (SrvDaemon:336) Xend changeset: unavailable.
[2009-08-26 01:27:29 2883] DEBUG (XendDomainInfo:141) XendDomainInfo.recreate({'max_vcpu_id': 1, 'cpu_time': 9322924443L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 0, 'dying': 0, 'online_vcpus': 2, 'domid': 0, 'paused': 0, 'crashed': 0, 'running': 1, 'maxmem_kb': 4294967292L, 'shutdown': 0, 'mem_kb': 555008L, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'blocked': 0, 'name': 'Domain-0'})
[2009-08-26 01:27:29 2883] INFO (XendDomainInfo:158) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000. at /local/domain/0
[2009-08-26 01:27:29 2883] DEBUG (XendDomainInfo:3082) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 'image': '(linux (kernel ))', 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '2', 'vcpu_avail': '3', 'bootloader': '', 'name': 'Domain-0'}
[2009-08-26 01:27:29 2883] DEBUG (XendDomainInfo:1638) Storing domain details: {'name': 'Domain-0', 'console/limit': '1048576', 'memory/target': '555008', 'vm': '/vm/00000000-0000-0000-0000-000000000000', 'domid': '0', 'cpu/0/availability': 'online', 'cpu/1/availability': 'online', 'control/platform-feature-multiprocessor-suspend': '1', 'console/type': 'xenconsoled'}
[2009-08-26 01:27:29 2883] DEBUG (XendDomain:452) Adding Domain: 0
[2009-08-26 01:27:29 2883] DEBUG (XendDomain:386) number of vcpus to use is 0
[2009-08-26 01:27:30 2883] INFO (SrvServer:177) unix path=/var/lib/xend/xend-socket
[2009-08-26 01:27:30 2883] WARNING (XendAPI:695) API call: VBD.set_device not found
[2009-08-26 01:27:30 2883] WARNING (XendAPI:695) API call: VBD.set_type not found
[2009-08-26 01:27:30 2883] WARNING (XendAPI:695) API call: session.get_all_records not found
[2009-08-26 01:27:30 2883] WARNING (XendAPI:695) API call: event.get_record not found
[2009-08-26 01:27:30 2883] WARNING (XendAPI:695) API call: event.get_all not found
[2009-08-26 01:27:30 2883] DEBUG (XendDomainInfo:1725) XendDomainInfo.handleShutdownWatch
[2009-08-26 01:27:30 2883] WARNING (XendAPI:695) API call: VIF.get_network not found
[2009-08-26 01:27:30 2883] WARNING (XendAPI:695) API call: VIF.set_device not found
[2009-08-26 01:27:30 2883] WARNING (XendAPI:695) API call: VIF.set_MAC not found
[2009-08-26 01:27:30 2883] WARNING (XendAPI:695) API call: VIF.set_MTU not found
[2009-08-26 01:27:30 2883] WARNING (XendAPI:695) API call: debug.get_all not found
[2009-08-26 01:27:30 2883] INFO (XMLRPCServer:156) Opening Unix domain socket XML-RPC server on /var/run/xend/xen-api.sock; authentication has been disabled for this server.
[2009-08-26 01:27:30 2883] INFO (XMLRPCServer:156) Opening Unix domain socket XML-RPC server on /var/run/xend/xmlrpc.sock.
[2009-08-26 01:32:48 2883] DEBUG (XendDomainInfo:93) XendDomainInfo.create(['vm', ['name', 'srv2008_free'], ['memory', '600'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['vcpus', 1], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['bootloader', '/usr/bin/pygrub'], ['bootloader_args', '-q'], ['image', ['hvm', ['kernel', '/usr/lib/xen/boot/hvmloader'], ['videoram', 4], ['device_model', '/usr/lib/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 1], ['boot', 'c'], ['fda', ''], ['fdb', ''], ['timer_mode', 1], ['localtime', 0], ['serial', 'pty'], ['stdvga', 0], ['isa', 0], ['nographic', 0], ['soundhw', ''], ['vnc', 1], ['vncunused', 1], ['vncconsole', 1], ['vnclisten', '0.0.0.0'], ['xauthority', '/root/.Xauthority'], ['rtc_timeoffset', 0], ['monitor', 0], ['acpi', 1], ['apic', 1], ['usb', 0], ['usbdevice', 'tablet'], ['keymap', ''], ['pci', []], ['hpet', 0], ['guest_os_type', 'default'], ['hap', 1], ['cpuid', []], ['cpuid_check', []], ['viridian', 0], ['pci_msitranslate', 1], ['vpt_align', 1], ['pci_power_mgmt', 0], ['xen_platform_pci', 1], ['vncpasswd', 'XXXXXXXX']]], ['s3_integrity', 1], ['device', ['vbd', ['uname', 'file:/mnt/data/mgi/srv2008/System.img'], ['dev', 'hda'], ['mode', 'w!']]], ['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3E:55:D1:AA']]]])
[2009-08-26 01:32:48 2883] DEBUG (XendDomainInfo:2317) XendDomainInfo.constructDomain
[2009-08-26 01:32:48 2883] DEBUG (balloon:181) Balloon: 1472776 KiB free; need 4096; done.
[2009-08-26 01:32:48 2883] DEBUG (XendDomain:452) Adding Domain: 1
[2009-08-26 01:32:48 2883] DEBUG (XendDomainInfo:2518) XendDomainInfo.initDomain: 1 256
[2009-08-26 01:32:48 2883] DEBUG (image:319) No VNC passwd configured for vfb access
[2009-08-26 01:32:48 2883] DEBUG (image:806) args: boot, val: c
[2009-08-26 01:32:48 2883] DEBUG (image:806) args: fda, val: None
[2009-08-26 01:32:48 2883] DEBUG (image:806) args: fdb, val: None
[2009-08-26 01:32:48 2883] DEBUG (image:806) args: soundhw, val: None
[2009-08-26 01:32:48 2883] DEBUG (image:806) args: localtime, val: 0
[2009-08-26 01:32:48 2883] DEBUG (image:806) args: serial, val: ['pty']
[2009-08-26 01:32:48 2883] DEBUG (image:806) args: std-vga, val: 0
[2009-08-26 01:32:48 2883] DEBUG (image:806) args: isa, val: 0
[2009-08-26 01:32:48 2883] DEBUG (image:806) args: acpi, val: 1
[2009-08-26 01:32:48 2883] DEBUG (image:806) args: usb, val: 0
[2009-08-26 01:32:48 2883] DEBUG (image:806) args: usbdevice, val: tablet
[2009-08-26 01:32:48 2883] INFO (image:742) Need to create platform device.[domid:1]
[2009-08-26 01:32:48 2883] DEBUG (XendDomainInfo:2542) _initDomain:shadow_memory=0x0, memory_static_max=0x25800000, memory_static_min=0x0.
[2009-08-26 01:32:48 2883] DEBUG (balloon:181) Balloon: 1471364 KiB free; need 627712; done.
[2009-08-26 01:32:48 2883] INFO (image:170) buildDomain os=hvm dom=1 vcpus=1
[2009-08-26 01:32:48 2883] DEBUG (image:859) domid          = 1
[2009-08-26 01:32:48 2883] DEBUG (image:860) image          = /usr/lib/xen/boot/hvmloader
[2009-08-26 01:32:48 2883] DEBUG (image:861) store_evtchn   = 2
[2009-08-26 01:32:48 2883] DEBUG (image:862) memsize        = 600
[2009-08-26 01:32:48 2883] DEBUG (image:863) target         = 600
[2009-08-26 01:32:48 2883] DEBUG (image:864) vcpus          = 1
[2009-08-26 01:32:48 2883] DEBUG (image:865) acpi           = 1
[2009-08-26 01:32:48 2883] DEBUG (image:866) apic           = 1
[2009-08-26 01:32:48 2883] INFO (XendDomainInfo:2178) createDevice: vfb : {'vncunused': 1, 'vnclisten': '0.0.0.0', 'vnc': '1', 'uuid': '5fc79804-6d30-4b3a-7dee-b6f6f07774e3', 'other_config': {'vncunused': 1, 'vnclisten': '0.0.0.0', 'vnc': '1'}}
[2009-08-26 01:32:48 2883] DEBUG (DevController:95) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/vfb/1/0'} to /local/domain/1/device/vfb/0.
[2009-08-26 01:32:48 2883] DEBUG (DevController:97) DevController: writing {'vncunused': '1', 'domain': 'srv2008_free', 'frontend': '/local/domain/1/device/vfb/0', 'uuid': '5fc79804-6d30-4b3a-7dee-b6f6f07774e3', 'frontend-id': '1', 'vnclisten': '0.0.0.0', 'state': '1', 'online': '1', 'vnc': '1'} to /local/domain/0/backend/vfb/1/0.
[2009-08-26 01:32:48 2883] INFO (XendDomainInfo:2178) createDevice: vbd : {'uuid': '0d11ea8b-358f-0bdb-5aa3-193cb944dd25', 'bootable': 1, 'driver': 'paravirtualised', 'dev': 'hda', 'uname': 'file:/mnt/data/mgi/srv2008/System.img', 'mode': 'w!'}
[2009-08-26 01:32:48 2883] DEBUG (DevController:95) DevController: writing {'backend-id': '0', 'virtual-device': '768', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/1/768'} to /local/domain/1/device/vbd/768.
[2009-08-26 01:32:48 2883] DEBUG (DevController:97) DevController: writing {'domain': 'srv2008_free', 'frontend': '/local/domain/1/device/vbd/768', 'uuid': '0d11ea8b-358f-0bdb-5aa3-193cb944dd25', 'bootable': '1', 'dev': 'hda', 'state': '1', 'params': '/mnt/data/mgi/srv2008/System.img', 'mode': 'w!', 'online': '1', 'frontend-id': '1', 'type': 'file'} to /local/domain/0/backend/vbd/1/768.
[2009-08-26 01:32:48 2883] INFO (XendDomainInfo:2178) createDevice: vif : {'bridge': 'xenbr0', 'mac': '00:16:3E:55:D1:AA', 'uuid': 'dd13ff5c-2ac3-bbaa-2019-b0201a133f6a'}
[2009-08-26 01:32:48 2883] DEBUG (DevController:95) DevController: writing {'backend-id': '0', 'mac': '00:16:3E:55:D1:AA', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/1/0'} to /local/domain/1/device/vif/0.
[2009-08-26 01:32:48 2883] DEBUG (DevController:97) DevController: writing {'bridge': 'xenbr0', 'domain': 'srv2008_free', 'handle': '0', 'uuid': 'dd13ff5c-2ac3-bbaa-2019-b0201a133f6a', 'script': '/etc/xen/scripts/vif-bridge', 'mac': '00:16:3E:55:D1:AA', 'frontend-id': '1', 'state': '1', 'online': '1', 'frontend': '/local/domain/1/device/vif/0'} to /local/domain/0/backend/vif/1/0.
[2009-08-26 01:32:48 2883] INFO (image:391) spawning device models: /usr/lib/xen/bin/qemu-dm ['/usr/lib/xen/bin/qemu-dm', '-d', '1', '-domain-name', 'srv2008_free', '-videoram', '4', '-vnc', '0.0.0.0:0', '-vncunused', '-vcpus', '1', '-boot', 'c', '-serial', 'pty', '-acpi', '-usbdevice', 'tablet', '-net', 'nic,vlan=1,macaddr=00:16:3E:55:D1:AA,model=rtl8139', '-net', 'tap,vlan=1,ifname=tap1.0,bridge=xenbr0', '-M', 'xenfv']
[2009-08-26 01:32:48 2883] INFO (image:440) device model pid: 3321
[2009-08-26 01:32:48 2883] INFO (image:528) waiting for sentinel_fifo
[2009-08-26 01:32:48 2883] DEBUG (XendDomainInfo:3082) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '6', 'uuid': '166f15a6-6e1d-4d5c-5517-4064fd810951', 'on_reboot': 'restart', 'start_time': '1251221568.66', 'on_poweroff': 'destroy', 'bootloader_args': '-q', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'bootloader': '/usr/bin/pygrub', 'image': '(hvm (kernel ) (videoram 4) (hpet 0) (stdvga 0) (vnclisten 0.0.0.0) (loader /usr/lib/xen/boot/hvmloader) (vncconsole 1) (serial pty) (vncunused 1) (xen_platform_pci 1) (boot c) (rtc_timeoffset 0) (pci ()) (pae 1) (vpt_align 1) (hap 1) (viridian 0) (acpi 1) (localtime 0) (timer_mode 1) (vnc 1) (nographic 0) (guest_os_type default) (pci_msitranslate 1) (apic 1) (monitor 0) (usbdevice tablet) (device_model /usr/lib/xen/bin/qemu-dm) (pci_power_mgmt 0) (usb 0) (xauthority /root/.Xauthority) (isa 0) (notes (SUSPEND_CANCEL 1)))', 'name': 'srv2008_free'}
[2009-08-26 01:32:48 2883] DEBUG (XendDomainInfo:1638) Storing domain details: {'console/port': '3', 'name': 'srv2008_free', 'console/limit': '1048576', 'store/port': '2', 'vm': '/vm/166f15a6-6e1d-4d5c-5517-4064fd810951', 'domid': '1', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'memory/target': '614400', 'control/platform-feature-multiprocessor-suspend': '1', 'store/ring-ref': '1044476', 'console/type': 'ioemu'}
[2009-08-26 01:32:48 2883] DEBUG (DevController:95) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/1/0'} to /local/domain/1/device/console/0.
[2009-08-26 01:32:48 2883] DEBUG (DevController:97) DevController: writing {'domain': 'srv2008_free', 'frontend': '/local/domain/1/device/console/0', 'uuid': 'f5a40d7b-8bc1-48bd-d4f2-6c7c2bb9e723', 'frontend-id': '1', 'state': '1', 'location': '3', 'online': '1', 'protocol': 'vt100'} to /local/domain/0/backend/console/1/0.
[2009-08-26 01:32:48 2883] DEBUG (XendDomainInfo:1725) XendDomainInfo.handleShutdownWatch
[2009-08-26 01:32:48 2883] DEBUG (DevController:139) Waiting for devices vif.
[2009-08-26 01:32:48 2883] DEBUG (DevController:144) Waiting for 0.
[2009-08-26 01:32:48 2883] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2009-08-26 01:32:49 2883] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vif/1/0/hotplug-status.
[2009-08-26 01:32:49 2883] DEBUG (DevController:643) hotplugStatusCallback 1.
[2009-08-26 01:32:49 2883] DEBUG (DevController:139) Waiting for devices vscsi.
[2009-08-26 01:32:49 2883] DEBUG (DevController:139) Waiting for devices vbd.
[2009-08-26 01:32:49 2883] DEBUG (DevController:144) Waiting for 768.
[2009-08-26 01:32:49 2883] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vbd/1/768/hotplug-status.
[2009-08-26 01:32:49 2883] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vbd/1/768/hotplug-status.
[2009-08-26 01:32:49 2883] DEBUG (DevController:643) hotplugStatusCallback 1.
[2009-08-26 01:32:49 2883] DEBUG (DevController:139) Waiting for devices irq.
[2009-08-26 01:32:49 2883] DEBUG (DevController:139) Waiting for devices vkbd.
[2009-08-26 01:32:49 2883] DEBUG (DevController:139) Waiting for devices vfb.
[2009-08-26 01:32:49 2883] DEBUG (DevController:139) Waiting for devices console.
[2009-08-26 01:32:49 2883] DEBUG (DevController:144) Waiting for 0.
[2009-08-26 01:32:49 2883] DEBUG (DevController:139) Waiting for devices pci.
[2009-08-26 01:32:49 2883] DEBUG (DevController:139) Waiting for devices ioports.
[2009-08-26 01:32:49 2883] DEBUG (DevController:139) Waiting for devices tap.
[2009-08-26 01:32:49 2883] DEBUG (DevController:139) Waiting for devices vtpm.
[2009-08-26 01:32:49 2883] INFO (XendDomain:1180) Domain srv2008_free (1) unpaused.
[2009-08-26 01:34:24 2883] DEBUG (XendCheckpoint:110) [xc_save]: /usr/lib/xen/bin/xc_save 31 1 0 0 4
[2009-08-26 01:34:24 2883] DEBUG (XendCheckpoint:389) suspend
[2009-08-26 01:34:24 2883] DEBUG (XendCheckpoint:113) In saveInputHandler suspend
[2009-08-26 01:34:24 2883] INFO (XendCheckpoint:418) xc_save: failed to get the suspend evtchn port
[2009-08-26 01:34:24 2883] DEBUG (XendCheckpoint:115) Suspending 1 ...
[2009-08-26 01:34:24 2883] INFO (XendCheckpoint:418) 
[2009-08-26 01:34:24 2883] INFO (XendCheckpoint:418) -->xc_domain_save()
[2009-08-26 01:34:24 2883] INFO (XendCheckpoint:418) platform info, max_mfn: 7d4b0, hvirt_start ffc00000, pt_lvevel 4, guest_width 8
[2009-08-26 01:34:24 2883] DEBUG (XendDomainInfo:518) XendDomainInfo.shutdown(suspend)
[2009-08-26 01:34:24 2883] INFO (XendCheckpoint:418) shared_info_frame: 0x41ec
[2009-08-26 01:34:24 2883] DEBUG (XendDomainInfo:1725) XendDomainInfo.handleShutdownWatch
[2009-08-26 01:34:24 2883] INFO (XendDomainInfo:1913) Domain has shutdown: name=migrating-srv2008_free id=1 reason=suspend.
[2009-08-26 01:34:24 2883] INFO (XendCheckpoint:121) Domain 1 suspended.
[2009-08-26 01:34:24 2883] INFO (image:476) signalDeviceModel:restore dm state to running
[2009-08-26 01:34:24 2883] DEBUG (XendCheckpoint:130) Written done
[2009-08-26 01:34:24 2883] INFO (XendCheckpoint:418) p2m_size: 0x100000Need another buffer for HVM context
[2009-08-26 01:34:24 2883] INFO (XendCheckpoint:418) Setup the mfn_to_pfn table mapping
[2009-08-26 01:34:24 2883] INFO (XendCheckpoint:418) Start writing out the saved-domain record.
[2009-08-26 01:34:31 2883] INFO (XendCheckpoint:418) Saving memory pages: iter 1   0%\b\b\b\b  5%\b\b\b\b 10%\b\b\b\b 15%\b\b\b\b 20%\b\b\b\b 25%\b\b\b\b 30%\b\b\b\b 35%\b\b\b\b 40%\b\b\b\b 45%\b\b\b\b 50%\b\b\b\b 55%\b\b\b\b 60%\b\b\b\b 65%\b\b\b\b 70%\b\b\b\b 75%\b\b\b\b 80%\b\b\b\b 85%\b\b\b\b 90%\b\b\b\b 95%\r 1: sent 157696, skipped 0, delta 6910ms, dom0 57%, target 0%, sent 747Mb/s, dirtied 0Mb/s 0 pages
[2009-08-26 01:34:31 2883] INFO (XendCheckpoint:418) Total pages sent= 157696 (0.15x)
[2009-08-26 01:34:31 2883] INFO (XendCheckpoint:418) (of which 0 were fixups)
[2009-08-26 01:34:31 2883] INFO (XendCheckpoint:418) All memory is saved
[2009-08-26 01:34:33 2883] INFO (XendCheckpoint:418) Save exit rc=0
[2009-08-26 01:34:33 2883] DEBUG (XendDomainInfo:2750) XendDomainInfo.destroy: domid=1
[2009-08-26 01:34:33 2883] DEBUG (XendDomainInfo:2220) Destroying device model
[2009-08-26 01:34:33 2883] INFO (image:553) migrating-srv2008_free device model terminated
[2009-08-26 01:34:33 2883] DEBUG (XendDomainInfo:2227) Releasing devices
[2009-08-26 01:34:33 2883] DEBUG (XendDomainInfo:2240) Removing vif/0
[2009-08-26 01:34:33 2883] DEBUG (XendDomainInfo:1142) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2009-08-26 01:34:33 2883] DEBUG (XendDomainInfo:2240) Removing vbd/768
[2009-08-26 01:34:33 2883] DEBUG (XendDomainInfo:1142) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/768
[2009-08-26 01:34:33 2883] DEBUG (XendDomainInfo:2240) Removing vfb/0
[2009-08-26 01:34:33 2883] DEBUG (XendDomainInfo:1142) XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0
[2009-08-26 01:34:33 2883] DEBUG (XendDomainInfo:2240) Removing console/0
[2009-08-26 01:34:33 2883] DEBUG (XendDomainInfo:1142) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0
[2009-08-26 01:34:38 2883] DEBUG (XendDomainInfo:227) XendDomainInfo.restore(['domain', ['domid', '1'], ['on_crash', 'restart'], ['uuid', '166f15a6-6e1d-4d5c-5517-4064fd810951'], ['bootloader_args', '-q'], ['vcpus', '1'], ['name', 'srv2008_free'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['cpus', [[]]], ['bootloader', '/usr/bin/pygrub'], ['maxmem', '600'], ['memory', '600'], ['shadow_memory', '6'], ['vcpu_avail', '1'], ['features'], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['start_time', '1251221568.66'], ['cpu_time', '0.008097537'], ['online_vcpus', '1'], ['image', ['hvm', ['kernel'], ['videoram', '4'], ['hpet', '0'], ['stdvga', '0'], ['vnclisten', '0.0.0.0'], ['loader', '/usr/lib/xen/boot/hvmloader'], ['vncconsole', '1'], ['serial', 'pty'], ['vncunused', '1'], ['xen_platform_pci', '1'], ['boot', 'c'], ['rtc_timeoffset', '0'], ['pci', []], ['pae', '1'], ['vpt_align', '1'], ['hap', '1'], ['viridian', '0'], ['acpi', '1'], ['localtime', '0'], ['timer_mode', '1'], ['vnc', '1'], ['nographic', '0'], ['guest_os_type', 'default'], ['pci_msitranslate', '1'], ['apic', '1'], ['monitor', '0'], ['usbdevice', 'tablet'], ['device_model', '/usr/lib/xen/bin/qemu-dm'], ['pci_power_mgmt', '0'], ['usb', '0'], ['xauthority', '/root/.Xauthority'], ['isa', '0'], ['notes', ['SUSPEND_CANCEL', '1']]]], ['status', '2'], ['state', '------'], ['store_mfn', '1044476'], ['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3E:55:D1:AA'], ['script', '/etc/xen/scripts/vif-bridge'], ['uuid', 'dd13ff5c-2ac3-bbaa-2019-b0201a133f6a'], ['backend', '0']]], ['device', ['vbd', ['uuid', '0d11ea8b-358f-0bdb-5aa3-193cb944dd25'], ['bootable', '1'], ['dev', 'hda:disk'], ['uname', 'file:/mnt/data/mgi/srv2008/System.img'], ['mode', 'w!'], ['backend', '0'], ['bootable', '1'], ['VDI']]], ['device', ['vfb', ['vncunused', '1'], ['vnclisten', '0.0.0.0'], ['vnc', '1'], ['uuid', '5fc79804-6d30-4b3a-7dee-b6f6f07774e3'], ['location', '0.0.0.0:5900']]], ['device', ['console', ['protocol', 'vt100'], ['location', '3'], ['uuid', 'f5a40d7b-8bc1-48bd-d4f2-6c7c2bb9e723']]]])
[2009-08-26 01:34:38 2883] DEBUG (XendDomainInfo:2317) XendDomainInfo.constructDomain
[2009-08-26 01:34:38 2883] DEBUG (balloon:181) Balloon: 1472776 KiB free; need 4096; done.
[2009-08-26 01:34:38 2883] DEBUG (XendDomain:452) Adding Domain: 2
[2009-08-26 01:34:38 2883] DEBUG (XendDomainInfo:3082) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '6', 'uuid': '166f15a6-6e1d-4d5c-5517-4064fd810951', 'on_reboot': 'restart', 'start_time': '1251221568.66', 'on_poweroff': 'destroy', 'bootloader_args': '-q', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'bootloader': '/usr/bin/pygrub', 'image': '(hvm (kernel ) (videoram 4) (hpet 0) (stdvga 0) (vnclisten 0.0.0.0) (loader /usr/lib/xen/boot/hvmloader) (vncconsole 1) (serial pty) (vncunused 1) (xen_platform_pci 1) (boot c) (rtc_timeoffset 0) (pci ()) (pae 1) (vpt_align 1) (hap 1) (viridian 0) (acpi 1) (localtime 0) (timer_mode 1) (vnc 1) (nographic 0) (guest_os_type default) (pci_msitranslate 1) (apic 1) (monitor 0) (usbdevice tablet) (device_model /usr/lib/xen/bin/qemu-dm) (pci_power_mgmt 0) (usb 0) (xauthority /root/.Xauthority) (isa 0) (notes (SUSPEND_CANCEL 1)))', 'name': 'srv2008_free'}
[2009-08-26 01:34:38 2883] INFO (XendDomainInfo:2178) createDevice: vfb : {'vncunused': '1', 'other_config': {'vncunused': '1', 'vnclisten': '0.0.0.0', 'vnc': '1'}, 'vnc': '1', 'uuid': '5fc79804-6d30-4b3a-7dee-b6f6f07774e3', 'vnclisten': '0.0.0.0', 'location': '0.0.0.0:5900'}
[2009-08-26 01:34:38 2883] DEBUG (DevController:95) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/vfb/2/0'} to /local/domain/2/device/vfb/0.
[2009-08-26 01:34:38 2883] DEBUG (DevController:97) DevController: writing {'vncunused': '1', 'domain': 'srv2008_free', 'frontend': '/local/domain/2/device/vfb/0', 'uuid': '5fc79804-6d30-4b3a-7dee-b6f6f07774e3', 'frontend-id': '2', 'vnclisten': '0.0.0.0', 'state': '1', 'location': '0.0.0.0:5900', 'online': '1', 'vnc': '1'} to /local/domain/0/backend/vfb/2/0.
[2009-08-26 01:34:38 2883] INFO (XendDomainInfo:2178) createDevice: console : {'protocol': 'vt100', 'location': '3', 'uuid': 'f5a40d7b-8bc1-48bd-d4f2-6c7c2bb9e723'}
[2009-08-26 01:34:38 2883] DEBUG (DevController:95) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/2/0'} to /local/domain/2/device/console/0.
[2009-08-26 01:34:38 2883] DEBUG (DevController:97) DevController: writing {'domain': 'srv2008_free', 'frontend': '/local/domain/2/device/console/0', 'uuid': 'f5a40d7b-8bc1-48bd-d4f2-6c7c2bb9e723', 'frontend-id': '2', 'state': '1', 'location': '3', 'online': '1', 'protocol': 'vt100'} to /local/domain/0/backend/console/2/0.
[2009-08-26 01:34:38 2883] INFO (XendDomainInfo:2178) createDevice: vbd : {'uuid': '0d11ea8b-358f-0bdb-5aa3-193cb944dd25', 'bootable': '1', 'driver': 'paravirtualised', 'dev': 'hda:disk', 'uname': 'file:/mnt/data/mgi/srv2008/System.img', 'mode': 'w!', 'backend': '0'}
[2009-08-26 01:34:38 2883] DEBUG (DevController:95) DevController: writing {'backend-id': '0', 'virtual-device': '768', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/2/768'} to /local/domain/2/device/vbd/768.
[2009-08-26 01:34:38 2883] DEBUG (DevController:97) DevController: writing {'domain': 'srv2008_free', 'frontend': '/local/domain/2/device/vbd/768', 'uuid': '0d11ea8b-358f-0bdb-5aa3-193cb944dd25', 'bootable': '1', 'dev': 'hda', 'state': '1', 'params': '/mnt/data/mgi/srv2008/System.img', 'mode': 'w!', 'online': '1', 'frontend-id': '2', 'type': 'file'} to /local/domain/0/backend/vbd/2/768.
[2009-08-26 01:34:38 2883] INFO (XendDomainInfo:2178) createDevice: vif : {'bridge': 'xenbr0', 'mac': '00:16:3E:55:D1:AA', 'script': '/etc/xen/scripts/vif-bridge', 'uuid': 'dd13ff5c-2ac3-bbaa-2019-b0201a133f6a', 'backend': '0'}
[2009-08-26 01:34:38 2883] DEBUG (DevController:95) DevController: writing {'backend-id': '0', 'mac': '00:16:3E:55:D1:AA', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/2/0'} to /local/domain/2/device/vif/0.
[2009-08-26 01:34:38 2883] DEBUG (DevController:97) DevController: writing {'bridge': 'xenbr0', 'domain': 'srv2008_free', 'handle': '0', 'uuid': 'dd13ff5c-2ac3-bbaa-2019-b0201a133f6a', 'script': '/etc/xen/scripts/vif-bridge', 'mac': '00:16:3E:55:D1:AA', 'frontend-id': '2', 'state': '1', 'online': '1', 'frontend': '/local/domain/2/device/vif/0'} to /local/domain/0/backend/vif/2/0.
[2009-08-26 01:34:38 2883] DEBUG (XendDomainInfo:1638) Storing domain details: {'console/port': '3', 'name': 'srv2008_free', 'console/limit': '1048576', 'image/suspend-cancel': '1', 'domid': '2', 'vm': '/vm/166f15a6-6e1d-4d5c-5517-4064fd810951', 'cpu/0/availability': 'online', 'memory/target': '614400', 'control/platform-feature-multiprocessor-suspend': '1', 'console/type': 'ioemu', 'store/port': '2'}
[2009-08-26 01:34:38 2883] INFO (XendCheckpoint:243) restore hvm domain 2, apic=1, pae=1
[2009-08-26 01:34:38 2883] DEBUG (image:319) No VNC passwd configured for vfb access
[2009-08-26 01:34:38 2883] DEBUG (image:806) args: boot, val: c
[2009-08-26 01:34:38 2883] DEBUG (image:806) args: fda, val: None
[2009-08-26 01:34:38 2883] DEBUG (image:806) args: fdb, val: None
[2009-08-26 01:34:38 2883] DEBUG (image:806) args: soundhw, val: None
[2009-08-26 01:34:38 2883] DEBUG (image:806) args: localtime, val: 0
[2009-08-26 01:34:38 2883] DEBUG (image:806) args: serial, val: ['pty']
[2009-08-26 01:34:38 2883] DEBUG (image:806) args: std-vga, val: 0
[2009-08-26 01:34:38 2883] DEBUG (image:806) args: isa, val: 0
[2009-08-26 01:34:38 2883] DEBUG (image:806) args: acpi, val: 1
[2009-08-26 01:34:38 2883] DEBUG (image:806) args: usb, val: 0
[2009-08-26 01:34:38 2883] DEBUG (image:806) args: usbdevice, val: tablet
[2009-08-26 01:34:38 2883] INFO (image:742) Need to create platform device.[domid:2]
[2009-08-26 01:34:38 2883] DEBUG (XendCheckpoint:261) restore:shadow=0x6, _static_max=0x25800000, _static_min=0x0, 
[2009-08-26 01:34:38 2883] DEBUG (balloon:181) Balloon: 1471364 KiB free; need 624640; done.
[2009-08-26 01:34:38 2883] DEBUG (XendCheckpoint:278) [xc_restore]: /usr/lib/xen/bin/xc_restore 31 2 2 3 1 1 1
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) -->xc_domain_restore()
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) xc_domain_restore start: p2m_size = 100000
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) platform info, max_mfn: 7d4b0, hvirt_start ffc00000, pt_lvevel 4, guest_width 8
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) memory alloc success
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) lock regin_mfn and p2m_batch success
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) shared_info_frame: 0xffffffff
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) all PFNs are marked as invalid
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) Reloading memory pages:   0%
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 991
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1023
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:39 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:40 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:41 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:42 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:43 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:44 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:45 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) nr_mfns: 31
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) nr_mfns: 7
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) Failed allocation for dom 2: 7 extents of order 0
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) ERROR Internal error: Failed to allocate memory for batch.!
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) 
[2009-08-26 01:34:46 2883] INFO (XendCheckpoint:418) Restore exit with rc=1
[2009-08-26 01:34:46 2883] DEBUG (XendDomainInfo:2750) XendDomainInfo.destroy: domid=2
[2009-08-26 01:34:46 2883] ERROR (XendDomainInfo:2764) XendDomainInfo.destroy: domain destruction failed.
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/xen/xend/XendDomainInfo.py", line 2757, in destroy
    xc.domain_pause(self.domid)
Error: (3, 'No such process')
[2009-08-26 01:34:46 2883] DEBUG (XendDomainInfo:2225) No device model
[2009-08-26 01:34:46 2883] DEBUG (XendDomainInfo:2227) Releasing devices
[2009-08-26 01:34:46 2883] DEBUG (XendDomainInfo:2240) Removing vif/0
[2009-08-26 01:34:46 2883] DEBUG (XendDomainInfo:1142) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2009-08-26 01:34:46 2883] DEBUG (XendDomainInfo:2240) Removing vbd/768
[2009-08-26 01:34:46 2883] DEBUG (XendDomainInfo:1142) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/768
[2009-08-26 01:34:46 2883] DEBUG (XendDomainInfo:2240) Removing vfb/0
[2009-08-26 01:34:46 2883] DEBUG (XendDomainInfo:1142) XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0
[2009-08-26 01:34:46 2883] DEBUG (XendDomainInfo:2240) Removing console/0
[2009-08-26 01:34:46 2883] DEBUG (XendDomainInfo:1142) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0
[2009-08-26 01:34:46 2883] ERROR (XendDomain:1149) Restore failed
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/xen/xend/XendDomain.py", line 1147, in domain_restore_fd
    return XendCheckpoint.restore(self, fd, paused=paused, relocating=relocating)
  File "/usr/lib/python2.4/site-packages/xen/xend/XendCheckpoint.py", line 282, in restore
    forkHelper(cmd, fd, handler.handler, True)
  File "/usr/lib/python2.4/site-packages/xen/xend/XendCheckpoint.py", line 406, in forkHelper
    raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib/xen/bin/xc_restore 31 2 2 3 1 1 1 failed
[2009-08-26 02:08:04 2883] DEBUG (XendDomainInfo:93) XendDomainInfo.create(['vm', ['name', 'Windows_Server2008'], ['memory', '600'], ['on_reboot', 'restart'], ['on_crash', 'restart'], ['vcpus', 1], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['bootloader', '/usr/bin/pygrub'], ['bootloader_args', '-q'], ['image', ['hvm', ['kernel', '/usr/lib/xen/boot/hvmloader'], ['videoram', 4], ['device_model', '/usr/lib/xen/bin/qemu-dm'], ['pae', 1], ['vcpus', 1], ['boot', 'dc'], ['fda', ''], ['fdb', ''], ['timer_mode', 1], ['localtime', 0], ['serial', 'pty'], ['stdvga', 0], ['isa', 0], ['nographic', 0], ['soundhw', ''], ['vnc', 1], ['vncunused', 1], ['vnclisten', '0.0.0.0'], ['xauthority', '/root/.Xauthority'], ['rtc_timeoffset', 0], ['monitor', 0], ['acpi', 1], ['apic', 1], ['usb', 0], ['usbdevice', 'tablet'], ['keymap', ''], ['pci', []], ['hpet', 0], ['guest_os_type', 'default'], ['hap', 1], ['cpuid', []], ['cpuid_check', []], ['viridian', 0], ['pci_msitranslate', 1], ['vpt_align', 1], ['pci_power_mgmt', 0], ['xen_platform_pci', 1], ['vncpasswd', 'XXXXXXXX']]], ['s3_integrity', 1], ['device', ['vbd', ['uname', 'file:/mnt/data/vm/srv2008/System.img'], ['dev', 'hda'], ['mode', 'w']]], ['device', ['vif', ['bridge', 'xenbr0']]]])
[2009-08-26 02:08:04 2883] DEBUG (XendDomainInfo:2317) XendDomainInfo.constructDomain
[2009-08-26 02:08:04 2883] DEBUG (balloon:181) Balloon: 1472776 KiB free; need 4096; done.
[2009-08-26 02:08:04 2883] DEBUG (XendDomain:452) Adding Domain: 3
[2009-08-26 02:08:04 2883] DEBUG (XendDomainInfo:2518) XendDomainInfo.initDomain: 3 256
[2009-08-26 02:08:04 2883] DEBUG (image:319) No VNC passwd configured for vfb access
[2009-08-26 02:08:04 2883] DEBUG (image:806) args: boot, val: dc
[2009-08-26 02:08:04 2883] DEBUG (image:806) args: fda, val: None
[2009-08-26 02:08:04 2883] DEBUG (image:806) args: fdb, val: None
[2009-08-26 02:08:04 2883] DEBUG (image:806) args: soundhw, val: None
[2009-08-26 02:08:04 2883] DEBUG (image:806) args: localtime, val: 0
[2009-08-26 02:08:04 2883] DEBUG (image:806) args: serial, val: ['pty']
[2009-08-26 02:08:04 2883] DEBUG (image:806) args: std-vga, val: 0
[2009-08-26 02:08:04 2883] DEBUG (image:806) args: isa, val: 0
[2009-08-26 02:08:04 2883] DEBUG (image:806) args: acpi, val: 1
[2009-08-26 02:08:04 2883] DEBUG (image:806) args: usb, val: 0
[2009-08-26 02:08:04 2883] DEBUG (image:806) args: usbdevice, val: tablet
[2009-08-26 02:08:04 2883] INFO (image:742) Need to create platform device.[domid:3]
[2009-08-26 02:08:04 2883] DEBUG (XendDomainInfo:2542) _initDomain:shadow_memory=0x0, memory_static_max=0x25800000, memory_static_min=0x0.
[2009-08-26 02:08:04 2883] DEBUG (balloon:181) Balloon: 1471364 KiB free; need 627712; done.
[2009-08-26 02:08:04 2883] INFO (image:170) buildDomain os=hvm dom=3 vcpus=1
[2009-08-26 02:08:04 2883] DEBUG (image:859) domid          = 3
[2009-08-26 02:08:04 2883] DEBUG (image:860) image          = /usr/lib/xen/boot/hvmloader
[2009-08-26 02:08:04 2883] DEBUG (image:861) store_evtchn   = 2
[2009-08-26 02:08:04 2883] DEBUG (image:862) memsize        = 600
[2009-08-26 02:08:04 2883] DEBUG (image:863) target         = 600
[2009-08-26 02:08:04 2883] DEBUG (image:864) vcpus          = 1
[2009-08-26 02:08:04 2883] DEBUG (image:865) acpi           = 1
[2009-08-26 02:08:04 2883] DEBUG (image:866) apic           = 1
[2009-08-26 02:08:04 2883] INFO (XendDomainInfo:2178) createDevice: vfb : {'vncunused': 1, 'vnclisten': '0.0.0.0', 'vnc': '1', 'uuid': 'a34d2d0b-8bff-5bbc-70fc-fa3b5c6f8262', 'other_config': {'vncunused': 1, 'vnclisten': '0.0.0.0', 'vnc': '1'}}
[2009-08-26 02:08:04 2883] DEBUG (DevController:95) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/vfb/3/0'} to /local/domain/3/device/vfb/0.
[2009-08-26 02:08:04 2883] DEBUG (DevController:97) DevController: writing {'vncunused': '1', 'domain': 'Windows_Server2008', 'frontend': '/local/domain/3/device/vfb/0', 'uuid': 'a34d2d0b-8bff-5bbc-70fc-fa3b5c6f8262', 'frontend-id': '3', 'vnclisten': '0.0.0.0', 'state': '1', 'online': '1', 'vnc': '1'} to /local/domain/0/backend/vfb/3/0.
[2009-08-26 02:08:04 2883] INFO (XendDomainInfo:2178) createDevice: vbd : {'uuid': 'ed0540fc-6021-53fe-75e8-b2e72cd335e2', 'bootable': 1, 'driver': 'paravirtualised', 'dev': 'hda', 'uname': 'file:/mnt/data/vm/srv2008/System.img', 'mode': 'w'}
[2009-08-26 02:08:04 2883] DEBUG (DevController:95) DevController: writing {'backend-id': '0', 'virtual-device': '768', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/3/768'} to /local/domain/3/device/vbd/768.
[2009-08-26 02:08:04 2883] DEBUG (DevController:97) DevController: writing {'domain': 'Windows_Server2008', 'frontend': '/local/domain/3/device/vbd/768', 'uuid': 'ed0540fc-6021-53fe-75e8-b2e72cd335e2', 'bootable': '1', 'dev': 'hda', 'state': '1', 'params': '/mnt/data/vm/srv2008/System.img', 'mode': 'w', 'online': '1', 'frontend-id': '3', 'type': 'file'} to /local/domain/0/backend/vbd/3/768.
[2009-08-26 02:08:04 2883] INFO (XendDomainInfo:2178) createDevice: vif : {'bridge': 'xenbr0', 'mac': '00:16:3e:21:0f:8e', 'uuid': 'c1729c89-6ca7-04bb-1eaa-225c0144d5fd'}
[2009-08-26 02:08:04 2883] DEBUG (DevController:95) DevController: writing {'backend-id': '0', 'mac': '00:16:3e:21:0f:8e', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/3/0'} to /local/domain/3/device/vif/0.
[2009-08-26 02:08:04 2883] DEBUG (DevController:97) DevController: writing {'bridge': 'xenbr0', 'domain': 'Windows_Server2008', 'handle': '0', 'uuid': 'c1729c89-6ca7-04bb-1eaa-225c0144d5fd', 'script': '/etc/xen/scripts/vif-bridge', 'mac': '00:16:3e:21:0f:8e', 'frontend-id': '3', 'state': '1', 'online': '1', 'frontend': '/local/domain/3/device/vif/0'} to /local/domain/0/backend/vif/3/0.
[2009-08-26 02:08:04 2883] INFO (image:391) spawning device models: /usr/lib/xen/bin/qemu-dm ['/usr/lib/xen/bin/qemu-dm', '-d', '3', '-domain-name', 'Windows_Server2008', '-videoram', '4', '-vnc', '0.0.0.0:0', '-vncunused', '-vcpus', '1', '-boot', 'dc', '-serial', 'pty', '-acpi', '-usbdevice', 'tablet', '-net', 'nic,vlan=1,macaddr=00:16:3e:21:0f:8e,model=rtl8139', '-net', 'tap,vlan=1,ifname=tap3.0,bridge=xenbr0', '-M', 'xenfv']
[2009-08-26 02:08:04 2883] INFO (image:440) device model pid: 6650
[2009-08-26 02:08:05 2883] INFO (image:528) waiting for sentinel_fifo
[2009-08-26 02:08:05 2883] DEBUG (XendDomainInfo:3082) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '6', 'uuid': '67f27bcf-96d8-a365-0f5a-d683bd36e281', 'on_reboot': 'restart', 'start_time': '1251223685.0', 'on_poweroff': 'destroy', 'bootloader_args': '-q', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'bootloader': '/usr/bin/pygrub', 'image': '(hvm (kernel ) (videoram 4) (hpet 0) (stdvga 0) (vnclisten 0.0.0.0) (loader /usr/lib/xen/boot/hvmloader) (serial pty) (vncunused 1) (xen_platform_pci 1) (boot dc) (rtc_timeoffset 0) (pci ()) (pae 1) (vpt_align 1) (hap 1) (viridian 0) (acpi 1) (localtime 0) (timer_mode 1) (vnc 1) (nographic 0) (guest_os_type default) (pci_msitranslate 1) (apic 1) (monitor 0) (usbdevice tablet) (device_model /usr/lib/xen/bin/qemu-dm) (pci_power_mgmt 0) (usb 0) (xauthority /root/.Xauthority) (isa 0) (notes (SUSPEND_CANCEL 1)))', 'name': 'Windows_Server2008'}
[2009-08-26 02:08:05 2883] DEBUG (XendDomainInfo:1638) Storing domain details: {'console/port': '3', 'name': 'Windows_Server2008', 'console/limit': '1048576', 'store/port': '2', 'vm': '/vm/67f27bcf-96d8-a365-0f5a-d683bd36e281', 'domid': '3', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'memory/target': '614400', 'control/platform-feature-multiprocessor-suspend': '1', 'store/ring-ref': '1044476', 'console/type': 'ioemu'}
[2009-08-26 02:08:05 2883] DEBUG (DevController:95) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/3/0'} to /local/domain/3/device/console/0.
[2009-08-26 02:08:05 2883] DEBUG (DevController:97) DevController: writing {'domain': 'Windows_Server2008', 'frontend': '/local/domain/3/device/console/0', 'uuid': '619b411b-be8b-ccd1-07ed-e8116969eb18', 'frontend-id': '3', 'state': '1', 'location': '3', 'online': '1', 'protocol': 'vt100'} to /local/domain/0/backend/console/3/0.
[2009-08-26 02:08:05 2883] DEBUG (XendDomainInfo:1725) XendDomainInfo.handleShutdownWatch
[2009-08-26 02:08:05 2883] DEBUG (DevController:139) Waiting for devices vif.
[2009-08-26 02:08:05 2883] DEBUG (DevController:144) Waiting for 0.
[2009-08-26 02:08:05 2883] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vif/3/0/hotplug-status.
[2009-08-26 02:08:06 2883] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vif/3/0/hotplug-status.
[2009-08-26 02:08:06 2883] DEBUG (DevController:643) hotplugStatusCallback 1.
[2009-08-26 02:08:06 2883] DEBUG (DevController:139) Waiting for devices vscsi.
[2009-08-26 02:08:06 2883] DEBUG (DevController:139) Waiting for devices vbd.
[2009-08-26 02:08:06 2883] DEBUG (DevController:144) Waiting for 768.
[2009-08-26 02:08:06 2883] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vbd/3/768/hotplug-status.
[2009-08-26 02:08:06 2883] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vbd/3/768/hotplug-status.
[2009-08-26 02:08:06 2883] DEBUG (DevController:643) hotplugStatusCallback 1.
[2009-08-26 02:08:06 2883] DEBUG (DevController:139) Waiting for devices irq.
[2009-08-26 02:08:06 2883] DEBUG (DevController:139) Waiting for devices vkbd.
[2009-08-26 02:08:06 2883] DEBUG (DevController:139) Waiting for devices vfb.
[2009-08-26 02:08:06 2883] DEBUG (DevController:139) Waiting for devices console.
[2009-08-26 02:08:06 2883] DEBUG (DevController:144) Waiting for 0.
[2009-08-26 02:08:06 2883] DEBUG (DevController:139) Waiting for devices pci.
[2009-08-26 02:08:06 2883] DEBUG (DevController:139) Waiting for devices ioports.
[2009-08-26 02:08:06 2883] DEBUG (DevController:139) Waiting for devices tap.
[2009-08-26 02:08:06 2883] DEBUG (DevController:139) Waiting for devices vtpm.
[2009-08-26 02:08:06 2883] INFO (XendDomain:1180) Domain Windows_Server2008 (3) unpaused.
[2009-08-26 02:10:47 2883] DEBUG (XendCheckpoint:110) [xc_save]: /usr/lib/xen/bin/xc_save 31 3 0 0 4
[2009-08-26 02:10:47 2883] INFO (XendCheckpoint:418) xc_save: failed to get the suspend evtchn port
[2009-08-26 02:10:47 2883] INFO (XendCheckpoint:418) 
[2009-08-26 02:10:47 2883] DEBUG (XendCheckpoint:389) suspend
[2009-08-26 02:10:47 2883] DEBUG (XendCheckpoint:113) In saveInputHandler suspend
[2009-08-26 02:10:47 2883] DEBUG (XendCheckpoint:115) Suspending 3 ...
[2009-08-26 02:10:47 2883] DEBUG (XendDomainInfo:518) XendDomainInfo.shutdown(suspend)
[2009-08-26 02:10:47 2883] INFO (XendCheckpoint:418) -->xc_domain_save()
[2009-08-26 02:10:47 2883] INFO (XendCheckpoint:418) platform info, max_mfn: 7d4b0, hvirt_start ffc00000, pt_lvevel 4, guest_width 8
[2009-08-26 02:10:47 2883] INFO (XendCheckpoint:418) shared_info_frame: 0xfffff
[2009-08-26 02:10:47 2883] DEBUG (XendDomainInfo:1725) XendDomainInfo.handleShutdownWatch
[2009-08-26 02:10:47 2883] INFO (XendDomainInfo:535) HVM save:remote shutdown dom 3!
[2009-08-26 02:10:47 2883] INFO (XendCheckpoint:121) Domain 3 suspended.
[2009-08-26 02:10:47 2883] INFO (XendDomainInfo:1913) Domain has shutdown: name=migrating-Windows_Server2008 id=3 reason=suspend.
[2009-08-26 02:10:47 2883] INFO (image:476) signalDeviceModel:restore dm state to running
[2009-08-26 02:10:47 2883] DEBUG (XendCheckpoint:130) Written done
[2009-08-26 02:10:47 2883] INFO (XendCheckpoint:418) p2m_size: 0x100000Need another buffer for HVM context
[2009-08-26 02:10:47 2883] INFO (XendCheckpoint:418) Setup the mfn_to_pfn table mapping
[2009-08-26 02:10:47 2883] INFO (XendCheckpoint:418) Start writing out the saved-domain record.
[2009-08-26 02:10:55 2883] INFO (XendCheckpoint:418) Saving memory pages: iter 1   0%\b\b\b\b  5%\b\b\b\b 10%\b\b\b\b 15%\b\b\b\b 20%\b\b\b\b 25%\b\b\b\b 30%\b\b\b\b 35%\b\b\b\b 40%\b\b\b\b 45%\b\b\b\b 50%\b\b\b\b 55%\b\b\b\b 60%\b\b\b\b 65%\b\b\b\b 70%\b\b\b\b 75%\b\b\b\b 80%\b\b\b\b 85%\b\b\b\b 90%\b\b\b\b 95%\r 1: sent 157696, skipped 0, delta 8022ms, dom0 50%, target 0%, sent 644Mb/s, dirtied 0Mb/s 0 pages
[2009-08-26 02:10:55 2883] INFO (XendCheckpoint:418) Total pages sent= 157696 (0.15x)
[2009-08-26 02:10:55 2883] INFO (XendCheckpoint:418) (of which 0 were fixups)
[2009-08-26 02:10:55 2883] INFO (XendCheckpoint:418) All memory is saved
[2009-08-26 02:10:56 2883] INFO (XendCheckpoint:418) Save exit rc=0
[2009-08-26 02:10:56 2883] DEBUG (XendDomainInfo:2750) XendDomainInfo.destroy: domid=3
[2009-08-26 02:10:56 2883] DEBUG (XendDomainInfo:2220) Destroying device model
[2009-08-26 02:10:56 2883] DEBUG (XendDomainInfo:2227) Releasing devices
[2009-08-26 02:10:56 2883] DEBUG (XendDomainInfo:2240) Removing vif/0
[2009-08-26 02:10:56 2883] DEBUG (XendDomainInfo:1142) XendDomainInfo.destroyDevice: deviceClass = vif, device = vif/0
[2009-08-26 02:10:56 2883] INFO (image:553) migrating-Windows_Server2008 device model terminated
[2009-08-26 02:10:56 2883] DEBUG (XendDomainInfo:2240) Removing vbd/768
[2009-08-26 02:10:56 2883] DEBUG (XendDomainInfo:1142) XendDomainInfo.destroyDevice: deviceClass = vbd, device = vbd/768
[2009-08-26 02:10:56 2883] DEBUG (XendDomainInfo:2240) Removing vfb/0
[2009-08-26 02:10:56 2883] DEBUG (XendDomainInfo:1142) XendDomainInfo.destroyDevice: deviceClass = vfb, device = vfb/0
[2009-08-26 02:10:56 2883] DEBUG (XendDomainInfo:2240) Removing console/0
[2009-08-26 02:10:56 2883] DEBUG (XendDomainInfo:1142) XendDomainInfo.destroyDevice: deviceClass = console, device = console/0
[2009-08-26 02:11:01 2883] DEBUG (XendDomainInfo:227) XendDomainInfo.restore(['domain', ['domid', '3'], ['on_crash', 'restart'], ['uuid', '67f27bcf-96d8-a365-0f5a-d683bd36e281'], ['bootloader_args', '-q'], ['vcpus', '1'], ['name', 'Windows_Server2008'], ['on_poweroff', 'destroy'], ['on_reboot', 'restart'], ['cpus', [[]]], ['bootloader', '/usr/bin/pygrub'], ['maxmem', '600'], ['memory', '600'], ['shadow_memory', '6'], ['vcpu_avail', '1'], ['features'], ['on_xend_start', 'ignore'], ['on_xend_stop', 'ignore'], ['start_time', '1251223685.0'], ['cpu_time', '0.0'], ['online_vcpus', '1'], ['image', ['hvm', ['kernel'], ['videoram', '4'], ['hpet', '0'], ['stdvga', '0'], ['vnclisten', '0.0.0.0'], ['loader', '/usr/lib/xen/boot/hvmloader'], ['serial', 'pty'], ['vncunused', '1'], ['xen_platform_pci', '1'], ['boot', 'dc'], ['rtc_timeoffset', '0'], ['pci', []], ['pae', '1'], ['vpt_align', '1'], ['hap', '1'], ['viridian', '0'], ['acpi', '1'], ['localtime', '0'], ['timer_mode', '1'], ['vnc', '1'], ['nographic', '0'], ['guest_os_type', 'default'], ['pci_msitranslate', '1'], ['apic', '1'], ['monitor', '0'], ['usbdevice', 'tablet'], ['device_model', '/usr/lib/xen/bin/qemu-dm'], ['pci_power_mgmt', '0'], ['usb', '0'], ['xauthority', '/root/.Xauthority'], ['isa', '0'], ['notes', ['SUSPEND_CANCEL', '1']]]], ['status', '2'], ['state', '--p---'], ['store_mfn', '1044476'], ['device', ['vif', ['bridge', 'xenbr0'], ['mac', '00:16:3e:21:0f:8e'], ['script', '/etc/xen/scripts/vif-bridge'], ['uuid', 'c1729c89-6ca7-04bb-1eaa-225c0144d5fd'], ['backend', '0']]], ['device', ['vbd', ['uuid', 'ed0540fc-6021-53fe-75e8-b2e72cd335e2'], ['bootable', '1'], ['dev', 'hda:disk'], ['uname', 'file:/mnt/data/vm/srv2008/System.img'], ['mode', 'w'], ['backend', '0'], ['bootable', '1'], ['VDI']]], ['device', ['vfb', ['vncunused', '1'], ['vnclisten', '0.0.0.0'], ['vnc', '1'], ['uuid', 'a34d2d0b-8bff-5bbc-70fc-fa3b5c6f8262']]], ['device', ['console', ['protocol', 'vt100'], ['location', '3'], ['uuid', '619b411b-be8b-ccd1-07ed-e8116969eb18']]]])
[2009-08-26 02:11:01 2883] DEBUG (XendDomainInfo:2317) XendDomainInfo.constructDomain
[2009-08-26 02:11:01 2883] DEBUG (balloon:181) Balloon: 1472776 KiB free; need 4096; done.
[2009-08-26 02:11:01 2883] DEBUG (XendDomain:452) Adding Domain: 4
[2009-08-26 02:11:01 2883] DEBUG (XendDomainInfo:3082) Storing VM details: {'on_xend_stop': 'ignore', 'shadow_memory': '6', 'uuid': '67f27bcf-96d8-a365-0f5a-d683bd36e281', 'on_reboot': 'restart', 'start_time': '1251223685.0', 'on_poweroff': 'destroy', 'bootloader_args': '-q', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '1', 'vcpu_avail': '1', 'bootloader': '/usr/bin/pygrub', 'image': '(hvm (kernel ) (videoram 4) (hpet 0) (stdvga 0) (vnclisten 0.0.0.0) (loader /usr/lib/xen/boot/hvmloader) (serial pty) (vncunused 1) (xen_platform_pci 1) (boot dc) (rtc_timeoffset 0) (pci ()) (pae 1) (vpt_align 1) (hap 1) (viridian 0) (acpi 1) (localtime 0) (timer_mode 1) (vnc 1) (nographic 0) (guest_os_type default) (pci_msitranslate 1) (apic 1) (monitor 0) (usbdevice tablet) (device_model /usr/lib/xen/bin/qemu-dm) (pci_power_mgmt 0) (usb 0) (xauthority /root/.Xauthority) (isa 0) (notes (SUSPEND_CANCEL 1)))', 'name': 'Windows_Server2008'}
[2009-08-26 02:11:01 2883] INFO (XendDomainInfo:2178) createDevice: vfb : {'vncunused': '1', 'vnclisten': '0.0.0.0', 'vnc': '1', 'uuid': 'a34d2d0b-8bff-5bbc-70fc-fa3b5c6f8262', 'other_config': {'vncunused': '1', 'vnclisten': '0.0.0.0', 'vnc': '1'}}
[2009-08-26 02:11:01 2883] DEBUG (DevController:95) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/vfb/4/0'} to /local/domain/4/device/vfb/0.
[2009-08-26 02:11:01 2883] DEBUG (DevController:97) DevController: writing {'vncunused': '1', 'domain': 'Windows_Server2008', 'frontend': '/local/domain/4/device/vfb/0', 'uuid': 'a34d2d0b-8bff-5bbc-70fc-fa3b5c6f8262', 'frontend-id': '4', 'vnclisten': '0.0.0.0', 'state': '1', 'online': '1', 'vnc': '1'} to /local/domain/0/backend/vfb/4/0.
[2009-08-26 02:11:01 2883] INFO (XendDomainInfo:2178) createDevice: console : {'protocol': 'vt100', 'location': '3', 'uuid': '619b411b-be8b-ccd1-07ed-e8116969eb18'}
[2009-08-26 02:11:01 2883] DEBUG (DevController:95) DevController: writing {'state': '1', 'backend-id': '0', 'backend': '/local/domain/0/backend/console/4/0'} to /local/domain/4/device/console/0.
[2009-08-26 02:11:01 2883] DEBUG (DevController:97) DevController: writing {'domain': 'Windows_Server2008', 'frontend': '/local/domain/4/device/console/0', 'uuid': '619b411b-be8b-ccd1-07ed-e8116969eb18', 'frontend-id': '4', 'state': '1', 'location': '3', 'online': '1', 'protocol': 'vt100'} to /local/domain/0/backend/console/4/0.
[2009-08-26 02:11:01 2883] INFO (XendDomainInfo:2178) createDevice: vbd : {'uuid': 'ed0540fc-6021-53fe-75e8-b2e72cd335e2', 'bootable': '1', 'driver': 'paravirtualised', 'dev': 'hda:disk', 'uname': 'file:/mnt/data/vm/srv2008/System.img', 'mode': 'w', 'backend': '0'}
[2009-08-26 02:11:01 2883] DEBUG (DevController:95) DevController: writing {'backend-id': '0', 'virtual-device': '768', 'device-type': 'disk', 'state': '1', 'backend': '/local/domain/0/backend/vbd/4/768'} to /local/domain/4/device/vbd/768.
[2009-08-26 02:11:01 2883] DEBUG (DevController:97) DevController: writing {'domain': 'Windows_Server2008', 'frontend': '/local/domain/4/device/vbd/768', 'uuid': 'ed0540fc-6021-53fe-75e8-b2e72cd335e2', 'bootable': '1', 'dev': 'hda', 'state': '1', 'params': '/mnt/data/vm/srv2008/System.img', 'mode': 'w', 'online': '1', 'frontend-id': '4', 'type': 'file'} to /local/domain/0/backend/vbd/4/768.
[2009-08-26 02:11:01 2883] INFO (XendDomainInfo:2178) createDevice: vif : {'bridge': 'xenbr0', 'mac': '00:16:3e:21:0f:8e', 'script': '/etc/xen/scripts/vif-bridge', 'uuid': 'c1729c89-6ca7-04bb-1eaa-225c0144d5fd', 'backend': '0'}
[2009-08-26 02:11:01 2883] DEBUG (DevController:95) DevController: writing {'backend-id': '0', 'mac': '00:16:3e:21:0f:8e', 'handle': '0', 'state': '1', 'backend': '/local/domain/0/backend/vif/4/0'} to /local/domain/4/device/vif/0.
[2009-08-26 02:11:01 2883] DEBUG (DevController:97) DevController: writing {'bridge': 'xenbr0', 'domain': 'Windows_Server2008', 'handle': '0', 'uuid': 'c1729c89-6ca7-04bb-1eaa-225c0144d5fd', 'script': '/etc/xen/scripts/vif-bridge', 'mac': '00:16:3e:21:0f:8e', 'frontend-id': '4', 'state': '1', 'online': '1', 'frontend': '/local/domain/4/device/vif/0'} to /local/domain/0/backend/vif/4/0.
[2009-08-26 02:11:01 2883] DEBUG (XendDomainInfo:1638) Storing domain details: {'console/port': '3', 'name': 'Windows_Server2008', 'console/limit': '1048576', 'image/suspend-cancel': '1', 'domid': '4', 'vm': '/vm/67f27bcf-96d8-a365-0f5a-d683bd36e281', 'cpu/0/availability': 'online', 'memory/target': '614400', 'control/platform-feature-multiprocessor-suspend': '1', 'console/type': 'ioemu', 'store/port': '2'}
[2009-08-26 02:11:01 2883] INFO (XendCheckpoint:243) restore hvm domain 4, apic=1, pae=1
[2009-08-26 02:11:01 2883] DEBUG (image:319) No VNC passwd configured for vfb access
[2009-08-26 02:11:01 2883] DEBUG (image:806) args: boot, val: dc
[2009-08-26 02:11:01 2883] DEBUG (image:806) args: fda, val: None
[2009-08-26 02:11:01 2883] DEBUG (image:806) args: fdb, val: None
[2009-08-26 02:11:01 2883] DEBUG (image:806) args: soundhw, val: None
[2009-08-26 02:11:01 2883] DEBUG (image:806) args: localtime, val: 0
[2009-08-26 02:11:01 2883] DEBUG (image:806) args: serial, val: ['pty']
[2009-08-26 02:11:01 2883] DEBUG (image:806) args: std-vga, val: 0
[2009-08-26 02:11:01 2883] DEBUG (image:806) args: isa, val: 0
[2009-08-26 02:11:01 2883] DEBUG (image:806) args: acpi, val: 1
[2009-08-26 02:11:01 2883] DEBUG (image:806) args: usb, val: 0
[2009-08-26 02:11:01 2883] DEBUG (image:806) args: usbdevice, val: tablet
[2009-08-26 02:11:01 2883] INFO (image:742) Need to create platform device.[domid:4]
[2009-08-26 02:11:01 2883] DEBUG (XendCheckpoint:261) restore:shadow=0x6, _static_max=0x25800000, _static_min=0x0, 
[2009-08-26 02:11:01 2883] DEBUG (balloon:181) Balloon: 1471364 KiB free; need 624640; done.
[2009-08-26 02:11:01 2883] DEBUG (XendCheckpoint:278) [xc_restore]: /usr/lib/xen/bin/xc_restore 31 4 2 3 1 1 1
[2009-08-26 02:11:01 2883] INFO (XendCheckpoint:418) -->xc_domain_restore()
[2009-08-26 02:11:01 2883] INFO (XendCheckpoint:418) xc_domain_restore start: p2m_size = 100000
[2009-08-26 02:11:01 2883] INFO (XendCheckpoint:418) platform info, max_mfn: 7d4b0, hvirt_start ffc00000, pt_lvevel 4, guest_width 8
[2009-08-26 02:11:01 2883] INFO (XendCheckpoint:418) memory alloc success
[2009-08-26 02:11:01 2883] INFO (XendCheckpoint:418) lock regin_mfn and p2m_batch success
[2009-08-26 02:11:01 2883] INFO (XendCheckpoint:418) shared_info_frame: 0xffffffff
[2009-08-26 02:11:01 2883] INFO (XendCheckpoint:418) all PFNs are marked as invalid
[2009-08-26 02:11:01 2883] INFO (XendCheckpoint:418) Reloading memory pages:   0%
[2009-08-26 02:11:01 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:01 2883] INFO (XendCheckpoint:418) nr_mfns: 992
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:02 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:03 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:04 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:05 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:06 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:07 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:08 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 7
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 4
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 1024
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) nr_mfns: 1
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch -2
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch -3
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch -4
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) batch 0
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) Received all pages (0 races)
[2009-08-26 02:11:09 2883] INFO (XendCheckpoint:418) Restore exit with rc=0
[2009-08-26 02:11:09 2883] DEBUG (XendCheckpoint:389) store-mfn 1044476
[2009-08-26 02:11:09 2883] DEBUG (XendDomainInfo:2688) XendDomainInfo.completeRestore
[2009-08-26 02:11:09 2883] DEBUG (image:319) No VNC passwd configured for vfb access
[2009-08-26 02:11:09 2883] DEBUG (image:806) args: boot, val: dc
[2009-08-26 02:11:09 2883] DEBUG (image:806) args: fda, val: None
[2009-08-26 02:11:09 2883] DEBUG (image:806) args: fdb, val: None
[2009-08-26 02:11:09 2883] DEBUG (image:806) args: soundhw, val: None
[2009-08-26 02:11:09 2883] DEBUG (image:806) args: localtime, val: 0
[2009-08-26 02:11:09 2883] DEBUG (image:806) args: serial, val: ['pty']
[2009-08-26 02:11:09 2883] DEBUG (image:806) args: std-vga, val: 0
[2009-08-26 02:11:09 2883] DEBUG (image:806) args: isa, val: 0
[2009-08-26 02:11:09 2883] DEBUG (image:806) args: acpi, val: 1
[2009-08-26 02:11:09 2883] DEBUG (image:806) args: usb, val: 0
[2009-08-26 02:11:09 2883] DEBUG (image:806) args: usbdevice, val: tablet
[2009-08-26 02:11:10 2883] INFO (image:742) Need to create platform device.[domid:4]
[2009-08-26 02:11:10 2883] INFO (image:391) spawning device models: /usr/lib/xen/bin/qemu-dm ['/usr/lib/xen/bin/qemu-dm', '-d', '4', '-domain-name', 'Windows_Server2008', '-videoram', '4', '-vnc', '0.0.0.0:0', '-vncunused', '-vcpus', '1', '-boot', 'dc', '-serial', 'pty', '-acpi', '-usbdevice', 'tablet', '-net', 'nic,vlan=1,macaddr=00:16:3e:21:0f:8e,model=rtl8139', '-net', 'tap,vlan=1,ifname=tap4.0,bridge=xenbr0', '-M', 'xenfv', '-loadvm', '/var/lib/xen/qemu-save.4']
[2009-08-26 02:11:10 2883] INFO (image:440) device model pid: 8109
[2009-08-26 02:11:10 2883] DEBUG (XendDomainInfo:1638) Storing domain details: {'console/port': '3', 'name': 'Windows_Server2008', 'console/limit': '1048576', 'store/port': '2', 'vm': '/vm/67f27bcf-96d8-a365-0f5a-d683bd36e281', 'domid': '4', 'image/suspend-cancel': '1', 'cpu/0/availability': 'online', 'memory/target': '614400', 'control/platform-feature-multiprocessor-suspend': '1', 'store/ring-ref': '1044476', 'console/type': 'ioemu'}
[2009-08-26 02:11:10 2883] INFO (image:528) waiting for sentinel_fifo
[2009-08-26 02:11:10 2883] DEBUG (XendDomainInfo:2701) XendDomainInfo.completeRestore done
[2009-08-26 02:11:10 2883] DEBUG (DevController:139) Waiting for devices vif.
[2009-08-26 02:11:10 2883] DEBUG (XendDomainInfo:1725) XendDomainInfo.handleShutdownWatch
[2009-08-26 02:11:10 2883] DEBUG (DevController:144) Waiting for 0.
[2009-08-26 02:11:10 2883] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vif/4/0/hotplug-status.
[2009-08-26 02:11:10 2883] DEBUG (DevController:643) hotplugStatusCallback 1.
[2009-08-26 02:11:10 2883] DEBUG (DevController:139) Waiting for devices vscsi.
[2009-08-26 02:11:10 2883] DEBUG (DevController:139) Waiting for devices vbd.
[2009-08-26 02:11:10 2883] DEBUG (DevController:144) Waiting for 768.
[2009-08-26 02:11:10 2883] DEBUG (DevController:629) hotplugStatusCallback /local/domain/0/backend/vbd/4/768/hotplug-status.
[2009-08-26 02:11:10 2883] DEBUG (DevController:643) hotplugStatusCallback 1.
[2009-08-26 02:11:10 2883] DEBUG (DevController:139) Waiting for devices irq.
[2009-08-26 02:11:10 2883] DEBUG (DevController:139) Waiting for devices vkbd.
[2009-08-26 02:11:10 2883] DEBUG (DevController:139) Waiting for devices vfb.
[2009-08-26 02:11:10 2883] DEBUG (DevController:139) Waiting for devices console.
[2009-08-26 02:11:10 2883] DEBUG (DevController:144) Waiting for 0.
[2009-08-26 02:11:10 2883] DEBUG (DevController:139) Waiting for devices pci.
[2009-08-26 02:11:10 2883] DEBUG (DevController:139) Waiting for devices ioports.
[2009-08-26 02:11:10 2883] DEBUG (DevController:139) Waiting for devices tap.
[2009-08-26 02:11:10 2883] DEBUG (DevController:139) Waiting for devices vtpm.

[-- Attachment #3: xc_domain_restore.c --]
[-- Type: text/plain, Size: 40832 bytes --]

/******************************************************************************
 * xc_domain_restore.c
 *
 * Restore the state of a guest session.
 *
 * Copyright (c) 2003, K A Fraser.
 * Copyright (c) 2006, Intel Corporation
 * Copyright (c) 2007, XenSource Inc.
 *
 * This program is free software; you can redistribute it and/or modify it
 * under the terms and conditions of the GNU General Public License,
 * version 2, as published by the Free Software Foundation.
 *
 * This program is distributed in the hope it will be useful, but WITHOUT
 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
 * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
 * more details.
 *
 * You should have received a copy of the GNU General Public License along with
 * this program; if not, write to the Free Software Foundation, Inc., 59 Temple
 * Place - Suite 330, Boston, MA 02111-1307 USA.
 *
 */

#include <stdlib.h>
#include <unistd.h>

#include "xg_private.h"
#include "xg_save_restore.h"
#include "xc_dom.h"

#include <xen/hvm/ioreq.h>
#include <xen/hvm/params.h>

/* max mfn of the current host machine */
static unsigned long max_mfn;

/* virtual starting address of the hypervisor */
static unsigned long hvirt_start;

/* #levels of page tables used by the current guest */
static unsigned int pt_levels;

/* number of pfns this guest has (i.e. number of entries in the P2M) */
static unsigned long p2m_size;

/* number of 'in use' pfns in the guest (i.e. #P2M entries with a valid mfn) */
static unsigned long nr_pfns;

/* Live mapping of the table mapping each PFN to its current MFN. */
static xen_pfn_t *live_p2m = NULL;

/* A table mapping each PFN to its new MFN. */
static xen_pfn_t *p2m = NULL;

/* A table of P2M mappings in the current region */
static xen_pfn_t *p2m_batch = NULL;

/* Address size of the guest, in bytes */
unsigned int guest_width;

/*
** In the state file (or during transfer), all page-table pages are
** converted into a 'canonical' form where references to actual mfns
** are replaced with references to the corresponding pfns.
** This function inverts that operation, replacing the pfn values with
** the (now known) appropriate mfn values.
*/
static int uncanonicalize_pagetable(int xc_handle, uint32_t dom, 
                                    unsigned long type, void *page)
{
    int i, pte_last;
    unsigned long pfn;
    uint64_t pte;
    int nr_mfns = 0; 

    pte_last = PAGE_SIZE / ((pt_levels == 2)? 4 : 8);

    /* First pass: work out how many (if any) MFNs we need to alloc */
    for ( i = 0; i < pte_last; i++ )
    {
        if ( pt_levels == 2 )
            pte = ((uint32_t *)page)[i];
        else
            pte = ((uint64_t *)page)[i];

        /* XXX SMH: below needs fixing for PROT_NONE etc */
        if ( !(pte & _PAGE_PRESENT) )
            continue;
        
        pfn = (pte >> PAGE_SHIFT) & MFN_MASK_X86;
        
        if ( pfn >= p2m_size )
        {
            /* This "page table page" is probably not one; bail. */
            ERROR("Frame number in type %lu page table is out of range: "
                  "i=%d pfn=0x%lx p2m_size=%lu",
                  type >> 28, i, pfn, p2m_size);
            return 0;
        }
        
        if ( p2m[pfn] == INVALID_P2M_ENTRY )
        {
            /* Have a 'valid' PFN without a matching MFN - need to alloc */
            p2m_batch[nr_mfns++] = pfn; 
            p2m[pfn]--;
        }
    }

    /* Allocate the requisite number of mfns. */
    if ( nr_mfns &&
         (xc_domain_memory_populate_physmap(xc_handle, dom, nr_mfns, 0, 0,
                                            p2m_batch) != 0) )
    { 
        ERROR("Failed to allocate memory for batch.!\n"); 
        errno = ENOMEM;
        return 0; 
    }
    
    /* Second pass: uncanonicalize each present PTE */
    nr_mfns = 0;
    for ( i = 0; i < pte_last; i++ )
    {
        if ( pt_levels == 2 )
            pte = ((uint32_t *)page)[i];
        else
            pte = ((uint64_t *)page)[i];
        
        /* XXX SMH: below needs fixing for PROT_NONE etc */
        if ( !(pte & _PAGE_PRESENT) )
            continue;
        
        pfn = (pte >> PAGE_SHIFT) & MFN_MASK_X86;

        if ( p2m[pfn] == (INVALID_P2M_ENTRY-1) )
            p2m[pfn] = p2m_batch[nr_mfns++];

        pte &= ~MADDR_MASK_X86;
        pte |= (uint64_t)p2m[pfn] << PAGE_SHIFT;

        if ( pt_levels == 2 )
            ((uint32_t *)page)[i] = (uint32_t)pte;
        else
            ((uint64_t *)page)[i] = (uint64_t)pte;
    }

    return 1;
}


/* Load the p2m frame list, plus potential extended info chunk */
static xen_pfn_t *load_p2m_frame_list(
    int io_fd, int *pae_extended_cr3, int *ext_vcpucontext)
{
    xen_pfn_t *p2m_frame_list;
    vcpu_guest_context_any_t ctxt;
    xen_pfn_t p2m_fl_zero;

    /* Read first entry of P2M list, or extended-info signature (~0UL). */
    if ( read_exact(io_fd, &p2m_fl_zero, sizeof(long)) )
    {
        ERROR("read extended-info signature failed");
        return NULL;
    }
    
    if ( p2m_fl_zero == ~0UL )
    {
        uint32_t tot_bytes;
        
        /* Next 4 bytes: total size of following extended info. */
        if ( read_exact(io_fd, &tot_bytes, sizeof(tot_bytes)) )
        {
            ERROR("read extended-info size failed");
            return NULL;
        }
        
        while ( tot_bytes )
        {
            uint32_t chunk_bytes;
            char     chunk_sig[4];
            
            /* 4-character chunk signature + 4-byte remaining chunk size. */
            if ( read_exact(io_fd, chunk_sig, sizeof(chunk_sig)) ||
                 read_exact(io_fd, &chunk_bytes, sizeof(chunk_bytes)) ||
                 (tot_bytes < (chunk_bytes + 8)) )
            {
                ERROR("read extended-info chunk signature failed");
                return NULL;
            }
            tot_bytes -= 8;

            /* VCPU context structure? */
            if ( !strncmp(chunk_sig, "vcpu", 4) )
            {
                /* Pick a guest word-size and PT depth from the ctxt size */
                if ( chunk_bytes == sizeof (ctxt.x32) )
                {
                    guest_width = 4;
                    if ( pt_levels > 2 ) 
                        pt_levels = 3; 
                }
                else if ( chunk_bytes == sizeof (ctxt.x64) )
                {
                    guest_width = 8;
                    pt_levels = 4;
                }
                else 
                {
                    ERROR("bad extended-info context size %d", chunk_bytes);
                    return NULL;
                }

                if ( read_exact(io_fd, &ctxt, chunk_bytes) )
                {
                    ERROR("read extended-info vcpu context failed");
                    return NULL;
                }
                tot_bytes -= chunk_bytes;
                chunk_bytes = 0;

                if ( GET_FIELD(&ctxt, vm_assist) 
                     & (1UL << VMASST_TYPE_pae_extended_cr3) )
                    *pae_extended_cr3 = 1;
            }
            else if ( !strncmp(chunk_sig, "extv", 4) )
            {
                *ext_vcpucontext = 1;
            }
            
            /* Any remaining bytes of this chunk: read and discard. */
            while ( chunk_bytes )
            {
                unsigned long sz = MIN(chunk_bytes, sizeof(xen_pfn_t));
                if ( read_exact(io_fd, &p2m_fl_zero, sz) )
                {
                    ERROR("read-and-discard extended-info chunk bytes failed");
                    return NULL;
                }
                chunk_bytes -= sz;
                tot_bytes   -= sz;
            }
        }

        /* Now read the real first entry of P2M list. */
        if ( read_exact(io_fd, &p2m_fl_zero, sizeof(xen_pfn_t)) )
        {
            ERROR("read first entry of p2m_frame_list failed");
            return NULL;
        }
    }

    /* Now that we know the guest's word-size, can safely allocate 
     * the p2m frame list */
    if ( (p2m_frame_list = malloc(P2M_TOOLS_FL_SIZE)) == NULL )
    {
        ERROR("Couldn't allocate p2m_frame_list array");
        return NULL;
    }

    /* First entry has already been read. */
    p2m_frame_list[0] = p2m_fl_zero;
    if ( read_exact(io_fd, &p2m_frame_list[1], 
                    (P2M_FL_ENTRIES - 1) * sizeof(xen_pfn_t)) )
    {
        ERROR("read p2m_frame_list failed");
        return NULL;
    }
    
    return p2m_frame_list;
}

int xc_domain_restore(int xc_handle, int io_fd, uint32_t dom,
                      unsigned int store_evtchn, unsigned long *store_mfn,
                      unsigned int console_evtchn, unsigned long *console_mfn,
                      unsigned int hvm, unsigned int pae)
{
    DECLARE_DOMCTL;
    int rc = 1, frc, i, j, n, m, pae_extended_cr3 = 0, ext_vcpucontext = 0;
    unsigned long mfn, pfn;
    unsigned int prev_pc, this_pc;
    int verify = 0;
    int nraces = 0;

    /* The new domain's shared-info frame number. */
    unsigned long shared_info_frame;
    unsigned char shared_info_page[PAGE_SIZE]; /* saved contents from file */
    shared_info_any_t *old_shared_info = 
        (shared_info_any_t *)shared_info_page;
    shared_info_any_t *new_shared_info;

    /* A copy of the CPU context of the guest. */
    vcpu_guest_context_any_t ctxt;

    /* A table containing the type of each PFN (/not/ MFN!). */
    unsigned long *pfn_type = NULL;

    /* A table of MFNs to map in the current region */
    xen_pfn_t *region_mfn = NULL;

    /* Types of the pfns in the current region */
    unsigned long region_pfn_type[MAX_BATCH_SIZE];

    /* A copy of the pfn-to-mfn table frame list. */
    xen_pfn_t *p2m_frame_list = NULL;
    
    /* A temporary mapping of the guest's start_info page. */
    start_info_any_t *start_info;

    /* Our mapping of the current region (batch) */
    char *region_base;

    struct xc_mmu *mmu = NULL;

    /* used by debug verify code */
    unsigned long buf[PAGE_SIZE/sizeof(unsigned long)];

    struct mmuext_op pin[MAX_PIN_BATCH];
    unsigned int nr_pins;

    uint64_t vcpumap = 1ULL;
    unsigned int max_vcpu_id = 0;
    int new_ctxt_format = 0;

    /* Magic frames in HVM guests: ioreqs and xenstore comms. */
    uint64_t magic_pfns[3]; /* ioreq_pfn, bufioreq_pfn, store_pfn */

    /* Buffer for holding HVM context */
    uint8_t *hvm_buf = NULL;

    /* For info only */
    nr_pfns = 0;

    DPRINTF("-->xc_domain_restore()\n");

    if ( read_exact(io_fd, &p2m_size, sizeof(unsigned long)) )
    {
        ERROR("read: p2m_size");
        goto out;
    }
    
    DPRINTF("xc_domain_restore start: p2m_size = %lx\n", p2m_size);
    
    if ( !get_platform_info(xc_handle, dom,
                            &max_mfn, &hvirt_start, &pt_levels, &guest_width) )
    {
        ERROR("Unable to get platform info.");
        return 1;
    }
    
    DPRINTF("platform info, max_mfn: %lx, hvirt_start %lx, pt_lvevel %x, guest_width %x\n",
                            max_mfn, hvirt_start, pt_levels, guest_width);
    
    /* The *current* word size of the guest isn't very interesting; for now
     * assume the guest will be the same as we are.  We'll fix that later
     * if we discover otherwise. */
    guest_width = sizeof(unsigned long);
    pt_levels = (guest_width == 8) ? 4 : (pt_levels == 2) ? 2 : 3; 
    
    if ( !hvm ) 
    {
        /* Load the p2m frame list, plus potential extended info chunk */
        p2m_frame_list = load_p2m_frame_list(
            io_fd, &pae_extended_cr3, &ext_vcpucontext);
        if ( !p2m_frame_list )
            goto out;

        /* Now that we know the word size, tell Xen about it */
        memset(&domctl, 0, sizeof(domctl));
        domctl.domain = dom;
        domctl.cmd    = XEN_DOMCTL_set_address_size;
        domctl.u.address_size.size = guest_width * 8;
        frc = do_domctl(xc_handle, &domctl);
        if ( frc != 0 )
        {
            ERROR("Unable to set guest address size.");
            goto out;
        }
    }

    /* We want zeroed memory so use calloc rather than malloc. */
    p2m        = calloc(p2m_size, sizeof(xen_pfn_t));
    pfn_type   = calloc(p2m_size, sizeof(unsigned long));

    region_mfn = xg_memalign(PAGE_SIZE, ROUNDUP(
                              MAX_BATCH_SIZE * sizeof(xen_pfn_t), PAGE_SHIFT));
    p2m_batch  = xg_memalign(PAGE_SIZE, ROUNDUP(
                              MAX_BATCH_SIZE * sizeof(xen_pfn_t), PAGE_SHIFT));

    if ( (p2m == NULL) || (pfn_type == NULL) ||
         (region_mfn == NULL) || (p2m_batch == NULL) )
    {
        ERROR("memory alloc failed");
        errno = ENOMEM;
        goto out;
    }
    
    DPRINTF("memory alloc success\n");
    
    memset(region_mfn, 0,
           ROUNDUP(MAX_BATCH_SIZE * sizeof(xen_pfn_t), PAGE_SHIFT)); 
    memset(p2m_batch, 0,
           ROUNDUP(MAX_BATCH_SIZE * sizeof(xen_pfn_t), PAGE_SHIFT)); 

    if ( lock_pages(region_mfn, sizeof(xen_pfn_t) * MAX_BATCH_SIZE) )
    {
        ERROR("Could not lock region_mfn");
        goto out;
    }

    if ( lock_pages(p2m_batch, sizeof(xen_pfn_t) * MAX_BATCH_SIZE) )
    {
        ERROR("Could not lock p2m_batch");
        goto out;
    }

    DPRINTF("lock regin_mfn and p2m_batch success\n");

    /* Get the domain's shared-info frame. */
    domctl.cmd = XEN_DOMCTL_getdomaininfo;
    domctl.domain = (domid_t)dom;
    if ( xc_domctl(xc_handle, &domctl) < 0 )
    {
        ERROR("Could not get information on new domain");
        goto out;
    }
    shared_info_frame = domctl.u.getdomaininfo.shared_info_frame;
    
    DPRINTF("shared_info_frame: 0x%lx\n", shared_info_frame);

    /* Mark all PFNs as invalid; we allocate on demand */
    for ( pfn = 0; pfn < p2m_size; pfn++ )
        p2m[pfn] = INVALID_P2M_ENTRY;

    DPRINTF("all PFNs are marked as invalid\n");

    mmu = xc_alloc_mmu_updates(xc_handle, dom);
    if ( mmu == NULL )
    {
        ERROR("Could not initialise for MMU updates");
        goto out;
    }

    DPRINTF("Reloading memory pages:   0%%\n");

    /*
     * Now simply read each saved frame into its new machine frame.
     * We uncanonicalise page tables as we go.
     */
    prev_pc = 0;

    n = m = 0;
    for ( ; ; )
    {
        int j, nr_mfns = 0; 

        this_pc = (n * 100) / p2m_size;
        if ( (this_pc - prev_pc) >= 5 )
        {
            PPRINTF("\b\b\b\b%3d%%", this_pc);
            prev_pc = this_pc;
        }

        if ( read_exact(io_fd, &j, sizeof(int)) )
        {
            ERROR("Error when reading batch size");
            goto out;
        }

        PPRINTF("batch %d\n",j);
        DPRINTF ("batch %d\n",j);

        if ( j == -1 )
        {
            verify = 1;
            DPRINTF("Entering page verify mode\n");
            continue;
        }

        if ( j == -2 )
        {
            new_ctxt_format = 1;
            if ( read_exact(io_fd, &max_vcpu_id, sizeof(int)) ||
                 (max_vcpu_id >= 64) ||
                 read_exact(io_fd, &vcpumap, sizeof(uint64_t)) )
            {
                ERROR("Error when reading max_vcpu_id");
                goto out;
            }
            continue;
        }

        if ( j == -3 )
        {
            uint64_t ident_pt;

            /* Skip padding 4 bytes then read the EPT identity PT location. */
            if ( read_exact(io_fd, &ident_pt, sizeof(uint32_t)) ||
                 read_exact(io_fd, &ident_pt, sizeof(uint64_t)) )
            {
                ERROR("error read the address of the EPT identity map");
                goto out;
            }

            xc_set_hvm_param(xc_handle, dom, HVM_PARAM_IDENT_PT, ident_pt);
            continue;
        }

        if ( j == -4 )
        {
            uint64_t vm86_tss;

            /* Skip padding 4 bytes then read the vm86 TSS location. */
            if ( read_exact(io_fd, &vm86_tss, sizeof(uint32_t)) ||
                 read_exact(io_fd, &vm86_tss, sizeof(uint64_t)) )
            {
                ERROR("error read the address of the vm86 TSS");
                goto out;
            }

            xc_set_hvm_param(xc_handle, dom, HVM_PARAM_VM86_TSS, vm86_tss);
            continue;
        }

        if ( j == -5 )
        {
            DPRINTF("xc_domain_restore start tmem\n");
            if ( xc_tmem_restore(xc_handle, dom, io_fd) )
            {
                ERROR("error reading/restoring tmem");
                goto out;
            }
            continue;
        }

        if ( j == -6 )
        {
            if ( xc_tmem_restore_extra(xc_handle, dom, io_fd) )
            {
                ERROR("error reading/restoring tmem extra");
                goto out;
            }
            continue;
        }

        if ( j == 0 )
            break;  /* our work here is done */

        if ( (j > MAX_BATCH_SIZE) || (j < 0) )
        {
            ERROR("Max batch size exceeded. Giving up.");
            goto out;
        }

        if ( read_exact(io_fd, region_pfn_type, j*sizeof(unsigned long)) )
        {
            ERROR("Error when reading region pfn types");
            goto out;
        }

        /* First pass for this batch: work out how much memory to alloc */
        nr_mfns = 0; 
        for ( i = 0; i < j; i++ )
        {
            unsigned long pfn, pagetype;
            pfn      = region_pfn_type[i] & ~XEN_DOMCTL_PFINFO_LTAB_MASK;
            pagetype = region_pfn_type[i] &  XEN_DOMCTL_PFINFO_LTAB_MASK;

            if ( (pagetype != XEN_DOMCTL_PFINFO_XTAB) && 
                 (p2m[pfn] == INVALID_P2M_ENTRY) )
            {
                /* Have a live PFN which hasn't had an MFN allocated */
                p2m_batch[nr_mfns++] = pfn; 
                p2m[pfn]--;
            }
        } 

        DPRINTF("nr_mfns: %d \n",nr_mfns);
        /* Now allocate a bunch of mfns for this batch */
        if ( nr_mfns &&
             (xc_domain_memory_populate_physmap(xc_handle, dom, nr_mfns, 0,
                                                0, p2m_batch) != 0) )
        { 
            ERROR("Failed to allocate memory for batch.!\n"); 
            errno = ENOMEM;
            goto out;
        }

        /* Second pass for this batch: update p2m[] and region_mfn[] */
        nr_mfns = 0; 
        for ( i = 0; i < j; i++ )
        {
            unsigned long pfn, pagetype;
            pfn      = region_pfn_type[i] & ~XEN_DOMCTL_PFINFO_LTAB_MASK;
            pagetype = region_pfn_type[i] &  XEN_DOMCTL_PFINFO_LTAB_MASK;

            if ( pagetype == XEN_DOMCTL_PFINFO_XTAB )
                region_mfn[i] = ~0UL; /* map will fail but we don't care */
            else 
            {
                if ( p2m[pfn] == (INVALID_P2M_ENTRY-1) )
                {
                    /* We just allocated a new mfn above; update p2m */
                    p2m[pfn] = p2m_batch[nr_mfns++]; 
                    nr_pfns++; 
                }

                /* setup region_mfn[] for batch map.
                 * For HVM guests, this interface takes PFNs, not MFNs */
                region_mfn[i] = hvm ? pfn : p2m[pfn]; 
            }
        } 

        /* Map relevant mfns */
        region_base = xc_map_foreign_batch(
            xc_handle, dom, PROT_WRITE, region_mfn, j);

        if ( region_base == NULL )
        {
            ERROR("map batch failed");
            goto out;
        }

        for ( i = 0; i < j; i++ )
        {
            void *page;
            unsigned long pagetype;

            pfn      = region_pfn_type[i] & ~XEN_DOMCTL_PFINFO_LTAB_MASK;
            pagetype = region_pfn_type[i] &  XEN_DOMCTL_PFINFO_LTAB_MASK;

            if ( pagetype == XEN_DOMCTL_PFINFO_XTAB )
                /* a bogus/unmapped page: skip it */
                continue;

            if ( pfn > p2m_size )
            {
                ERROR("pfn out of range");
                goto out;
            }

            pfn_type[pfn] = pagetype;

            mfn = p2m[pfn];

            /* In verify mode, we use a copy; otherwise we work in place */
            page = verify ? (void *)buf : (region_base + i*PAGE_SIZE);

            if ( read_exact(io_fd, page, PAGE_SIZE) )
            {
                ERROR("Error when reading page (type was %lx)", pagetype);
                goto out;
            }

            pagetype &= XEN_DOMCTL_PFINFO_LTABTYPE_MASK;

            if ( (pagetype >= XEN_DOMCTL_PFINFO_L1TAB) && 
                 (pagetype <= XEN_DOMCTL_PFINFO_L4TAB) )
            {
                /*
                ** A page table page - need to 'uncanonicalize' it, i.e.
                ** replace all the references to pfns with the corresponding
                ** mfns for the new domain.
                **
                ** On PAE we need to ensure that PGDs are in MFNs < 4G, and
                ** so we may need to update the p2m after the main loop.
                ** Hence we defer canonicalization of L1s until then.
                */
                if ((pt_levels != 3) ||
                    pae_extended_cr3 ||
                    (pagetype != XEN_DOMCTL_PFINFO_L1TAB)) {
                    DPRINTF("call uncanonicalize_pagetable \n");
                    if (!uncanonicalize_pagetable(xc_handle, dom, 
                                                  pagetype, page)) {
                        /*
                        ** Failing to uncanonicalize a page table can be ok
                        ** under live migration since the pages type may have
                        ** changed by now (and we'll get an update later).
                        */
                        DPRINTF("PT L%ld race on pfn=%08lx mfn=%08lx\n",
                                pagetype >> 28, pfn, mfn);
                        nraces++;
                        continue;
                    } 
                }
            }
            else if ( pagetype != XEN_DOMCTL_PFINFO_NOTAB )
            {
                ERROR("Bogus page type %lx page table is out of range: "
                    "i=%d p2m_size=%lu", pagetype, i, p2m_size);
                goto out;

            }

            if ( verify )
            {
                int res = memcmp(buf, (region_base + i*PAGE_SIZE), PAGE_SIZE);
                if ( res )
                {
                    int v;

                    DPRINTF("************** pfn=%lx type=%lx gotcs=%08lx "
                            "actualcs=%08lx\n", pfn, pfn_type[pfn],
                            csum_page(region_base + i*PAGE_SIZE),
                            csum_page(buf));

                    for ( v = 0; v < 4; v++ )
                    {
                        unsigned long *p = (unsigned long *)
                            (region_base + i*PAGE_SIZE);
                        if ( buf[v] != p[v] )
                            DPRINTF("    %d: %08lx %08lx\n", v, buf[v], p[v]);
                    }
                }
            }

            if ( !hvm &&
                 xc_add_mmu_update(xc_handle, mmu,
                                   (((unsigned long long)mfn) << PAGE_SHIFT)
                                   | MMU_MACHPHYS_UPDATE, pfn) )
            {
                ERROR("failed machpys update mfn=%lx pfn=%lx", mfn, pfn);
                goto out;
            }
        } /* end of 'batch' for loop */

        munmap(region_base, j*PAGE_SIZE);
        n+= j; /* crude stats */

        /* 
         * Discard cache for portion of file read so far up to last
         *  page boundary every 16MB or so.
         */
        m += j;
        if ( m > MAX_PAGECACHE_USAGE )
        {
            discard_file_cache(io_fd, 0 /* no flush */);
            m = 0;
        }
    }

    /*
     * Ensure we flush all machphys updates before potential PAE-specific
     * reallocations below.
     */
    if ( !hvm && xc_flush_mmu_updates(xc_handle, mmu) )
    {
        ERROR("Error doing flush_mmu_updates()");
        goto out;
    }

    DPRINTF("Received all pages (%d races)\n", nraces);

    if ( hvm ) 
    {
        uint32_t rec_len;

        /* Set HVM-specific parameters */
        if ( read_exact(io_fd, magic_pfns, sizeof(magic_pfns)) )
        {
            ERROR("error reading magic page addresses");
            goto out;
        }
        
        /* These comms pages need to be zeroed at the start of day */
        if ( xc_clear_domain_page(xc_handle, dom, magic_pfns[0]) ||
             xc_clear_domain_page(xc_handle, dom, magic_pfns[1]) ||
             xc_clear_domain_page(xc_handle, dom, magic_pfns[2]) )
        {
            ERROR("error zeroing magic pages");
            goto out;
        }
                
        if ( (frc = xc_set_hvm_param(xc_handle, dom, 
                                     HVM_PARAM_IOREQ_PFN, magic_pfns[0]))
             || (frc = xc_set_hvm_param(xc_handle, dom, 
                                        HVM_PARAM_BUFIOREQ_PFN, magic_pfns[1]))
             || (frc = xc_set_hvm_param(xc_handle, dom, 
                                        HVM_PARAM_STORE_PFN, magic_pfns[2]))
             || (frc = xc_set_hvm_param(xc_handle, dom, 
                                        HVM_PARAM_PAE_ENABLED, pae))
             || (frc = xc_set_hvm_param(xc_handle, dom, 
                                        HVM_PARAM_STORE_EVTCHN,
                                        store_evtchn)) )
        {
            ERROR("error setting HVM params: %i", frc);
            goto out;
        }
        *store_mfn = magic_pfns[2];

        /* Read HVM context */
        if ( read_exact(io_fd, &rec_len, sizeof(uint32_t)) )
        {
            ERROR("error read hvm context size!\n");
            goto out;
        }
        
        hvm_buf = malloc(rec_len);
        if ( hvm_buf == NULL )
        {
            ERROR("memory alloc for hvm context buffer failed");
            errno = ENOMEM;
            goto out;
        }
        
        if ( read_exact(io_fd, hvm_buf, rec_len) )
        {
            ERROR("error loading the HVM context");
            goto out;
        }
        
        frc = xc_domain_hvm_setcontext(xc_handle, dom, hvm_buf, rec_len);
        if ( frc )
        {
            ERROR("error setting the HVM context");
            goto out;
        }

        /* HVM success! */
        rc = 0;
        goto out;
    }

    /* Non-HVM guests only from here on */

    if ( (pt_levels == 3) && !pae_extended_cr3 )
    {
        /*
        ** XXX SMH on PAE we need to ensure PGDs are in MFNs < 4G. This
        ** is a little awkward and involves (a) finding all such PGDs and
        ** replacing them with 'lowmem' versions; (b) upating the p2m[]
        ** with the new info; and (c) canonicalizing all the L1s using the
        ** (potentially updated) p2m[].
        **
        ** This is relatively slow (and currently involves two passes through
        ** the pfn_type[] array), but at least seems to be correct. May wish
        ** to consider more complex approaches to optimize this later.
        */

        int j, k;
        
        /* First pass: find all L3TABs current in > 4G mfns and get new mfns */
        for ( i = 0; i < p2m_size; i++ )
        {
            if ( ((pfn_type[i] & XEN_DOMCTL_PFINFO_LTABTYPE_MASK) ==
                  XEN_DOMCTL_PFINFO_L3TAB) &&
                 (p2m[i] > 0xfffffUL) )
            {
                unsigned long new_mfn;
                uint64_t l3ptes[4];
                uint64_t *l3tab;

                l3tab = (uint64_t *)
                    xc_map_foreign_range(xc_handle, dom, PAGE_SIZE,
                                         PROT_READ, p2m[i]);

                for ( j = 0; j < 4; j++ )
                    l3ptes[j] = l3tab[j];

                munmap(l3tab, PAGE_SIZE);

                new_mfn = xc_make_page_below_4G(xc_handle, dom, p2m[i]);
                if ( !new_mfn )
                {
                    ERROR("Couldn't get a page below 4GB :-(");
                    goto out;
                }

                p2m[i] = new_mfn;
                if ( xc_add_mmu_update(xc_handle, mmu,
                                       (((unsigned long long)new_mfn)
                                        << PAGE_SHIFT) |
                                       MMU_MACHPHYS_UPDATE, i) )
                {
                    ERROR("Couldn't m2p on PAE root pgdir");
                    goto out;
                }

                l3tab = (uint64_t *)
                    xc_map_foreign_range(xc_handle, dom, PAGE_SIZE,
                                         PROT_READ | PROT_WRITE, p2m[i]);

                for ( j = 0; j < 4; j++ )
                    l3tab[j] = l3ptes[j];

                munmap(l3tab, PAGE_SIZE);
            }
        }

        /* Second pass: find all L1TABs and uncanonicalize them */
        j = 0;

        for ( i = 0; i < p2m_size; i++ )
        {
            if ( ((pfn_type[i] & XEN_DOMCTL_PFINFO_LTABTYPE_MASK) ==
                  XEN_DOMCTL_PFINFO_L1TAB) )
            {
                region_mfn[j] = p2m[i];
                j++;
            }

            if ( (i == (p2m_size-1)) || (j == MAX_BATCH_SIZE) )
            {
                region_base = xc_map_foreign_batch(
                    xc_handle, dom, PROT_READ | PROT_WRITE, region_mfn, j);
                if ( region_base == NULL )
                {
                    ERROR("map batch failed");
                    goto out;
                }

                for ( k = 0; k < j; k++ )
                {
                    if ( !uncanonicalize_pagetable(
                        xc_handle, dom, XEN_DOMCTL_PFINFO_L1TAB,
                        region_base + k*PAGE_SIZE) )
                    {
                        ERROR("failed uncanonicalize pt!");
                        goto out;
                    }
                }

                munmap(region_base, j*PAGE_SIZE);
                j = 0;
            }
        }

        if ( xc_flush_mmu_updates(xc_handle, mmu) )
        {
            ERROR("Error doing xc_flush_mmu_updates()");
            goto out;
        }
    }

    /*
     * Pin page tables. Do this after writing to them as otherwise Xen
     * will barf when doing the type-checking.
     */
    nr_pins = 0;
    for ( i = 0; i < p2m_size; i++ )
    {
        if ( (pfn_type[i] & XEN_DOMCTL_PFINFO_LPINTAB) == 0 )
            continue;

        switch ( pfn_type[i] & XEN_DOMCTL_PFINFO_LTABTYPE_MASK )
        {
        case XEN_DOMCTL_PFINFO_L1TAB:
            pin[nr_pins].cmd = MMUEXT_PIN_L1_TABLE;
            break;

        case XEN_DOMCTL_PFINFO_L2TAB:
            pin[nr_pins].cmd = MMUEXT_PIN_L2_TABLE;
            break;

        case XEN_DOMCTL_PFINFO_L3TAB:
            pin[nr_pins].cmd = MMUEXT_PIN_L3_TABLE;
            break;

        case XEN_DOMCTL_PFINFO_L4TAB:
            pin[nr_pins].cmd = MMUEXT_PIN_L4_TABLE;
            break;

        default:
            continue;
        }

        pin[nr_pins].arg1.mfn = p2m[i];
        nr_pins++;

        /* Batch full? Then flush. */
        if ( nr_pins == MAX_PIN_BATCH )
        {
            if ( xc_mmuext_op(xc_handle, pin, nr_pins, dom) < 0 )
            {
                ERROR("Failed to pin batch of %d page tables", nr_pins);
                goto out;
            }
            nr_pins = 0;
        }
    }

    /* Flush final partial batch. */
    if ( (nr_pins != 0) && (xc_mmuext_op(xc_handle, pin, nr_pins, dom) < 0) )
    {
        ERROR("Failed to pin batch of %d page tables", nr_pins);
        goto out;
    }

    DPRINTF("\b\b\b\b100%%\n");
    DPRINTF("Memory reloaded (%ld pages)\n", nr_pfns);

    /* Get the list of PFNs that are not in the psuedo-phys map */
    {
        unsigned int count = 0;
        unsigned long *pfntab;
        int nr_frees;

        if ( read_exact(io_fd, &count, sizeof(count)) ||
             (count > (1U << 28)) ) /* up to 1TB of address space */
        {
            ERROR("Error when reading pfn count (= %u)", count);
            goto out;
        }

        if ( !(pfntab = malloc(sizeof(unsigned long) * count)) )
        {
            ERROR("Out of memory");
            goto out;
        }

        if ( read_exact(io_fd, pfntab, sizeof(unsigned long)*count) )
        {
            ERROR("Error when reading pfntab");
            goto out;
        }

        nr_frees = 0; 
        for ( i = 0; i < count; i++ )
        {
            unsigned long pfn = pfntab[i];

            if ( p2m[pfn] != INVALID_P2M_ENTRY )
            {
                /* pfn is not in physmap now, but was at some point during 
                   the save/migration process - need to free it */
                pfntab[nr_frees++] = p2m[pfn];
                p2m[pfn]  = INVALID_P2M_ENTRY; /* not in pseudo-physical map */
            }
        }

        if ( nr_frees > 0 )
        {
            struct xen_memory_reservation reservation = {
                .nr_extents   = nr_frees,
                .extent_order = 0,
                .domid        = dom
            };
            set_xen_guest_handle(reservation.extent_start, pfntab);

            if ( (frc = xc_memory_op(xc_handle, XENMEM_decrease_reservation,
                                     &reservation)) != nr_frees )
            {
                ERROR("Could not decrease reservation : %d", frc);
                goto out;
            }
            else
                DPRINTF("Decreased reservation by %d pages\n", count);
        }
    }

    if ( lock_pages(&ctxt, sizeof(ctxt)) )
    {
        ERROR("Unable to lock ctxt");
        return 1;
    }

    for ( i = 0; i <= max_vcpu_id; i++ )
    {
        if ( !(vcpumap & (1ULL << i)) )
            continue;

        if ( read_exact(io_fd, &ctxt, ((guest_width == 8)
                                       ? sizeof(ctxt.x64)
                                       : sizeof(ctxt.x32))) )
        {
            ERROR("Error when reading ctxt %d", i);
            goto out;
        }

        if ( !new_ctxt_format )
            SET_FIELD(&ctxt, flags, GET_FIELD(&ctxt, flags) | VGCF_online);

        if ( i == 0 )
        {
            /*
             * Uncanonicalise the suspend-record frame number and poke
             * resume record.
             */
            pfn = GET_FIELD(&ctxt, user_regs.edx);
            if ( (pfn >= p2m_size) ||
                 (pfn_type[pfn] != XEN_DOMCTL_PFINFO_NOTAB) )
            {
                ERROR("Suspend record frame number is bad");
                goto out;
            }
            mfn = p2m[pfn];
            SET_FIELD(&ctxt, user_regs.edx, mfn);
            start_info = xc_map_foreign_range(
                xc_handle, dom, PAGE_SIZE, PROT_READ | PROT_WRITE, mfn);
            SET_FIELD(start_info, nr_pages, p2m_size);
            SET_FIELD(start_info, shared_info, shared_info_frame<<PAGE_SHIFT);
            SET_FIELD(start_info, flags, 0);
            *store_mfn = p2m[GET_FIELD(start_info, store_mfn)];
            SET_FIELD(start_info, store_mfn, *store_mfn);
            SET_FIELD(start_info, store_evtchn, store_evtchn);
            *console_mfn = p2m[GET_FIELD(start_info, console.domU.mfn)];
            SET_FIELD(start_info, console.domU.mfn, *console_mfn);
            SET_FIELD(start_info, console.domU.evtchn, console_evtchn);
            munmap(start_info, PAGE_SIZE);
        }
        /* Uncanonicalise each GDT frame number. */
        if ( GET_FIELD(&ctxt, gdt_ents) > 8192 )
        {
            ERROR("GDT entry count out of range");
            goto out;
        }

        for ( j = 0; (512*j) < GET_FIELD(&ctxt, gdt_ents); j++ )
        {
            pfn = GET_FIELD(&ctxt, gdt_frames[j]);
            if ( (pfn >= p2m_size) ||
                 (pfn_type[pfn] != XEN_DOMCTL_PFINFO_NOTAB) )
            {
                ERROR("GDT frame number %i (0x%lx) is bad", 
                      j, (unsigned long)pfn);
                goto out;
            }
            SET_FIELD(&ctxt, gdt_frames[j], p2m[pfn]);
        }
        /* Uncanonicalise the page table base pointer. */
        pfn = UNFOLD_CR3(GET_FIELD(&ctxt, ctrlreg[3]));

        if ( pfn >= p2m_size )
        {
            ERROR("PT base is bad: pfn=%lu p2m_size=%lu type=%08lx",
                  pfn, p2m_size, pfn_type[pfn]);
            goto out;
        }

        if ( (pfn_type[pfn] & XEN_DOMCTL_PFINFO_LTABTYPE_MASK) !=
             ((unsigned long)pt_levels<<XEN_DOMCTL_PFINFO_LTAB_SHIFT) )
        {
            ERROR("PT base is bad. pfn=%lu nr=%lu type=%08lx %08lx",
                  pfn, p2m_size, pfn_type[pfn],
                  (unsigned long)pt_levels<<XEN_DOMCTL_PFINFO_LTAB_SHIFT);
            goto out;
        }
        SET_FIELD(&ctxt, ctrlreg[3], FOLD_CR3(p2m[pfn]));

        /* Guest pagetable (x86/64) stored in otherwise-unused CR1. */
        if ( (pt_levels == 4) && (ctxt.x64.ctrlreg[1] & 1) )
        {
            pfn = UNFOLD_CR3(ctxt.x64.ctrlreg[1] & ~1);
            if ( pfn >= p2m_size )
            {
                ERROR("User PT base is bad: pfn=%lu p2m_size=%lu",
                      pfn, p2m_size);
                goto out;
            }
            if ( (pfn_type[pfn] & XEN_DOMCTL_PFINFO_LTABTYPE_MASK) !=
                 ((unsigned long)pt_levels<<XEN_DOMCTL_PFINFO_LTAB_SHIFT) )
            {
                ERROR("User PT base is bad. pfn=%lu nr=%lu type=%08lx %08lx",
                      pfn, p2m_size, pfn_type[pfn],
                      (unsigned long)pt_levels<<XEN_DOMCTL_PFINFO_LTAB_SHIFT);
                goto out;
            }
            ctxt.x64.ctrlreg[1] = FOLD_CR3(p2m[pfn]);
        }
        domctl.cmd = XEN_DOMCTL_setvcpucontext;
        domctl.domain = (domid_t)dom;
        domctl.u.vcpucontext.vcpu = i;
        set_xen_guest_handle(domctl.u.vcpucontext.ctxt, &ctxt.c);
        frc = xc_domctl(xc_handle, &domctl);
        if ( frc != 0 )
        {
            ERROR("Couldn't build vcpu%d", i);
            goto out;
        }

        if ( !ext_vcpucontext )
            continue;
        if ( read_exact(io_fd, &domctl.u.ext_vcpucontext, 128) ||
             (domctl.u.ext_vcpucontext.vcpu != i) )
        {
            ERROR("Error when reading extended ctxt %d", i);
            goto out;
        }
        domctl.cmd = XEN_DOMCTL_set_ext_vcpucontext;
        domctl.domain = dom;
        frc = xc_domctl(xc_handle, &domctl);
        if ( frc != 0 )
        {
            ERROR("Couldn't set extended vcpu%d info\n", i);
            goto out;
        }
    }

    if ( read_exact(io_fd, shared_info_page, PAGE_SIZE) )
    {
        ERROR("Error when reading shared info page");
        goto out;
    }

    /* Restore contents of shared-info page. No checking needed. */
    new_shared_info = xc_map_foreign_range(
        xc_handle, dom, PAGE_SIZE, PROT_WRITE, shared_info_frame);

    /* restore saved vcpu_info and arch specific info */
    MEMCPY_FIELD(new_shared_info, old_shared_info, vcpu_info);
    MEMCPY_FIELD(new_shared_info, old_shared_info, arch);

    /* clear any pending events and the selector */
    MEMSET_ARRAY_FIELD(new_shared_info, evtchn_pending, 0);
    for ( i = 0; i < MAX_VIRT_CPUS; i++ )
	    SET_FIELD(new_shared_info, vcpu_info[i].evtchn_pending_sel, 0);

    /* mask event channels */
    MEMSET_ARRAY_FIELD(new_shared_info, evtchn_mask, 0xff);

    /* leave wallclock time. set by hypervisor */
    munmap(new_shared_info, PAGE_SIZE);

    /* Uncanonicalise the pfn-to-mfn table frame-number list. */
    for ( i = 0; i < P2M_FL_ENTRIES; i++ )
    {
        pfn = p2m_frame_list[i];
        if ( (pfn >= p2m_size) || (pfn_type[pfn] != XEN_DOMCTL_PFINFO_NOTAB) )
        {
            ERROR("PFN-to-MFN frame number %i (%#lx) is bad", i, pfn);
            goto out;
        }
        p2m_frame_list[i] = p2m[pfn];
    }

    /* Copy the P2M we've constructed to the 'live' P2M */
    if ( !(live_p2m = xc_map_foreign_batch(xc_handle, dom, PROT_WRITE,
                                           p2m_frame_list, P2M_FL_ENTRIES)) )
    {
        ERROR("Couldn't map p2m table");
        goto out;
    }

    /* If the domain we're restoring has a different word size to ours,
     * we need to adjust the live_p2m assignment appropriately */
    if ( guest_width > sizeof (xen_pfn_t) )
        for ( i = p2m_size - 1; i >= 0; i-- )
            ((int64_t *)live_p2m)[i] = (long)p2m[i];
    else if ( guest_width < sizeof (xen_pfn_t) )
        for ( i = 0; i < p2m_size; i++ )   
            ((uint32_t *)live_p2m)[i] = p2m[i];
    else
        memcpy(live_p2m, p2m, p2m_size * sizeof(xen_pfn_t));
    munmap(live_p2m, P2M_FL_ENTRIES * PAGE_SIZE);

    DPRINTF("Domain ready to be built.\n");
    rc = 0;

 out:
    if ( (rc != 0) && (dom != 0) )
        xc_domain_destroy(xc_handle, dom);
    free(mmu);
    free(p2m);
    free(pfn_type);
    free(hvm_buf);

    /* discard cache for save file  */
    discard_file_cache(io_fd, 1 /*flush*/);

    DPRINTF("Restore exit with rc=%d\n", rc);
    
    return rc;
}

[-- Attachment #4: xc_domain_save.c --]
[-- Type: text/plain, Size: 52907 bytes --]

/******************************************************************************
 * xc_linux_save.c
 *
 * Save the state of a running Linux session.
 *
 * Copyright (c) 2003, K A Fraser.
 */

#include <inttypes.h>
#include <time.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/time.h>

#include "xc_private.h"
#include "xc_dom.h"
#include "xg_private.h"
#include "xg_save_restore.h"

#include <xen/hvm/params.h>
#include "xc_e820.h"

/*
** Default values for important tuning parameters. Can override by passing
** non-zero replacement values to xc_domain_save().
**
** XXX SMH: should consider if want to be able to override MAX_MBIT_RATE too.
**
*/
#define DEF_MAX_ITERS   29   /* limit us to 30 times round loop   */
#define DEF_MAX_FACTOR   3   /* never send more than 3x p2m_size  */

/* max mfn of the whole machine */
static unsigned long max_mfn;

/* virtual starting address of the hypervisor */
static unsigned long hvirt_start;

/* #levels of page tables used by the current guest */
static unsigned int pt_levels;

/* HVM: shared-memory bitmaps for getting log-dirty bits from qemu-dm */
static unsigned long *qemu_bitmaps[2];
static int qemu_active;
static int qemu_non_active;

/* number of pfns this guest has (i.e. number of entries in the P2M) */
static unsigned long p2m_size;

/* Live mapping of the table mapping each PFN to its current MFN. */
static xen_pfn_t *live_p2m = NULL;

/* Live mapping of system MFN to PFN table. */
static xen_pfn_t *live_m2p = NULL;
static unsigned long m2p_mfn0;

/* Address size of the guest */
unsigned int guest_width;

/* grep fodder: machine_to_phys */

#define mfn_to_pfn(_mfn)  (live_m2p[(_mfn)])

#define pfn_to_mfn(_pfn)                                            \
  ((xen_pfn_t) ((guest_width==8)                                    \
                ? (((uint64_t *)live_p2m)[(_pfn)])                  \
                : ((((uint32_t *)live_p2m)[(_pfn)]) == 0xffffffffU  \
                   ? (-1UL) : (((uint32_t *)live_p2m)[(_pfn)]))))

/*
 * Returns TRUE if the given machine frame number has a unique mapping
 * in the guest's pseudophysical map.
 */
#define MFN_IS_IN_PSEUDOPHYS_MAP(_mfn)          \
    (((_mfn) < (max_mfn)) &&                    \
     ((mfn_to_pfn(_mfn) < (p2m_size)) &&        \
      (pfn_to_mfn(mfn_to_pfn(_mfn)) == (_mfn))))

/*
** During (live) save/migrate, we maintain a number of bitmaps to track
** which pages we have to send, to fixup, and to skip.
*/

#define BITS_PER_LONG (sizeof(unsigned long) * 8)
#define BITS_TO_LONGS(bits) (((bits)+BITS_PER_LONG-1)/BITS_PER_LONG)
#define BITMAP_SIZE   (BITS_TO_LONGS(p2m_size) * sizeof(unsigned long))

#define BITMAP_ENTRY(_nr,_bmap) \
   ((volatile unsigned long *)(_bmap))[(_nr)/BITS_PER_LONG]

#define BITMAP_SHIFT(_nr) ((_nr) % BITS_PER_LONG)

static inline int test_bit (int nr, volatile void * addr)
{
    return (BITMAP_ENTRY(nr, addr) >> BITMAP_SHIFT(nr)) & 1;
}

static inline void clear_bit (int nr, volatile void * addr)
{
    BITMAP_ENTRY(nr, addr) &= ~(1UL << BITMAP_SHIFT(nr));
}

static inline void set_bit ( int nr, volatile void * addr)
{
    BITMAP_ENTRY(nr, addr) |= (1UL << BITMAP_SHIFT(nr));
}

/* Returns the hamming weight (i.e. the number of bits set) in a N-bit word */
static inline unsigned int hweight32(unsigned int w)
{
    unsigned int res = (w & 0x55555555) + ((w >> 1) & 0x55555555);
    res = (res & 0x33333333) + ((res >> 2) & 0x33333333);
    res = (res & 0x0F0F0F0F) + ((res >> 4) & 0x0F0F0F0F);
    res = (res & 0x00FF00FF) + ((res >> 8) & 0x00FF00FF);
    return (res & 0x0000FFFF) + ((res >> 16) & 0x0000FFFF);
}

static inline int count_bits ( int nr, volatile void *addr)
{
    int i, count = 0;
    volatile unsigned long *p = (volatile unsigned long *)addr;
    /* We know that the array is padded to unsigned long. */
    for ( i = 0; i < (nr / (sizeof(unsigned long)*8)); i++, p++ )
        count += hweight32(*p);
    return count;
}

static uint64_t tv_to_us(struct timeval *new)
{
    return (new->tv_sec * 1000000) + new->tv_usec;
}

static uint64_t llgettimeofday(void)
{
    struct timeval now;
    gettimeofday(&now, NULL);
    return tv_to_us(&now);
}

static uint64_t tv_delta(struct timeval *new, struct timeval *old)
{
    return (((new->tv_sec - old->tv_sec)*1000000) +
            (new->tv_usec - old->tv_usec));
}

static int noncached_write(int fd, int live, void *buffer, int len) 
{
    static int write_count = 0;
    int rc = (write_exact(fd, buffer, len) == 0) ? len : -1;

    write_count += len;
    if ( write_count >= (MAX_PAGECACHE_USAGE * PAGE_SIZE) )
    {
        /* Time to discard cache - dont care if this fails */
        discard_file_cache(fd, 0 /* no flush */);
        write_count = 0;
    }

    return rc;
}

#ifdef ADAPTIVE_SAVE

/*
** We control the rate at which we transmit (or save) to minimize impact
** on running domains (including the target if we're doing live migrate).
*/

#define MAX_MBIT_RATE    500      /* maximum transmit rate for migrate */
#define START_MBIT_RATE  100      /* initial transmit rate for migrate */

/* Scaling factor to convert between a rate (in Mb/s) and time (in usecs) */
#define RATE_TO_BTU      781250

/* Amount in bytes we allow ourselves to send in a burst */
#define BURST_BUDGET (100*1024)

/* We keep track of the current and previous transmission rate */
static int mbit_rate, ombit_rate = 0;

/* Have we reached the maximum transmission rate? */
#define RATE_IS_MAX() (mbit_rate == MAX_MBIT_RATE)

static inline void initialize_mbit_rate()
{
    mbit_rate = START_MBIT_RATE;
}

static int ratewrite(int io_fd, int live, void *buf, int n)
{
    static int budget = 0;
    static int burst_time_us = -1;
    static struct timeval last_put = { 0 };
    struct timeval now;
    struct timespec delay;
    long long delta;

    if ( START_MBIT_RATE == 0 )
        return noncached_write(io_fd, live, buf, n);

    budget -= n;
    if ( budget < 0 )
    {
        if ( mbit_rate != ombit_rate )
        {
            burst_time_us = RATE_TO_BTU / mbit_rate;
            ombit_rate = mbit_rate;
            DPRINTF("rate limit: %d mbit/s burst budget %d slot time %d\n",
                    mbit_rate, BURST_BUDGET, burst_time_us);
        }
        if ( last_put.tv_sec == 0 )
        {
            budget += BURST_BUDGET;
            gettimeofday(&last_put, NULL);
        }
        else
        {
            while ( budget < 0 )
            {
                gettimeofday(&now, NULL);
                delta = tv_delta(&now, &last_put);
                while ( delta > burst_time_us )
                {
                    budget += BURST_BUDGET;
                    last_put.tv_usec += burst_time_us;
                    if ( last_put.tv_usec > 1000000 )
                    {
                        last_put.tv_usec -= 1000000;
                        last_put.tv_sec++;
                    }
                    delta -= burst_time_us;
                }
                if ( budget > 0 )
                    break;
                delay.tv_sec = 0;
                delay.tv_nsec = 1000 * (burst_time_us - delta);
                while ( delay.tv_nsec > 0 )
                    if ( nanosleep(&delay, &delay) == 0 )
                        break;
            }
        }
    }
    return noncached_write(io_fd, live, buf, n);
}

#else /* ! ADAPTIVE SAVE */

#define RATE_IS_MAX() (0)
#define ratewrite(_io_fd, _live, _buf, _n) noncached_write((_io_fd), (_live), (_buf), (_n))
#define initialize_mbit_rate()

#endif

static int print_stats(int xc_handle, uint32_t domid, int pages_sent,
                       xc_shadow_op_stats_t *stats, int print)
{
    static struct timeval wall_last;
    static long long      d0_cpu_last;
    static long long      d1_cpu_last;

    struct timeval        wall_now;
    long long             wall_delta;
    long long             d0_cpu_now, d0_cpu_delta;
    long long             d1_cpu_now, d1_cpu_delta;

    gettimeofday(&wall_now, NULL);

    d0_cpu_now = xc_domain_get_cpu_usage(xc_handle, 0, /* FIXME */ 0)/1000;
    d1_cpu_now = xc_domain_get_cpu_usage(xc_handle, domid, /* FIXME */ 0)/1000;

    if ( (d0_cpu_now == -1) || (d1_cpu_now == -1) )
        DPRINTF("ARRHHH!!\n");

    wall_delta = tv_delta(&wall_now,&wall_last)/1000;
    if ( wall_delta == 0 )
        wall_delta = 1;

    d0_cpu_delta = (d0_cpu_now - d0_cpu_last)/1000;
    d1_cpu_delta = (d1_cpu_now - d1_cpu_last)/1000;

    if ( print )
        DPRINTF("delta %lldms, dom0 %d%%, target %d%%, sent %dMb/s, "
                "dirtied %dMb/s %" PRId32 " pages\n",
                wall_delta,
                (int)((d0_cpu_delta*100)/wall_delta),
                (int)((d1_cpu_delta*100)/wall_delta),
                (int)((pages_sent*PAGE_SIZE)/(wall_delta*(1000/8))),
                (int)((stats->dirty_count*PAGE_SIZE)/(wall_delta*(1000/8))),
                stats->dirty_count);

#ifdef ADAPTIVE_SAVE
    if ( ((stats->dirty_count*PAGE_SIZE)/(wall_delta*(1000/8))) > mbit_rate )
    {
        mbit_rate = (int)((stats->dirty_count*PAGE_SIZE)/(wall_delta*(1000/8)))
            + 50;
        if ( mbit_rate > MAX_MBIT_RATE )
            mbit_rate = MAX_MBIT_RATE;
    }
#endif

    d0_cpu_last = d0_cpu_now;
    d1_cpu_last = d1_cpu_now;
    wall_last   = wall_now;

    return 0;
}


static int analysis_phase(int xc_handle, uint32_t domid, int p2m_size,
                          unsigned long *arr, int runs)
{
    long long start, now;
    xc_shadow_op_stats_t stats;
    int j;

    start = llgettimeofday();

    for ( j = 0; j < runs; j++ )
    {
        int i;

        xc_shadow_control(xc_handle, domid, XEN_DOMCTL_SHADOW_OP_CLEAN,
                          arr, p2m_size, NULL, 0, NULL);
        DPRINTF("#Flush\n");
        for ( i = 0; i < 40; i++ )
        {
            usleep(50000);
            now = llgettimeofday();
            xc_shadow_control(xc_handle, domid, XEN_DOMCTL_SHADOW_OP_PEEK,
                              NULL, 0, NULL, 0, &stats);
            DPRINTF("now= %lld faults= %"PRId32" dirty= %"PRId32"\n",
                    ((now-start)+500)/1000,
                    stats.fault_count, stats.dirty_count);
        }
    }

    return -1;
}


static int suspend_and_state(int (*suspend)(void), int xc_handle, int io_fd,
                             int dom, xc_dominfo_t *info)
{
    if ( !(*suspend)() )
    {
        ERROR("Suspend request failed");
        return -1;
    }

    if ( (xc_domain_getinfo(xc_handle, dom, 1, info) != 1) ||
         !info->shutdown || (info->shutdown_reason != SHUTDOWN_suspend) )
    {
        ERROR("Domain not in suspended state");
        return -1;
    }

    return 0;
}

/*
** Map the top-level page of MFNs from the guest. The guest might not have
** finished resuming from a previous restore operation, so we wait a while for
** it to update the MFN to a reasonable value.
*/
static void *map_frame_list_list(int xc_handle, uint32_t dom,
                                 shared_info_any_t *shinfo)
{
    int count = 100;
    void *p;
    uint64_t fll = GET_FIELD(shinfo, arch.pfn_to_mfn_frame_list_list);

    while ( count-- && (fll == 0) )
    {
        usleep(10000);
        fll = GET_FIELD(shinfo, arch.pfn_to_mfn_frame_list_list);
    }

    if ( fll == 0 )
    {
        ERROR("Timed out waiting for frame list updated.");
        return NULL;
    }

    p = xc_map_foreign_range(xc_handle, dom, PAGE_SIZE, PROT_READ, fll);
    if ( p == NULL )
        ERROR("Couldn't map p2m_frame_list_list (errno %d)", errno);

    return p;
}

/*
** During transfer (or in the state file), all page-table pages must be
** converted into a 'canonical' form where references to actual mfns
** are replaced with references to the corresponding pfns.
**
** This function performs the appropriate conversion, taking into account
** which entries do not require canonicalization (in particular, those
** entries which map the virtual address reserved for the hypervisor).
*/
static int canonicalize_pagetable(unsigned long type, unsigned long pfn,
                           const void *spage, void *dpage)
{

    int i, pte_last, xen_start, xen_end, race = 0; 
    uint64_t pte;

    /*
    ** We need to determine which entries in this page table hold
    ** reserved hypervisor mappings. This depends on the current
    ** page table type as well as the number of paging levels.
    */
    xen_start = xen_end = pte_last = PAGE_SIZE / ((pt_levels == 2) ? 4 : 8);

    if ( (pt_levels == 2) && (type == XEN_DOMCTL_PFINFO_L2TAB) )
        xen_start = (hvirt_start >> L2_PAGETABLE_SHIFT);

    if ( (pt_levels == 3) && (type == XEN_DOMCTL_PFINFO_L3TAB) )
        xen_start = L3_PAGETABLE_ENTRIES_PAE;

    /*
    ** In PAE only the L2 mapping the top 1GB contains Xen mappings.
    ** We can spot this by looking for the guest's mappingof the m2p.
    ** Guests must ensure that this check will fail for other L2s.
    */
    if ( (pt_levels == 3) && (type == XEN_DOMCTL_PFINFO_L2TAB) )
    {
        int hstart;
        uint64_t he;

        hstart = (hvirt_start >> L2_PAGETABLE_SHIFT_PAE) & 0x1ff;
        he = ((const uint64_t *) spage)[hstart];

        if ( ((he >> PAGE_SHIFT) & MFN_MASK_X86) == m2p_mfn0 )
        {
            /* hvirt starts with xen stuff... */
            xen_start = hstart;
        }
        else if ( hvirt_start != 0xf5800000 )
        {
            /* old L2s from before hole was shrunk... */
            hstart = (0xf5800000 >> L2_PAGETABLE_SHIFT_PAE) & 0x1ff;
            he = ((const uint64_t *) spage)[hstart];
            if ( ((he >> PAGE_SHIFT) & MFN_MASK_X86) == m2p_mfn0 )
                xen_start = hstart;
        }
    }

    if ( (pt_levels == 4) && (type == XEN_DOMCTL_PFINFO_L4TAB) )
    {
        /*
        ** XXX SMH: should compute these from hvirt_start (which we have)
        ** and hvirt_end (which we don't)
        */
        xen_start = 256;
        xen_end   = 272;
    }

    /* Now iterate through the page table, canonicalizing each PTE */
    for (i = 0; i < pte_last; i++ )
    {
        unsigned long pfn, mfn;

        if ( pt_levels == 2 )
            pte = ((const uint32_t*)spage)[i];
        else
            pte = ((const uint64_t*)spage)[i];

        if ( (i >= xen_start) && (i < xen_end) )
            pte = 0;

        if ( pte & _PAGE_PRESENT )
        {
            mfn = (pte >> PAGE_SHIFT) & MFN_MASK_X86;
            if ( !MFN_IS_IN_PSEUDOPHYS_MAP(mfn) )
            {
                /* This will happen if the type info is stale which
                   is quite feasible under live migration */
                pfn  = 0;  /* zap it - we'll retransmit this page later */
                /* XXX: We can't spot Xen mappings in compat-mode L2es 
                 * from 64-bit tools, but the only thing in them is the
                 * compat m2p, so we quietly zap them.  This doesn't
                 * count as a race, so don't report it. */
                if ( !(type == XEN_DOMCTL_PFINFO_L2TAB 
                       && sizeof (unsigned long) > guest_width) )
                     race = 1;  /* inform the caller; fatal if !live */ 
            }
            else
                pfn = mfn_to_pfn(mfn);

            pte &= ~MADDR_MASK_X86;
            pte |= (uint64_t)pfn << PAGE_SHIFT;

            /*
             * PAE guest L3Es can contain these flags when running on
             * a 64bit hypervisor. We zap these here to avoid any
             * surprise at restore time...
             */
            if ( (pt_levels == 3) &&
                 (type == XEN_DOMCTL_PFINFO_L3TAB) &&
                 (pte & (_PAGE_USER|_PAGE_RW|_PAGE_ACCESSED)) )
                pte &= ~(_PAGE_USER|_PAGE_RW|_PAGE_ACCESSED);
        }

        if ( pt_levels == 2 )
            ((uint32_t*)dpage)[i] = pte;
        else
            ((uint64_t*)dpage)[i] = pte;
    }

    return race;
}

static xen_pfn_t *xc_map_m2p(int xc_handle,
                                 unsigned long max_mfn,
                                 int prot)
{
    struct xen_machphys_mfn_list xmml;
    privcmd_mmap_entry_t *entries;
    unsigned long m2p_chunks, m2p_size;
    xen_pfn_t *m2p;
    xen_pfn_t *extent_start;
    int i;

    m2p = NULL;
    m2p_size   = M2P_SIZE(max_mfn);
    m2p_chunks = M2P_CHUNKS(max_mfn);

    xmml.max_extents = m2p_chunks;

    extent_start = calloc(m2p_chunks, sizeof(xen_pfn_t));
    if ( !extent_start )
    {
        ERROR("failed to allocate space for m2p mfns");
        goto err0;
    }
    set_xen_guest_handle(xmml.extent_start, extent_start);

    if ( xc_memory_op(xc_handle, XENMEM_machphys_mfn_list, &xmml) ||
         (xmml.nr_extents != m2p_chunks) )
    {
        ERROR("xc_get_m2p_mfns");
        goto err1;
    }

    entries = calloc(m2p_chunks, sizeof(privcmd_mmap_entry_t));
    if (entries == NULL)
    {
        ERROR("failed to allocate space for mmap entries");
        goto err1;
    }

    for ( i = 0; i < m2p_chunks; i++ )
        entries[i].mfn = extent_start[i];

    m2p = xc_map_foreign_ranges(xc_handle, DOMID_XEN,
			m2p_size, prot, M2P_CHUNK_SIZE,
			entries, m2p_chunks);
    if (m2p == NULL)
    {
        ERROR("xc_mmap_foreign_ranges failed");
        goto err2;
    }

    m2p_mfn0 = entries[0].mfn;

err2:
    free(entries);
err1:
    free(extent_start);

err0:
    return m2p;
}


static xen_pfn_t *map_and_save_p2m_table(int xc_handle, 
                                         int io_fd, 
                                         uint32_t dom,
                                         unsigned long p2m_size,
                                         shared_info_any_t *live_shinfo)
{
    vcpu_guest_context_any_t ctxt;

    /* Double and single indirect references to the live P2M table */
    void *live_p2m_frame_list_list = NULL;
    void *live_p2m_frame_list = NULL;

    /* Copies of the above. */
    xen_pfn_t *p2m_frame_list_list = NULL;
    xen_pfn_t *p2m_frame_list = NULL;

    /* The mapping of the live p2m table itself */
    xen_pfn_t *p2m = NULL;

    int i, success = 0;

    live_p2m_frame_list_list = map_frame_list_list(xc_handle, dom,
                                                   live_shinfo);
    if ( !live_p2m_frame_list_list )
        goto out;

    /* Get a local copy of the live_P2M_frame_list_list */
    if ( !(p2m_frame_list_list = malloc(PAGE_SIZE)) )
    {
        ERROR("Couldn't allocate p2m_frame_list_list array");
        goto out;
    }
    memcpy(p2m_frame_list_list, live_p2m_frame_list_list, PAGE_SIZE);

    /* Canonicalize guest's unsigned long vs ours */
    if ( guest_width > sizeof(unsigned long) )
        for ( i = 0; i < PAGE_SIZE/sizeof(unsigned long); i++ )
            if ( i < PAGE_SIZE/guest_width )
                p2m_frame_list_list[i] = ((uint64_t *)p2m_frame_list_list)[i];
            else
                p2m_frame_list_list[i] = 0;
    else if ( guest_width < sizeof(unsigned long) )
        for ( i = PAGE_SIZE/sizeof(unsigned long) - 1; i >= 0; i-- )
            p2m_frame_list_list[i] = ((uint32_t *)p2m_frame_list_list)[i];

    live_p2m_frame_list =
        xc_map_foreign_batch(xc_handle, dom, PROT_READ,
                             p2m_frame_list_list,
                             P2M_FLL_ENTRIES);
    if ( !live_p2m_frame_list )
    {
        ERROR("Couldn't map p2m_frame_list");
        goto out;
    }

    /* Get a local copy of the live_P2M_frame_list */
    if ( !(p2m_frame_list = malloc(P2M_TOOLS_FL_SIZE)) )
    {
        ERROR("Couldn't allocate p2m_frame_list array");
        goto out;
    }
    memset(p2m_frame_list, 0, P2M_TOOLS_FL_SIZE);
    memcpy(p2m_frame_list, live_p2m_frame_list, P2M_GUEST_FL_SIZE);

    /* Canonicalize guest's unsigned long vs ours */
    if ( guest_width > sizeof(unsigned long) )
        for ( i = 0; i < P2M_FL_ENTRIES; i++ )
            p2m_frame_list[i] = ((uint64_t *)p2m_frame_list)[i];
    else if ( guest_width < sizeof(unsigned long) )
        for ( i = P2M_FL_ENTRIES - 1; i >= 0; i-- )
            p2m_frame_list[i] = ((uint32_t *)p2m_frame_list)[i];


    /* Map all the frames of the pfn->mfn table. For migrate to succeed,
       the guest must not change which frames are used for this purpose.
       (its not clear why it would want to change them, and we'll be OK
       from a safety POV anyhow. */

    p2m = xc_map_foreign_batch(xc_handle, dom, PROT_READ,
                               p2m_frame_list,
                               P2M_FL_ENTRIES);
    if ( !p2m )
    {
        ERROR("Couldn't map p2m table");
        goto out;
    }
    live_p2m = p2m; /* So that translation macros will work */
    
    /* Canonicalise the pfn-to-mfn table frame-number list. */
    for ( i = 0; i < p2m_size; i += FPP )
    {
        if ( !MFN_IS_IN_PSEUDOPHYS_MAP(p2m_frame_list[i/FPP]) )
        {
            ERROR("Frame# in pfn-to-mfn frame list is not in pseudophys");
            ERROR("entry %d: p2m_frame_list[%ld] is 0x%"PRIx64", max 0x%lx",
                  i, i/FPP, (uint64_t)p2m_frame_list[i/FPP], max_mfn);
            if ( p2m_frame_list[i/FPP] < max_mfn ) 
            {
                ERROR("m2p[0x%"PRIx64"] = 0x%"PRIx64, 
                      (uint64_t)p2m_frame_list[i/FPP],
                      (uint64_t)live_m2p[p2m_frame_list[i/FPP]]);
                ERROR("p2m[0x%"PRIx64"] = 0x%"PRIx64, 
                      (uint64_t)live_m2p[p2m_frame_list[i/FPP]],
                      (uint64_t)p2m[live_m2p[p2m_frame_list[i/FPP]]]);

            }
            goto out;
        }
        p2m_frame_list[i/FPP] = mfn_to_pfn(p2m_frame_list[i/FPP]);
    }

    if ( xc_vcpu_getcontext(xc_handle, dom, 0, &ctxt) )
    {
        ERROR("Could not get vcpu context");
        goto out;
    }

    /*
     * Write an extended-info structure to inform the restore code that
     * a PAE guest understands extended CR3 (PDPTs above 4GB). Turns off
     * slow paths in the restore code.
     */
    {
        unsigned long signature = ~0UL;
        uint32_t chunk1_sz = ((guest_width==8) 
                              ? sizeof(ctxt.x64) 
                              : sizeof(ctxt.x32));
        uint32_t chunk2_sz = 0;
        uint32_t tot_sz    = (chunk1_sz + 8) + (chunk2_sz + 8);
        if ( write_exact(io_fd, &signature, sizeof(signature)) ||
             write_exact(io_fd, &tot_sz, sizeof(tot_sz)) ||
             write_exact(io_fd, "vcpu", 4) ||
             write_exact(io_fd, &chunk1_sz, sizeof(chunk1_sz)) ||
             write_exact(io_fd, &ctxt, chunk1_sz) ||
             write_exact(io_fd, "extv", 4) ||
             write_exact(io_fd, &chunk2_sz, sizeof(chunk2_sz)) )
        {
            PERROR("write: extended info");
            goto out;
        }
    }

    if ( write_exact(io_fd, p2m_frame_list, 
                     P2M_FL_ENTRIES * sizeof(xen_pfn_t)) )
    {
        PERROR("write: p2m_frame_list");
        goto out;
    }

    success = 1;

 out:
    
    if ( !success && p2m )
        munmap(p2m, P2M_FLL_ENTRIES * PAGE_SIZE);

    if ( live_p2m_frame_list_list )
        munmap(live_p2m_frame_list_list, PAGE_SIZE);

    if ( live_p2m_frame_list )
        munmap(live_p2m_frame_list, P2M_FLL_ENTRIES * PAGE_SIZE);

    if ( p2m_frame_list_list ) 
        free(p2m_frame_list_list);

    if ( p2m_frame_list ) 
        free(p2m_frame_list);

    return success ? p2m : NULL;
}

int xc_domain_save(int xc_handle, int io_fd, uint32_t dom, uint32_t max_iters,
                   uint32_t max_factor, uint32_t flags, int (*suspend)(void),
                   int hvm, void *(*init_qemu_maps)(int, unsigned), 
                   void (*qemu_flip_buffer)(int, int))
{
    xc_dominfo_t info;
    DECLARE_DOMCTL;

    int rc = 1, frc, i, j, last_iter, iter = 0;
    int live  = (flags & XCFLAGS_LIVE);
    int debug = (flags & XCFLAGS_DEBUG);
    int race = 0, sent_last_iter, skip_this_iter;
    int tmem_saved = 0;

    /* The new domain's shared-info frame number. */
    unsigned long shared_info_frame;

    /* A copy of the CPU context of the guest. */
    vcpu_guest_context_any_t ctxt;

    /* A table containing the type of each PFN (/not/ MFN!). */
    unsigned long *pfn_type = NULL;
    unsigned long *pfn_batch = NULL;

    /* A copy of one frame of guest memory. */
    char page[PAGE_SIZE];

    /* Live mapping of shared info structure */
    shared_info_any_t *live_shinfo = NULL;

    /* base of the region in which domain memory is mapped */
    unsigned char *region_base = NULL;

    /* bitmap of pages:
       - that should be sent this iteration (unless later marked as skip);
       - to skip this iteration because already dirty;
       - to fixup by sending at the end if not already resent; */
    unsigned long *to_send = NULL, *to_skip = NULL, *to_fix = NULL;

    xc_shadow_op_stats_t stats;

    unsigned long needed_to_fix = 0;
    unsigned long total_sent    = 0;

    uint64_t vcpumap = 1ULL;

    /* HVM: a buffer for holding HVM context */
    uint32_t hvm_buf_size = 0;
    uint8_t *hvm_buf = NULL;

    /* HVM: magic frames for ioreqs and xenstore comms. */
    uint64_t magic_pfns[3]; /* ioreq_pfn, bufioreq_pfn, store_pfn */

    unsigned long mfn;

    /* If no explicit control parameters given, use defaults */
    max_iters  = max_iters  ? : DEF_MAX_ITERS;
    max_factor = max_factor ? : DEF_MAX_FACTOR;

    initialize_mbit_rate();
    
    DPRINTF("-->xc_domain_save()\n");

    if ( !get_platform_info(xc_handle, dom,
                            &max_mfn, &hvirt_start, &pt_levels, &guest_width) )
    {
        ERROR("Unable to get platform info.");
        return 1;
    }
    
    DPRINTF("platform info, max_mfn: %lx, hvirt_start %lx, pt_lvevel %x, guest_width %x\n",
                            max_mfn, hvirt_start, pt_levels, guest_width);

    if ( xc_domain_getinfo(xc_handle, dom, 1, &info) != 1 )
    {
        ERROR("Could not get domain info");
        return 1;
    }

    shared_info_frame = info.shared_info_frame;    
    DPRINTF("shared_info_frame: 0x%lx\n", shared_info_frame);

    /* Map the shared info frame */
    if ( !hvm )
    {
        live_shinfo = xc_map_foreign_range(xc_handle, dom, PAGE_SIZE,
                                           PROT_READ, shared_info_frame);
        if ( !live_shinfo )
        {
            ERROR("Couldn't map live_shinfo");
            goto out;
        }
    }

    /* Get the size of the P2M table */
    p2m_size = xc_memory_op(xc_handle, XENMEM_maximum_gpfn, &dom) + 1;
    
    DPRINTF("p2m_size: 0x%lx", p2m_size);

    /* Domain is still running at this point */
    if ( live )
    {
        DPRINTF("live suspend\n");
        /* Live suspend. Enable log-dirty mode. */
        if ( xc_shadow_control(xc_handle, dom,
                               XEN_DOMCTL_SHADOW_OP_ENABLE_LOGDIRTY,
                               NULL, 0, NULL, 0, NULL) < 0 )
        {
            /* log-dirty already enabled? There's no test op,
               so attempt to disable then reenable it */
            frc = xc_shadow_control(xc_handle, dom, XEN_DOMCTL_SHADOW_OP_OFF,
                                    NULL, 0, NULL, 0, NULL);
            if ( frc >= 0 )
            {
                frc = xc_shadow_control(xc_handle, dom,
                                        XEN_DOMCTL_SHADOW_OP_ENABLE_LOGDIRTY,
                                        NULL, 0, NULL, 0, NULL);
            }
            
            if ( frc < 0 )
            {
                ERROR("Couldn't enable shadow mode (rc %d) (errno %d)", frc, errno );
                goto out;
            }
        }

        if ( hvm )
        {
            /* Get qemu-dm logging dirty pages too */
            void *seg = init_qemu_maps(dom, BITMAP_SIZE);
            qemu_bitmaps[0] = seg;
            qemu_bitmaps[1] = seg + BITMAP_SIZE;
            qemu_active = 0;
            qemu_non_active = 1;
        }
    }
    else
    {
        /* This is a non-live suspend. Suspend the domain .*/
        if ( suspend_and_state(suspend, xc_handle, io_fd, dom, &info) )
        {
            ERROR("Domain appears not to have suspended");
            goto out;
        }
    }

    last_iter = !live;

    /* pretend we sent all the pages last iteration */
    sent_last_iter = p2m_size;

    /* Setup to_send / to_fix and to_skip bitmaps */
    to_send = xg_memalign(PAGE_SIZE, ROUNDUP(BITMAP_SIZE, PAGE_SHIFT)); 
    to_fix  = calloc(1, BITMAP_SIZE);
    to_skip = xg_memalign(PAGE_SIZE, ROUNDUP(BITMAP_SIZE, PAGE_SHIFT)); 

    if ( !to_send || !to_fix || !to_skip )
    {
        ERROR("Couldn't allocate to_send array");
        goto out;
    }

    memset(to_send, 0xff, BITMAP_SIZE);

    if ( lock_pages(to_send, BITMAP_SIZE) )
    {
        ERROR("Unable to lock to_send");
        return 1;
    }

    /* (to fix is local only) */
    if ( lock_pages(to_skip, BITMAP_SIZE) )
    {
        ERROR("Unable to lock to_skip");
        return 1;
    }

    if ( hvm ) 
    {
        DPRINTF("Need another buffer for HVM context\n");
        /* Need another buffer for HVM context */
        hvm_buf_size = xc_domain_hvm_getcontext(xc_handle, dom, 0, 0);
        if ( hvm_buf_size == -1 )
        {
            ERROR("Couldn't get HVM context size from Xen");
            goto out;
        }
        hvm_buf = malloc(hvm_buf_size);
        if ( !hvm_buf )
        {
            ERROR("Couldn't allocate memory");
            goto out;
        }
    }

    analysis_phase(xc_handle, dom, p2m_size, to_skip, 0);

    pfn_type   = xg_memalign(PAGE_SIZE, ROUNDUP(
                              MAX_BATCH_SIZE * sizeof(*pfn_type), PAGE_SHIFT));
    pfn_batch  = calloc(MAX_BATCH_SIZE, sizeof(*pfn_batch));
    if ( (pfn_type == NULL) || (pfn_batch == NULL) )
    {
        ERROR("failed to alloc memory for pfn_type and/or pfn_batch arrays");
        errno = ENOMEM;
        goto out;
    }
    memset(pfn_type, 0,
           ROUNDUP(MAX_BATCH_SIZE * sizeof(*pfn_type), PAGE_SHIFT));

    if ( lock_pages(pfn_type, MAX_BATCH_SIZE * sizeof(*pfn_type)) )
    {
        ERROR("Unable to lock pfn_type array");
        goto out;
    }

    DPRINTF("Setup the mfn_to_pfn table mapping\n");
    /* Setup the mfn_to_pfn table mapping */
    if ( !(live_m2p = xc_map_m2p(xc_handle, max_mfn, PROT_READ)) )
    {
        ERROR("Failed to map live M2P table");
        goto out;
    }
    
    DPRINTF("Start writing out the saved-domain record.\n");
    /* Start writing out the saved-domain record. */
    if ( write_exact(io_fd, &p2m_size, sizeof(unsigned long)) )
    {
        PERROR("write: p2m_size");
        goto out;
    }

    if ( !hvm )
    {
        int err = 0;

        /* Map the P2M table, and write the list of P2M frames */
        live_p2m = map_and_save_p2m_table(xc_handle, io_fd, dom, 
                                          p2m_size, live_shinfo);
        if ( live_p2m == NULL )
        {
            ERROR("Failed to map/save the p2m frame list");
            goto out;
        }

        /*
         * Quick belt and braces sanity check.
         */
        
        for ( i = 0; i < p2m_size; i++ )
        {
            mfn = pfn_to_mfn(i);
            if( (mfn != INVALID_P2M_ENTRY) && (mfn_to_pfn(mfn) != i) )
            {
                DPRINTF("i=0x%x mfn=%lx live_m2p=%lx\n", i,
                        mfn, mfn_to_pfn(mfn));
                err++;
            }
        }
        DPRINTF("Had %d unexplained entries in p2m table\n", err);
    }

    print_stats(xc_handle, dom, 0, &stats, 0);

    tmem_saved = xc_tmem_save(xc_handle, dom, io_fd, live, -5);
    if ( tmem_saved == -1 )
    {
        ERROR("Error when writing to state file (tmem)");
        goto out;
    }

    /* Now write out each data page, canonicalising page tables as we go... */
    for ( ; ; )
    {
        unsigned int prev_pc, sent_this_iter, N, batch, run;

        iter++;
        sent_this_iter = 0;
        skip_this_iter = 0;
        prev_pc = 0;
        N = 0;

        DPRINTF("Saving memory pages: iter %d   0%%", iter);

        while ( N < p2m_size )
        {
            unsigned int this_pc = (N * 100) / p2m_size;

            if ( (this_pc - prev_pc) >= 5 )
            {
                DPRINTF("\b\b\b\b%3d%%", this_pc);
                prev_pc = this_pc;
            }

            if ( !last_iter )
            {
                /* Slightly wasteful to peek the whole array evey time,
                   but this is fast enough for the moment. */
                frc = xc_shadow_control(
                    xc_handle, dom, XEN_DOMCTL_SHADOW_OP_PEEK, to_skip, 
                    p2m_size, NULL, 0, NULL);
                if ( frc != p2m_size )
                {
                    ERROR("Error peeking shadow bitmap");
                    goto out;
                }
            }

            /* load pfn_type[] with the mfn of all the pages we're doing in
               this batch. */
            for  ( batch = 0;
                   (batch < MAX_BATCH_SIZE) && (N < p2m_size);
                   N++ )
            {
                int n = N;

                if ( debug )
                {
                    DPRINTF("%d pfn= %08lx mfn= %08lx %d",
                            iter, (unsigned long)n,
                            hvm ? 0 : pfn_to_mfn(n),
                            test_bit(n, to_send));
                    if ( !hvm && is_mapped(pfn_to_mfn(n)) )
                        DPRINTF("  [mfn]= %08lx",
                                mfn_to_pfn(pfn_to_mfn(n)&0xFFFFF));
                    DPRINTF("\n");
                }
                if ( !last_iter &&
                     test_bit(n, to_send) &&
                     test_bit(n, to_skip) )
                    skip_this_iter++; /* stats keeping */

                if ( !((test_bit(n, to_send) && !test_bit(n, to_skip)) ||
                       (test_bit(n, to_send) && last_iter) ||
                       (test_bit(n, to_fix)  && last_iter)) )
                    continue;

                /*
                ** we get here if:
                **  1. page is marked to_send & hasn't already been re-dirtied
                **  2. (ignore to_skip in last iteration)
                **  3. add in pages that still need fixup (net bufs)
                */

                pfn_batch[batch] = n;

                /* Hypercall interfaces operate in PFNs for HVM guests
                * and MFNs for PV guests */
                if ( hvm ) 
                    pfn_type[batch] = n;
                else
                    pfn_type[batch] = pfn_to_mfn(n);
                    
                if ( !is_mapped(pfn_type[batch]) )
                {
                    /*
                    ** not currently in psuedo-physical map -- set bit
                    ** in to_fix since we must send this page in last_iter
                    ** unless its sent sooner anyhow, or it never enters
                    ** pseudo-physical map (e.g. for ballooned down doms)
                    */
                    set_bit(n, to_fix);
                    continue;
                }

                if ( last_iter &&
                     test_bit(n, to_fix) &&
                     !test_bit(n, to_send) )
                {
                    needed_to_fix++;
                    DPRINTF("Fix! iter %d, pfn %x. mfn %lx\n",
                            iter, n, pfn_type[batch]);
                }
                
                clear_bit(n, to_fix);
                
                batch++;
            }

            if ( batch == 0 )
                goto skip; /* vanishingly unlikely... */

            region_base = xc_map_foreign_batch(
                xc_handle, dom, PROT_READ, pfn_type, batch);
            if ( region_base == NULL )
            {
                ERROR("map batch failed");
                goto out;
            }

            if ( hvm )
            {
                /* Look for and skip completely empty batches. */
                for ( j = 0; j < batch; j++ )
                    if ( (pfn_type[j] & XEN_DOMCTL_PFINFO_LTAB_MASK) !=
                         XEN_DOMCTL_PFINFO_XTAB )
                        break;
                if ( j == batch )
                {
                    munmap(region_base, batch*PAGE_SIZE);
                    continue; /* bail on this batch: no valid pages */
                }
            }
            else
            {
                /* Get page types */
                for ( j = 0; j < batch; j++ )
                    ((uint32_t *)pfn_type)[j] = pfn_type[j];
                if ( xc_get_pfn_type_batch(xc_handle, dom, batch,
                                           (uint32_t *)pfn_type) )
                {
                    ERROR("get_pfn_type_batch failed");
                    goto out;
                }
                for ( j = batch-1; j >= 0; j-- )
                    pfn_type[j] = ((uint32_t *)pfn_type)[j];

                for ( j = 0; j < batch; j++ )
                {
                    
                    if ( (pfn_type[j] & XEN_DOMCTL_PFINFO_LTAB_MASK) ==
                         XEN_DOMCTL_PFINFO_XTAB )
                    {
                        DPRINTF("type fail: page %i mfn %08lx\n", 
                                j, pfn_type[j]);
                        continue;
                    }
                    
                    if ( debug )
                        DPRINTF("%d pfn= %08lx mfn= %08lx [mfn]= %08lx"
                                " sum= %08lx\n",
                                iter,
                                (pfn_type[j] & XEN_DOMCTL_PFINFO_LTAB_MASK) |
                                pfn_batch[j],
                                pfn_type[j],
                                mfn_to_pfn(pfn_type[j] &
                                           ~XEN_DOMCTL_PFINFO_LTAB_MASK),
                                csum_page(region_base + (PAGE_SIZE*j)));
                    
                    /* canonicalise mfn->pfn */
                    pfn_type[j] = (pfn_type[j] & XEN_DOMCTL_PFINFO_LTAB_MASK) |
                        pfn_batch[j];
                }
            }

            if ( write_exact(io_fd, &batch, sizeof(unsigned int)) )
            {
                PERROR("Error when writing to state file (2)");
                goto out;
            }

            if ( write_exact(io_fd, pfn_type, sizeof(unsigned long)*batch) )
            {
                PERROR("Error when writing to state file (3)");
                goto out;
            }

            /* entering this loop, pfn_type is now in pfns (Not mfns) */
            run = 0;
            for ( j = 0; j < batch; j++ )
            {
                unsigned long pfn, pagetype;
                void *spage = (char *)region_base + (PAGE_SIZE*j);

                pfn      = pfn_type[j] & ~XEN_DOMCTL_PFINFO_LTAB_MASK;
                pagetype = pfn_type[j] &  XEN_DOMCTL_PFINFO_LTAB_MASK;

                if ( pagetype != 0 )
                {
                    /* If the page is not a normal data page, write out any
                       run of pages we may have previously acumulated */
                    if ( run )
                    {
                        if ( ratewrite(io_fd, live, 
                                       (char*)region_base+(PAGE_SIZE*(j-run)), 
                                       PAGE_SIZE*run) != PAGE_SIZE*run )
                        {
                            ERROR("Error when writing to state file (4a)"
                                  " (errno %d)", errno);
                            goto out;
                        }                        
                        run = 0;
                    }
                }

                /* skip pages that aren't present */
                if ( pagetype == XEN_DOMCTL_PFINFO_XTAB )
                    continue;

                pagetype &= XEN_DOMCTL_PFINFO_LTABTYPE_MASK;

                if ( (pagetype >= XEN_DOMCTL_PFINFO_L1TAB) &&
                     (pagetype <= XEN_DOMCTL_PFINFO_L4TAB) )
                {
                    /* We have a pagetable page: need to rewrite it. */
                    DPRINTF(" We have a pagetable page: need to rewrite it\n");
                    race = 
                        canonicalize_pagetable(pagetype, pfn, spage, page); 

                    if ( race && !live )
                    {
                        ERROR("Fatal PT race (pfn %lx, type %08lx)", pfn,
                              pagetype);
                        goto out;
                    }

                    if ( ratewrite(io_fd, live, page, PAGE_SIZE) != PAGE_SIZE )
                    {
                        ERROR("Error when writing to state file (4b)"
                              " (errno %d)", errno);
                        goto out;
                    }
                }
                else
                {
                    /* We have a normal page: accumulate it for writing. */
                    run++;
                }
            } /* end of the write out for this batch */

            if ( run )
            {
                /* write out the last accumulated run of pages */
                if ( ratewrite(io_fd, live, 
                               (char*)region_base+(PAGE_SIZE*(j-run)), 
                               PAGE_SIZE*run) != PAGE_SIZE*run )
                {
                    ERROR("Error when writing to state file (4c)"
                          " (errno %d)", errno);
                    goto out;
                }                        
            }

            sent_this_iter += batch;

            munmap(region_base, batch*PAGE_SIZE);

        } /* end of this while loop for this iteration */

      skip:

        total_sent += sent_this_iter;

        DPRINTF("\r %d: sent %d, skipped %d, ",
                iter, sent_this_iter, skip_this_iter );

        if ( last_iter )
        {
            print_stats( xc_handle, dom, sent_this_iter, &stats, 1);

            DPRINTF("Total pages sent= %ld (%.2fx)\n",
                    total_sent, ((float)total_sent)/p2m_size );
            DPRINTF("(of which %ld were fixups)\n", needed_to_fix  );
        }

        if ( last_iter && debug )
        {
            int minusone = -1;
            memset(to_send, 0xff, BITMAP_SIZE);
            debug = 0;
            DPRINTF("Entering debug resend-all mode\n");

            /* send "-1" to put receiver into debug mode */
            if ( write_exact(io_fd, &minusone, sizeof(int)) )
            {
                PERROR("Error when writing to state file (6)");
                goto out;
            }

            continue;
        }

        if ( last_iter )
            break;

        if ( live )
        {
            if ( ((sent_this_iter > sent_last_iter) && RATE_IS_MAX()) ||
                 (iter >= max_iters) ||
                 (sent_this_iter+skip_this_iter < 50) ||
                 (total_sent > p2m_size*max_factor) )
            {
                DPRINTF("Start last iteration\n");
                last_iter = 1;

                if ( suspend_and_state(suspend, xc_handle, io_fd, dom, &info) )
                {
                    ERROR("Domain appears not to have suspended");
                    goto out;
                }

                DPRINTF("SUSPEND shinfo %08lx\n", info.shared_info_frame);
                if ( (tmem_saved > 0) &&
                     (xc_tmem_save_extra(xc_handle,dom,io_fd,-6) == -1) )
                {
                        ERROR("Error when writing to state file (tmem)");
                        goto out;
                }

            }

            if ( xc_shadow_control(xc_handle, dom, 
                                   XEN_DOMCTL_SHADOW_OP_CLEAN, to_send, 
                                   p2m_size, NULL, 0, &stats) != p2m_size )
            {
                ERROR("Error flushing shadow PT");
                goto out;
            }

            if ( hvm ) 
            {
                /* Pull in the dirty bits from qemu-dm too */
                if ( !last_iter )
                {
                    qemu_active = qemu_non_active;
                    qemu_non_active = qemu_active ? 0 : 1;
                    qemu_flip_buffer(dom, qemu_active);
                    for ( j = 0; j < BITMAP_SIZE / sizeof(unsigned long); j++ )
                    {
                        to_send[j] |= qemu_bitmaps[qemu_non_active][j];
                        qemu_bitmaps[qemu_non_active][j] = 0;
                    }
                }
                else
                {
                    for ( j = 0; j < BITMAP_SIZE / sizeof(unsigned long); j++ )
                        to_send[j] |= qemu_bitmaps[qemu_active][j];
                }
            }

            sent_last_iter = sent_this_iter;

            print_stats(xc_handle, dom, sent_this_iter, &stats, 1);

        }
    } /* end of infinite for loop */

    DPRINTF("All memory is saved\n");

    {
        struct {
            int minustwo;
            int max_vcpu_id;
            uint64_t vcpumap;
        } chunk = { -2, info.max_vcpu_id };

        if ( info.max_vcpu_id >= 64 )
        {
            ERROR("Too many VCPUS in guest!");
            goto out;
        }

        for ( i = 1; i <= info.max_vcpu_id; i++ )
        {
            xc_vcpuinfo_t vinfo;
            if ( (xc_vcpu_getinfo(xc_handle, dom, i, &vinfo) == 0) &&
                 vinfo.online )
                vcpumap |= 1ULL << i;
        }

        chunk.vcpumap = vcpumap;
        if ( write_exact(io_fd, &chunk, sizeof(chunk)) )
        {
            PERROR("Error when writing to state file");
            goto out;
        }
    }

    if ( hvm )
    {
        struct {
            int id;
            uint32_t pad;
            uint64_t data;
        } chunk = { 0, };

        chunk.id = -3;
        xc_get_hvm_param(xc_handle, dom, HVM_PARAM_IDENT_PT,
                         (unsigned long *)&chunk.data);

        if ( (chunk.data != 0) &&
             write_exact(io_fd, &chunk, sizeof(chunk)) )
        {
            PERROR("Error when writing the ident_pt for EPT guest");
            goto out;
        }

        chunk.id = -4;
        xc_get_hvm_param(xc_handle, dom, HVM_PARAM_VM86_TSS,
                         (unsigned long *)&chunk.data);

        if ( (chunk.data != 0) &&
             write_exact(io_fd, &chunk, sizeof(chunk)) )
        {
            PERROR("Error when writing the vm86 TSS for guest");
            goto out;
        }
    }

    /* Zero terminate */
    i = 0;
    if ( write_exact(io_fd, &i, sizeof(int)) )
    {
        PERROR("Error when writing to state file (6')");
        goto out;
    }

    if ( hvm ) 
    {
        uint32_t rec_size;

        /* Save magic-page locations. */
        memset(magic_pfns, 0, sizeof(magic_pfns));
        xc_get_hvm_param(xc_handle, dom, HVM_PARAM_IOREQ_PFN,
                         (unsigned long *)&magic_pfns[0]);
        xc_get_hvm_param(xc_handle, dom, HVM_PARAM_BUFIOREQ_PFN,
                         (unsigned long *)&magic_pfns[1]);
        xc_get_hvm_param(xc_handle, dom, HVM_PARAM_STORE_PFN,
                         (unsigned long *)&magic_pfns[2]);
        if ( write_exact(io_fd, magic_pfns, sizeof(magic_pfns)) )
        {
            PERROR("Error when writing to state file (7)");
            goto out;
        }

        /* Get HVM context from Xen and save it too */
        if ( (rec_size = xc_domain_hvm_getcontext(xc_handle, dom, hvm_buf, 
                                                  hvm_buf_size)) == -1 )
        {
            ERROR("HVM:Could not get hvm buffer");
            goto out;
        }
        
        if ( write_exact(io_fd, &rec_size, sizeof(uint32_t)) )
        {
            PERROR("error write hvm buffer size");
            goto out;
        }
        
        if ( write_exact(io_fd, hvm_buf, rec_size) )
        {
            PERROR("write HVM info failed!\n");
            goto out;
        }
        
        /* HVM guests are done now */
        rc = 0;
        goto out;
    }

    /* PV guests only from now on */

    /* Send through a list of all the PFNs that were not in map at the close */
    {
        unsigned int i,j;
        unsigned long pfntab[1024];

        for ( i = 0, j = 0; i < p2m_size; i++ )
        {
            if ( !is_mapped(pfn_to_mfn(i)) )
                j++;
        }

        if ( write_exact(io_fd, &j, sizeof(unsigned int)) )
        {
            PERROR("Error when writing to state file (6a)");
            goto out;
        }

        for ( i = 0, j = 0; i < p2m_size; )
        {
            if ( !is_mapped(pfn_to_mfn(i)) )
                pfntab[j++] = i;

            i++;
            if ( (j == 1024) || (i == p2m_size) )
            {
                if ( write_exact(io_fd, &pfntab, sizeof(unsigned long)*j) )
                {
                    PERROR("Error when writing to state file (6b)");
                    goto out;
                }
                j = 0;
            }
        }
    }

    if ( xc_vcpu_getcontext(xc_handle, dom, 0, &ctxt) )
    {
        ERROR("Could not get vcpu context");
        goto out;
    }

    /* Canonicalise the suspend-record frame number. */
    mfn = GET_FIELD(&ctxt, user_regs.edx);
    if ( !MFN_IS_IN_PSEUDOPHYS_MAP(mfn) )
    {
        ERROR("Suspend record is not in range of pseudophys map");
        goto out;
    }
    SET_FIELD(&ctxt, user_regs.edx, mfn_to_pfn(mfn));

    for ( i = 0; i <= info.max_vcpu_id; i++ )
    {
        if ( !(vcpumap & (1ULL << i)) )
            continue;

        if ( (i != 0) && xc_vcpu_getcontext(xc_handle, dom, i, &ctxt) )
        {
            ERROR("No context for VCPU%d", i);
            goto out;
        }

        /* Canonicalise each GDT frame number. */
        for ( j = 0; (512*j) < GET_FIELD(&ctxt, gdt_ents); j++ )
        {
            mfn = GET_FIELD(&ctxt, gdt_frames[j]);
            if ( !MFN_IS_IN_PSEUDOPHYS_MAP(mfn) )
            {
                ERROR("GDT frame is not in range of pseudophys map");
                goto out;
            }
            SET_FIELD(&ctxt, gdt_frames[j], mfn_to_pfn(mfn));
        }

        /* Canonicalise the page table base pointer. */
        if ( !MFN_IS_IN_PSEUDOPHYS_MAP(UNFOLD_CR3(
                                           GET_FIELD(&ctxt, ctrlreg[3]))) )
        {
            ERROR("PT base is not in range of pseudophys map");
            goto out;
        }
        SET_FIELD(&ctxt, ctrlreg[3], 
            FOLD_CR3(mfn_to_pfn(UNFOLD_CR3(GET_FIELD(&ctxt, ctrlreg[3])))));

        /* Guest pagetable (x86/64) stored in otherwise-unused CR1. */
        if ( (pt_levels == 4) && ctxt.x64.ctrlreg[1] )
        {
            if ( !MFN_IS_IN_PSEUDOPHYS_MAP(UNFOLD_CR3(ctxt.x64.ctrlreg[1])) )
            {
                ERROR("PT base is not in range of pseudophys map");
                goto out;
            }
            /* Least-significant bit means 'valid PFN'. */
            ctxt.x64.ctrlreg[1] = 1 |
                FOLD_CR3(mfn_to_pfn(UNFOLD_CR3(ctxt.x64.ctrlreg[1])));
        }

        if ( write_exact(io_fd, &ctxt, ((guest_width==8) 
                                        ? sizeof(ctxt.x64) 
                                        : sizeof(ctxt.x32))) )
        {
            PERROR("Error when writing to state file (1)");
            goto out;
        }

        domctl.cmd = XEN_DOMCTL_get_ext_vcpucontext;
        domctl.domain = dom;
        domctl.u.ext_vcpucontext.vcpu = i;
        if ( xc_domctl(xc_handle, &domctl) < 0 )
        {
            ERROR("No extended context for VCPU%d", i);
            goto out;
        }
        if ( write_exact(io_fd, &domctl.u.ext_vcpucontext, 128) )
        {
            PERROR("Error when writing to state file (2)");
            goto out;
        }
    }

    /*
     * Reset the MFN to be a known-invalid value. See map_frame_list_list().
     */
    memcpy(page, live_shinfo, PAGE_SIZE);
    SET_FIELD(((shared_info_any_t *)page), 
              arch.pfn_to_mfn_frame_list_list, 0);
    if ( write_exact(io_fd, page, PAGE_SIZE) )
    {
        PERROR("Error when writing to state file (1)");
        goto out;
    }

    /* Success! */
    rc = 0;

 out:

    if ( tmem_saved != 0 && live )
        xc_tmem_save_done(xc_handle, dom);

    if ( live )
    {
        if ( xc_shadow_control(xc_handle, dom, 
                               XEN_DOMCTL_SHADOW_OP_OFF,
                               NULL, 0, NULL, 0, NULL) < 0 )
            DPRINTF("Warning - couldn't disable shadow mode");
    }

    /* Flush last write and discard cache for file. */
    discard_file_cache(io_fd, 1 /* flush */);

    if ( live_shinfo )
        munmap(live_shinfo, PAGE_SIZE);

    if ( live_p2m )
        munmap(live_p2m, P2M_FLL_ENTRIES * PAGE_SIZE);

    if ( live_m2p )
        munmap(live_m2p, M2P_SIZE(max_mfn));

    free(pfn_type);
    free(pfn_batch);
    free(to_send);
    free(to_fix);
    free(to_skip);

    DPRINTF("Save exit rc=%d\n",rc);

    return !!rc;
}

/*
 * Local variables:
 * mode: C
 * c-set-style: "BSD"
 * c-basic-offset: 4
 * tab-width: 4
 * indent-tabs-mode: nil
 * End:
 */

[-- Attachment #5: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Error restoring DomU when using GPLPV

[ANNIE LI]

Hi,

>> Yes, that's weird. Do you know what condition causes guest memory allocation
>> failure on xc_domain_restore? Is it due to hitting the guest maxmem limit in
>> Xen? If so, is maxmem the same value across multiple iterations of
>> save/restore or migration?
>>   
> Sorry, i have no idea about it. Maybe I need to print more log in 
> for(;;) in xc_domain_restore to see what is the difference between 
> without and with balooning down pages.
I did some migration test on linux/windows PVHVM on Xen3.4.

*  I printed the value of  "pfn      = region_pfn_type[i] & 
~XEN_DOMCTL_PFINFO_LTAB_MASK;"  in xc_domain_restore.c. When restoring 
fails with error "Failed allocation for dom 2: 33 extents of order 0", 
the value of pfn is less than that of restoring successfully. So i think 
it should not due to hitting the guest maxmem limit in Xen. Is it correct?

* After comparing difference between with and without ballooning down 
(gnttab+shinfo) pages, i find that:

If the windows pv driver balloon down those pages, there will be more 
pages with XEN_DOMCTL_PFINFO_XTAB type in saving process. Furthermore, 
more bogus/unmapped page are skipped in restoring process.
If the winpv driver do not balloon down those pages,  there are only a 
little such pages with XEN_DOMCTL_PFINFO_XTAB type to be processed 
during save/restore process.

* Another result about winpv driver with ballooning down those pages
 When doing save/restore for the second time, i find p2msize in 
restoring process become 0xfefff which is less than the normal size 
0x100000.

Any suggestion about those test result? Or any idea to resolve this 
problem in winpv or xen?

Thanks
Annie.

Re: Error restoring DomU when using GPLPV

[ANNIE LI]

Hi,

> I did some migration test on linux/windows PVHVM on Xen3.4.
>
> *  I printed the value of  "pfn      = region_pfn_type[i] & 
> ~XEN_DOMCTL_PFINFO_LTAB_MASK;"  in xc_domain_restore.c. When restoring 
> fails with error "Failed allocation for dom 2: 33 extents of order 0", 
> the value of pfn is less than that of restoring successfully. So i 
> think it should not due to hitting the guest maxmem limit in Xen. Is 
> it correct?
>
> * After comparing difference between with and without ballooning down 
> (gnttab+shinfo) pages, i find that:
>
> If the windows pv driver balloon down those pages, there will be more 
> pages with XEN_DOMCTL_PFINFO_XTAB type in saving process. Furthermore, 
> more bogus/unmapped page are skipped in restoring process.
> If the winpv driver do not balloon down those pages,  there are only a 
> little such pages with XEN_DOMCTL_PFINFO_XTAB type to be processed 
> during save/restore process.
>
> * Another result about winpv driver with ballooning down those pages
> When doing save/restore for the second time, i find p2msize in 
> restoring process become 0xfefff which is less than the normal size 
> 0x100000.
>
> Any suggestion about those test result? Or any idea to resolve this 
> problem in winpv or xen?
I did more save/restore test, and compare the logs between linux and 
windows PVHVM. Those two vms have same memory size.
It seems that most log of them are identical, but the only difference 
between them is also connected with XEN_DOMCTL_PFINFO_XTAB type pages. 
 From the comments in the code, XEN_DOMCTL_PFINFO_XTAB type means 
invalid page.

For linux PVHVM, it have more 31 invalid pages than windows PVHVM during 
saving process. 
In for ( j = 0; j < batch; j++ ) of xc_domain_save.c, linux PVHVM will 
take those pages with pfn value between f2003 and f2021 as invalid 
pages. But windows PVHVM took those pages as normal pages.

Then in restoring process,  more memory are allocated for windows PVHVM 
than linux PVHVM. For example:
When windows PVHVM hit the issue: "Failed allocation for dom 2: 33 
extents of order 0",  and the log shows that nr_mfns before 
"xc_domain_memory_populate_physmap" is 33. However it is only 14 at the 
same process of restoring linux PVHVM.

It seems there should be more invalid pages in the saving process of 
windows PVHVM. But i failed to get the root cause of it. Any suggestions?

Thanks
Annie.

Re: Error restoring DomU when using GPLPV

[ANNIE LI]

Hi

As we discussed in this thread before, all windows PVHVM should fail to 
migration on Xen3.4.
Can anyone tell me whether Citrix windows pv driver 
save/restore/migration work properly or not on Xen3.4?

Thanks
Annie.

Re: Error restoring DomU when using GPLPV

[ANNIE LI]

Hi

It seems this problem is connected with gnttab, not shareinfo.
I changed some code about grant table in winpv driver (not using balloon 
down shinfo+gnttab method), save/restore/migration can work properly on 
Xen3.4 now.

What i changed is winpv driver use hypercall XENMEM_add_to_physmap to 
map corresponding grant tables which devices require, instead of mapping 
all 32 pages grant table during initialization.  It seems those extra 
grant table mapping cause this problem.

Thanks
Annie.

Re: Error restoring DomU when using GPLPV

[ANNIE LI]

> It seems this problem is connected with gnttab, not shareinfo.
> I changed some code about grant table in winpv driver (not using 
> balloon down shinfo+gnttab method), save/restore/migration can work 
> properly on Xen3.4 now.
>
> What i changed is winpv driver use hypercall XENMEM_add_to_physmap to 
> map corresponding grant tables which devices require, instead of 
> mapping all 32 pages grant table during initialization.  It seems 
> those extra grant table mapping cause this problem. 

Wondering whether those extra grant table mapping is the root cause of 
the migration problem? or by luck as linux PVHVM too?

Thanks
Annie.

RE: Error restoring DomU when using GPLPV

[Dan Magenheimer]

I think I've tracked down the cause of this problem
in the hypervisor, but am unsure how to best fix it.

In tools/libxc/xc_domain_save.c, the static variable p2m_size
is said to be "number of pfns this guest has (i.e. number of
entries in the P2M)".  But apparently p2m_size is getting
set to a very large number (0x100000) regardless of the
maximum psuedophysical memory for the hvm guest.  As a result,
some "magic" pages in the 0xf0000-0xfefff range are getting
placed in the save file.  But since they are not "real"
pages, the restore process runs beyond the maximum number
of physical pages allowed for the domain and fails.
(The gpfn of the last 24 pages saved are f2020, fc000-fc012,
feffb, feffc, feffd, feffe.)

p2m_size is set in "save" with a call to a memory_op hypercall
(XENMEM_maximum_gpfn) which for an hvm domain returns
d->arch.p2m->max_mapped_pfn.  I suspect that the meaning
of max_mapped_pfn changed at some point to more match
its name, but this changed the semantics of the hypercall
as used by xc_domain_restore, resulting in this curious
problem.

Any thoughts on how to fix this?

> -----Original Message-----
> From: Annie Li 
> Sent: Tuesday, September 01, 2009 10:27 PM
> To: Keir Fraser
> Cc: Joshua West; James Harper; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Error restoring DomU when using GPLPV
> 
> 
> 
> > It seems this problem is connected with gnttab, not shareinfo.
> > I changed some code about grant table in winpv driver (not using 
> > balloon down shinfo+gnttab method), save/restore/migration can work 
> > properly on Xen3.4 now.
> >
> > What i changed is winpv driver use hypercall 
> XENMEM_add_to_physmap to 
> > map corresponding grant tables which devices require, instead of 
> > mapping all 32 pages grant table during initialization.  It seems 
> > those extra grant table mapping cause this problem. 
> 
> Wondering whether those extra grant table mapping is the root 
> cause of 
> the migration problem? or by luck as linux PVHVM too?
> 
> Thanks
> Annie.
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

RE: Error restoring DomU when using GPLPV

[Dan Magenheimer]

On further debugging, it appears that the
p2m_size may be OK, but there's something about
those 24 "magic" gpfns that isn't quite right.

> -----Original Message-----
> From: Dan Magenheimer 
> Sent: Friday, September 04, 2009 3:29 PM
> To: Wayne Gong; Annie Li; Keir Fraser
> Cc: Joshua West; James Harper; xen-devel@lists.xensource.com
> Subject: RE: [Xen-devel] Error restoring DomU when using GPLPV
> 
> 
> I think I've tracked down the cause of this problem
> in the hypervisor, but am unsure how to best fix it.
> 
> In tools/libxc/xc_domain_save.c, the static variable p2m_size
> is said to be "number of pfns this guest has (i.e. number of
> entries in the P2M)".  But apparently p2m_size is getting
> set to a very large number (0x100000) regardless of the
> maximum psuedophysical memory for the hvm guest.  As a result,
> some "magic" pages in the 0xf0000-0xfefff range are getting
> placed in the save file.  But since they are not "real"
> pages, the restore process runs beyond the maximum number
> of physical pages allowed for the domain and fails.
> (The gpfn of the last 24 pages saved are f2020, fc000-fc012,
> feffb, feffc, feffd, feffe.)
> 
> p2m_size is set in "save" with a call to a memory_op hypercall
> (XENMEM_maximum_gpfn) which for an hvm domain returns
> d->arch.p2m->max_mapped_pfn.  I suspect that the meaning
> of max_mapped_pfn changed at some point to more match
> its name, but this changed the semantics of the hypercall
> as used by xc_domain_restore, resulting in this curious
> problem.
> 
> Any thoughts on how to fix this?
> 
> > -----Original Message-----
> > From: Annie Li 
> > Sent: Tuesday, September 01, 2009 10:27 PM
> > To: Keir Fraser
> > Cc: Joshua West; James Harper; xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] Error restoring DomU when using GPLPV
> > 
> > 
> > 
> > > It seems this problem is connected with gnttab, not shareinfo.
> > > I changed some code about grant table in winpv driver (not using 
> > > balloon down shinfo+gnttab method), 
> save/restore/migration can work 
> > > properly on Xen3.4 now.
> > >
> > > What i changed is winpv driver use hypercall 
> > XENMEM_add_to_physmap to 
> > > map corresponding grant tables which devices require, instead of 
> > > mapping all 32 pages grant table during initialization.  It seems 
> > > those extra grant table mapping cause this problem. 
> > 
> > Wondering whether those extra grant table mapping is the root 
> > cause of 
> > the migration problem? or by luck as linux PVHVM too?
> > 
> > Thanks
> > Annie.
> > 
> > 
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
> >
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>

Re: Error restoring DomU when using GPLPV

[Mukesh Rathor]

Keir Fraser wrote:
> On 04/08/2009 12:34, "James Harper" <james.harper@bendigoit.com.au> wrote:
> 
>>> Like I said before -- unmapping the gnttab pages I think will not help
>> you
>>> for live migration, but I suppose it is a reasonable thing to do
>> anyway. For
>>> live migration I think xc_domain_save needs t get a bit smarter about
>>> Xenheap pages in HVM guests.
>> Understood. Do you have any idea about why it worked fine under 3.3.x
>> but not 3.4.x?
> 
> The bit of code in 3.3's xc_domain_save.c that is commented "Skip PFNs that
> aren't really there" is removed in 3.4. That will be the reason.
> 
>  -- Keir

Hi,

I started looking at this couple days ago, and finally understand
what's going on. In our case, win migration/save-restore just fails, as
Annie/Wayne had posted.

In the short run, since frames for vga etc are skipped anyways, can we
just put the above change back in libxc (xen 3.4) and be ok?

thanks,
Mukesh


changeset:   18383:dade7f0bdc8d
user:        Keir Fraser <keir.fraser@citrix.com>
date:        Wed Aug 27 14:53:39 2008 +0100
summary:     hvm: Use main memory for video memory.

diff -r 2397555ebcc2 -r dade7f0bdc8d tools/libxc/xc_domain_save.c
--- a/tools/libxc/xc_domain_save.c      Wed Aug 27 13:31:01 2008 +0100
+++ b/tools/libxc/xc_domain_save.c      Wed Aug 27 14:53:39 2008 +0100
@@ -1111,12 +1111,6 @@
                         (test_bit(n, to_fix)  && last_iter)) )
                      continue;

-                /* Skip PFNs that aren't really there */
-                if ( hvm && ((n >= 0xa0 && n < 0xc0) /* VGA hole */
-                             || (n >= (HVM_BELOW_4G_MMIO_START >> PAGE_SHIFT)
-                                 && n < (1ULL<<32) >> PAGE_SHIFT)) /* MMIO */ )
-                    continue;
-

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 05/09/2009 05:02, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:

>> The bit of code in 3.3's xc_domain_save.c that is commented "Skip PFNs that
>> aren't really there" is removed in 3.4. That will be the reason.
> 
> I started looking at this couple days ago, and finally understand
> what's going on. In our case, win migration/save-restore just fails, as
> Annie/Wayne had posted.
> 
> In the short run, since frames for vga etc are skipped anyways, can we
> just put the above change back in libxc (xen 3.4) and be ok?

I don't think vga frames are skipped. I think vga now gets saved by
xc_domain_save.

Also some real RAM is up in that area these days. Like ACPI tables for
example.

 -- Keir

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

Not all those pages are special. Frames fc0xx will be ACPI tables, resident
in ordinary guest memory pages, for example. Only the Xen-heap pages are
special and need to be (1) skipped; or (2) unmapped by the HVMPV drivers on
suspend; or (3) accounted for by HVMPV drivers by unmapping and freeing an
equal number of domain-heap pages. (1) is 'nicest' but actually a bit of a
pain to implement; (2) won't work well for live migration, where the pages
wouldn't get unmapped by the drivers until the last round of page copying;
and (3) was apparently tried by Annie but didn't work? I'm curious why (3)
didn't work - I can't explain that.

 -- Keir

On 05/09/2009 00:02, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> On further debugging, it appears that the
> p2m_size may be OK, but there's something about
> those 24 "magic" gpfns that isn't quite right.
> 
>> -----Original Message-----
>> From: Dan Magenheimer
>> Sent: Friday, September 04, 2009 3:29 PM
>> To: Wayne Gong; Annie Li; Keir Fraser
>> Cc: Joshua West; James Harper; xen-devel@lists.xensource.com
>> Subject: RE: [Xen-devel] Error restoring DomU when using GPLPV
>> 
>> 
>> I think I've tracked down the cause of this problem
>> in the hypervisor, but am unsure how to best fix it.
>> 
>> In tools/libxc/xc_domain_save.c, the static variable p2m_size
>> is said to be "number of pfns this guest has (i.e. number of
>> entries in the P2M)".  But apparently p2m_size is getting
>> set to a very large number (0x100000) regardless of the
>> maximum psuedophysical memory for the hvm guest.  As a result,
>> some "magic" pages in the 0xf0000-0xfefff range are getting
>> placed in the save file.  But since they are not "real"
>> pages, the restore process runs beyond the maximum number
>> of physical pages allowed for the domain and fails.
>> (The gpfn of the last 24 pages saved are f2020, fc000-fc012,
>> feffb, feffc, feffd, feffe.)
>> 
>> p2m_size is set in "save" with a call to a memory_op hypercall
>> (XENMEM_maximum_gpfn) which for an hvm domain returns
>> d->arch.p2m->max_mapped_pfn.  I suspect that the meaning
>> of max_mapped_pfn changed at some point to more match
>> its name, but this changed the semantics of the hypercall
>> as used by xc_domain_restore, resulting in this curious
>> problem.
>> 
>> Any thoughts on how to fix this?
>> 
>>> -----Original Message-----
>>> From: Annie Li 
>>> Sent: Tuesday, September 01, 2009 10:27 PM
>>> To: Keir Fraser
>>> Cc: Joshua West; James Harper; xen-devel@lists.xensource.com
>>> Subject: Re: [Xen-devel] Error restoring DomU when using GPLPV
>>> 
>>> 
>>> 
>>>> It seems this problem is connected with gnttab, not shareinfo.
>>>> I changed some code about grant table in winpv driver (not using
>>>> balloon down shinfo+gnttab method),
>> save/restore/migration can work
>>>> properly on Xen3.4 now.
>>>> 
>>>> What i changed is winpv driver use hypercall
>>> XENMEM_add_to_physmap to
>>>> map corresponding grant tables which devices require, instead of
>>>> mapping all 32 pages grant table during initialization.  It seems
>>>> those extra grant table mapping cause this problem.
>>> 
>>> Wondering whether those extra grant table mapping is the root
>>> cause of 
>>> the migration problem? or by luck as linux PVHVM too?
>>> 
>>> Thanks
>>> Annie.
>>> 
>>> 
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>> 

Re: Error restoring DomU when using GPLPV

[ANNIE LI]

[-- Attachment #1.1: Type: text/plain, Size: 4204 bytes --]

Yes. About (3), my test result is save/restore can work only once if 
ballooning down pages when driver
 first load. But it works if ballooning down when driver load every times.

Thanks
Annie.

Keir Fraser wrote:
> Not all those pages are special. Frames fc0xx will be ACPI tables, resident
> in ordinary guest memory pages, for example. Only the Xen-heap pages are
> special and need to be (1) skipped; or (2) unmapped by the HVMPV drivers on
> suspend; or (3) accounted for by HVMPV drivers by unmapping and freeing an
> equal number of domain-heap pages. (1) is 'nicest' but actually a bit of a
> pain to implement; (2) won't work well for live migration, where the pages
> wouldn't get unmapped by the drivers until the last round of page copying;
> and (3) was apparently tried by Annie but didn't work? I'm curious why (3)
> didn't work - I can't explain that.
>
>  -- Keir
>
> On 05/09/2009 00:02, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:
>
>   
>> On further debugging, it appears that the
>> p2m_size may be OK, but there's something about
>> those 24 "magic" gpfns that isn't quite right.
>>
>>     
>>> -----Original Message-----
>>> From: Dan Magenheimer
>>> Sent: Friday, September 04, 2009 3:29 PM
>>> To: Wayne Gong; Annie Li; Keir Fraser
>>> Cc: Joshua West; James Harper; xen-devel@lists.xensource.com
>>> Subject: RE: [Xen-devel] Error restoring DomU when using GPLPV
>>>
>>>
>>> I think I've tracked down the cause of this problem
>>> in the hypervisor, but am unsure how to best fix it.
>>>
>>> In tools/libxc/xc_domain_save.c, the static variable p2m_size
>>> is said to be "number of pfns this guest has (i.e. number of
>>> entries in the P2M)".  But apparently p2m_size is getting
>>> set to a very large number (0x100000) regardless of the
>>> maximum psuedophysical memory for the hvm guest.  As a result,
>>> some "magic" pages in the 0xf0000-0xfefff range are getting
>>> placed in the save file.  But since they are not "real"
>>> pages, the restore process runs beyond the maximum number
>>> of physical pages allowed for the domain and fails.
>>> (The gpfn of the last 24 pages saved are f2020, fc000-fc012,
>>> feffb, feffc, feffd, feffe.)
>>>
>>> p2m_size is set in "save" with a call to a memory_op hypercall
>>> (XENMEM_maximum_gpfn) which for an hvm domain returns
>>> d->arch.p2m->max_mapped_pfn.  I suspect that the meaning
>>> of max_mapped_pfn changed at some point to more match
>>> its name, but this changed the semantics of the hypercall
>>> as used by xc_domain_restore, resulting in this curious
>>> problem.
>>>
>>> Any thoughts on how to fix this?
>>>
>>>       
>>>> -----Original Message-----
>>>> From: Annie Li 
>>>> Sent: Tuesday, September 01, 2009 10:27 PM
>>>> To: Keir Fraser
>>>> Cc: Joshua West; James Harper; xen-devel@lists.xensource.com
>>>> Subject: Re: [Xen-devel] Error restoring DomU when using GPLPV
>>>>
>>>>
>>>>
>>>>         
>>>>> It seems this problem is connected with gnttab, not shareinfo.
>>>>> I changed some code about grant table in winpv driver (not using
>>>>> balloon down shinfo+gnttab method),
>>>>>           
>>> save/restore/migration can work
>>>       
>>>>> properly on Xen3.4 now.
>>>>>
>>>>> What i changed is winpv driver use hypercall
>>>>>           
>>>> XENMEM_add_to_physmap to
>>>>         
>>>>> map corresponding grant tables which devices require, instead of
>>>>> mapping all 32 pages grant table during initialization.  It seems
>>>>> those extra grant table mapping cause this problem.
>>>>>           
>>>> Wondering whether those extra grant table mapping is the root
>>>> cause of 
>>>> the migration problem? or by luck as linux PVHVM too?
>>>>
>>>> Thanks
>>>> Annie.
>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>>>
>>>>         
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>
>>>       
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
>   

[-- Attachment #1.2: Type: text/html, Size: 5558 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Error restoring DomU when using GPLPV

[Mukesh Rathor]

Ok, I've been looking at this and figured what's going on. Annie's problem
lies in not remapping the grant frames post migration. Hence the leak,
tot_pages goes up every time until migration fails. On linux, remapping
is where the frames created by restore (for heap pfn's), get freed back to
the dom heap, is what I found.  So that's a fix to be made on win
pv driver side.

Now back to orig problem. As you already know, because libxc is not
skipping heap pages, tot_pages in struct domain{} temporarily goes up
by (shared-info-frame + gnt-frames) until guest remaps these pages.
Hence, migration fails if
       (max_pages - tot_pages) < (shared-info-frame + gnt-frames).

Occassionally, I see tot_pages nearly same as max_pages, and I don't
know of all ways that may happen or what causes that to happen
(by default, i see tot_pages short by 21).

Anyways, of two solutions:

1. Always balloon down, shinfo+gnttab frames: This needs to be done just
    once during load, right? I'm not sure how it would work tho if mem gets
    ballooned up subsequently. I suppose the driver will have to intercept
    every increase in reservation and balloon down everytime?

    Also, balloon down during suspend call would prob be too late, right?

2. libxc fix: I wonder how much work this will be. Good thing here is,
    it'll take care of both linux and PV HVM guests avoiding driver
    updates in many versions, and hence appealing to us. Can we somehow
    mark the frames special to be skipped? Looking at biiig xc_domain_save
    function, not sure in case of HVM, how pfn_type gets set. May be before the
    outer loop, it could ask hyp for all xen heap page list, but then what if a
    new page gets added to the list in between.....


Also, unfortunately, the failure case is not handled properly sometimes.
If migration fails after suspend, then no way to get the guest
back. I even noticed, the guest disappeared totally from both source and
target when failed, couple times of several dozen migrations I did.


thanks,
Mukesh



Keir Fraser wrote:
> Not all those pages are special. Frames fc0xx will be ACPI tables, resident
> in ordinary guest memory pages, for example. Only the Xen-heap pages are
> special and need to be (1) skipped; or (2) unmapped by the HVMPV drivers on
> suspend; or (3) accounted for by HVMPV drivers by unmapping and freeing an
> equal number of domain-heap pages. (1) is 'nicest' but actually a bit of a
> pain to implement; (2) won't work well for live migration, where the pages
> wouldn't get unmapped by the drivers until the last round of page copying;
> and (3) was apparently tried by Annie but didn't work? I'm curious why (3)
> didn't work - I can't explain that.
> 
>  -- Keir
> 
> On 05/09/2009 00:02, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:
> 
>> On further debugging, it appears that the
>> p2m_size may be OK, but there's something about
>> those 24 "magic" gpfns that isn't quite right.
>>
>>> -----Original Message-----
>>> From: Dan Magenheimer
>>> Sent: Friday, September 04, 2009 3:29 PM
>>> To: Wayne Gong; Annie Li; Keir Fraser
>>> Cc: Joshua West; James Harper; xen-devel@lists.xensource.com
>>> Subject: RE: [Xen-devel] Error restoring DomU when using GPLPV
>>>
>>>
>>> I think I've tracked down the cause of this problem
>>> in the hypervisor, but am unsure how to best fix it.
>>>
>>> In tools/libxc/xc_domain_save.c, the static variable p2m_size
>>> is said to be "number of pfns this guest has (i.e. number of
>>> entries in the P2M)".  But apparently p2m_size is getting
>>> set to a very large number (0x100000) regardless of the
>>> maximum psuedophysical memory for the hvm guest.  As a result,
>>> some "magic" pages in the 0xf0000-0xfefff range are getting
>>> placed in the save file.  But since they are not "real"
>>> pages, the restore process runs beyond the maximum number
>>> of physical pages allowed for the domain and fails.
>>> (The gpfn of the last 24 pages saved are f2020, fc000-fc012,
>>> feffb, feffc, feffd, feffe.)
>>>
>>> p2m_size is set in "save" with a call to a memory_op hypercall
>>> (XENMEM_maximum_gpfn) which for an hvm domain returns
>>> d->arch.p2m->max_mapped_pfn.  I suspect that the meaning
>>> of max_mapped_pfn changed at some point to more match
>>> its name, but this changed the semantics of the hypercall
>>> as used by xc_domain_restore, resulting in this curious
>>> problem.
>>>
>>> Any thoughts on how to fix this?
>>>
>>>> -----Original Message-----
>>>> From: Annie Li 
>>>> Sent: Tuesday, September 01, 2009 10:27 PM
>>>> To: Keir Fraser
>>>> Cc: Joshua West; James Harper; xen-devel@lists.xensource.com
>>>> Subject: Re: [Xen-devel] Error restoring DomU when using GPLPV
>>>>
>>>>
>>>>
>>>>> It seems this problem is connected with gnttab, not shareinfo.
>>>>> I changed some code about grant table in winpv driver (not using
>>>>> balloon down shinfo+gnttab method),
>>> save/restore/migration can work
>>>>> properly on Xen3.4 now.
>>>>>
>>>>> What i changed is winpv driver use hypercall
>>>> XENMEM_add_to_physmap to
>>>>> map corresponding grant tables which devices require, instead of
>>>>> mapping all 32 pages grant table during initialization.  It seems
>>>>> those extra grant table mapping cause this problem.
>>>> Wondering whether those extra grant table mapping is the root
>>>> cause of 
>>>> the migration problem? or by luck as linux PVHVM too?
>>>>
>>>> Thanks
>>>> Annie.
>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xensource.com
>>>> http://lists.xensource.com/xen-devel
>>>>
>>> _______________________________________________
>>> Xen-devel mailing list
>>> Xen-devel@lists.xensource.com
>>> http://lists.xensource.com/xen-devel
>>>
> 

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 15/09/2009 03:25, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:

> Ok, I've been looking at this and figured what's going on. Annie's problem
> lies in not remapping the grant frames post migration. Hence the leak,
> tot_pages goes up every time until migration fails. On linux, remapping
> is where the frames created by restore (for heap pfn's), get freed back to
> the dom heap, is what I found.  So that's a fix to be made on win
> pv driver side.

Although obviosuly that is a bug, I'm not sure why it would cause this
particular issue? The domheap pages do not get freed and replaced with
xenheap pages, but why does that affect the next save/restore cycle? After
all, xc_domain_save does not distinguish between Xenheap and domheap pages?

> 1. Always balloon down, shinfo+gnttab frames: This needs to be done just
>     once during load, right? I'm not sure how it would work tho if mem gets
>     ballooned up subsequently. I suppose the driver will have to intercept
>     every increase in reservation and balloon down everytime?

Well, it is the same driver that is doing the ballooning, so it's kind of
easy to intercept, right? Just need to track how many Xenheap pages are
mapped and maintain that amount of 'balloon down'.

>     Also, balloon down during suspend call would prob be too late, right?

Indeed it would. Need to do it during boot. It's only a few pages though, so
noone will miss them.

> 2. libxc fix: I wonder how much work this will be. Good thing here is,
>     it'll take care of both linux and PV HVM guests avoiding driver
>     updates in many versions, and hence appealing to us. Can we somehow
>     mark the frames special to be skipped? Looking at biiig xc_domain_save
>     function, not sure in case of HVM, how pfn_type gets set. May be before
> the
>     outer loop, it could ask hyp for all xen heap page list, but then what if
> a
>     new page gets added to the list in between.....

It's a pain. Pfn_type[] I think doesn't really get used. Xc_domain_save()
just tries to map PFNs and saves all the ones it successfully maps. So the
problem is it is allowed to map Xenheap pages. But we can't always disallow
that because sometimes the tools have good reason to map Xenheap pages. So
we'd need a new hypercall, or a flag, or something, and that would need dom0
kernel changes as well as Xen and toolstack changes. So it's rather a pain.

> Also, unfortunately, the failure case is not handled properly sometimes.
> If migration fails after suspend, then no way to get the guest
> back. I even noticed, the guest disappeared totally from both source and
> target when failed, couple times of several dozen migrations I did.

That shouldn't happen since there is a mechanism to cancel the suspension of
a suspended guest. Possibly xend doesn't get it right every time, as it's
error handling is pretty poor in general. I trust the underlying mechanisms
below xend pretty well however.

 -- Keir

Re: Error restoring DomU when using GPLPV

[Mukesh Rathor]

Keir Fraser wrote:
> On 15/09/2009 03:25, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:
> 
>> Ok, I've been looking at this and figured what's going on. Annie's problem
>> lies in not remapping the grant frames post migration. Hence the leak,
>> tot_pages goes up every time until migration fails. On linux, remapping
>> is where the frames created by restore (for heap pfn's), get freed back to
>> the dom heap, is what I found.  So that's a fix to be made on win
>> pv driver side.
> 
> Although obviosuly that is a bug, I'm not sure why it would cause this
> particular issue? The domheap pages do not get freed and replaced with
> xenheap pages, but why does that affect the next save/restore cycle? After
> all, xc_domain_save does not distinguish between Xenheap and domheap pages?

xc_domain_save doesn't distinguish is actually the problem, as
xc_domain_restore then backs xenheap pfn's for shinfo/gnt frames with dom
heap pages. These dom heap pages do get freed and replaced by xenheap pages
on target host (upon guest remap in gnttab_map()) in following code:

arch_memory_op():
         /* Remove previously mapped page if it was present. */
         prev_mfn = gmfn_to_mfn(d, xatp.gpfn);
         if ( mfn_valid(prev_mfn) )
         {
                 .....
                 guest_remove_page(d, xatp.gpfn);  <=======
         }

Eg. my guest with 128M gets created with tot_pages=0x83eb
max_pages:0x8400. Now xc_domain_save saves all, 0x83eb+shinfo+gnt
frames(2), so I see tot_pages on target go upto 0x83ee. Now, guest
remaps() shinfo and gnt frames. The dom heap pages are returned in
guest_remove_page(), tot_pages goes back to 0x83eb. In Annie's case,
driver forgets to remap the 2 gnt frames, so dom heap pages are wrongly
mapped and tot_pages remains at 0x83ed, and after few more when it reaches
0x83ff, migration fails as save is not be able to create
0x83ff+shinfo+gntframes temporarily, max_page being 0x8400.

Hope that makes sense.


>> 1. Always balloon down, shinfo+gnttab frames: This needs to be done just
>>     once during load, right? I'm not sure how it would work tho if mem gets
>>     ballooned up subsequently. I suppose the driver will have to intercept
>>     every increase in reservation and balloon down everytime?
> 
> Well, it is the same driver that is doing the ballooning, so it's kind of
> easy to intercept, right? Just need to track how many Xenheap pages are
> mapped and maintain that amount of 'balloon down'.

Yup, that's what I thought, but just wanted to make sure.

>>     Also, balloon down during suspend call would prob be too late, right?
> 
> Indeed it would. Need to do it during boot. It's only a few pages though, so
> noone will miss them.
> 
>> 2. libxc fix: I wonder how much work this will be. Good thing here is,
>>     it'll take care of both linux and PV HVM guests avoiding driver
>>     updates in many versions, and hence appealing to us. Can we somehow
>>     mark the frames special to be skipped? Looking at biiig xc_domain_save
>>     function, not sure in case of HVM, how pfn_type gets set. May be before
>> the
>>     outer loop, it could ask hyp for all xen heap page list, but then what if
>> a
>>     new page gets added to the list in between.....
> 
> It's a pain. Pfn_type[] I think doesn't really get used. Xc_domain_save()
> just tries to map PFNs and saves all the ones it successfully maps. So the
> problem is it is allowed to map Xenheap pages. But we can't always disallow
> that because sometimes the tools have good reason to map Xenheap pages. So
> we'd need a new hypercall, or a flag, or something, and that would need dom0
> kernel changes as well as Xen and toolstack changes. So it's rather a pain.

Ok got it, I think driver change is the way to go.

>> Also, unfortunately, the failure case is not handled properly sometimes.
>> If migration fails after suspend, then no way to get the guest
>> back. I even noticed, the guest disappeared totally from both source and
>> target when failed, couple times of several dozen migrations I did.
> 
> That shouldn't happen since there is a mechanism to cancel the suspension of
> a suspended guest. Possibly xend doesn't get it right every time, as it's
> error handling is pretty poor in general. I trust the underlying mechanisms
> below xend pretty well however.

>  -- Keir


thanks a lot,
Mukesh

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 15/09/2009 20:14, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:

> Eg. my guest with 128M gets created with tot_pages=0x83eb
> max_pages:0x8400. Now xc_domain_save saves all, 0x83eb+shinfo+gnt
> frames(2), so I see tot_pages on target go upto 0x83ee. Now, guest
> remaps() shinfo and gnt frames. The dom heap pages are returned in
> guest_remove_page(), tot_pages goes back to 0x83eb. In Annie's case,
> driver forgets to remap the 2 gnt frames, so dom heap pages are wrongly
> mapped and tot_pages remains at 0x83ed, and after few more when it reaches
> 0x83ff, migration fails as save is not be able to create
> 0x83ff+shinfo+gntframes temporarily, max_page being 0x8400.
> 
> Hope that makes sense.

No, it doesn't. I agree that after the first migration tot_pages will have
increased to 0x83ed. But I do not agree that it will continue to increase by
three pages on each future migration. Look at it this way -- three GPFNs
(guest-physical pages) have changed from xenheap pages to domheap pages
across that first migration. On future migrations they will be migrated just
like any other ordinary domheap page, since that's what they now are. And
tot_pages will therefore not change. Right?

This is why I still cannot understand or explain Annie's experimental
result.

 -- Keir

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 15/09/2009 22:25, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:

> No, it doesn't. I agree that after the first migration tot_pages will have
> increased to 0x83ed. But I do not agree that it will continue to increase by
> three pages on each future migration. Look at it this way -- three GPFNs
> (guest-physical pages) have changed from xenheap pages to domheap pages
> across that first migration. On future migrations they will be migrated just
> like any other ordinary domheap page, since that's what they now are. And
> tot_pages will therefore not change. Right?

Actually of course you do the right thing with the shinfo page, so actually
one page per migration does get switched back to being a Xenheap page (the
shinfo page) and tot_pages actually increases by 3 on the first migration,
then decreases by 1 when shinfo gets remapped by the PV drivers. Then
increases by 1 on every future migration (which is the shinfo Xenheap page
getting changed into a domheap page), and then decreases by 1 when shinfo
gets remapped by the PV drivers.

But even setting things out exactly right as above, the end result is the
same: I *still* cannot explain Annie's result.

 -- Keir

> This is why I still cannot understand or explain Annie's experimental
> result.

Re: Error restoring DomU when using GPLPV

[Mukesh Rathor]

Keir Fraser wrote:
> On 15/09/2009 22:25, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:
> 
>> No, it doesn't. I agree that after the first migration tot_pages will have
>> increased to 0x83ed. But I do not agree that it will continue to increase by
>> three pages on each future migration. Look at it this way -- three GPFNs
>> (guest-physical pages) have changed from xenheap pages to domheap pages
>> across that first migration. On future migrations they will be migrated just
>> like any other ordinary domheap page, since that's what they now are. And
>> tot_pages will therefore not change. Right?
> 
> Actually of course you do the right thing with the shinfo page, so actually
> one page per migration does get switched back to being a Xenheap page (the
> shinfo page) and tot_pages actually increases by 3 on the first migration,
> then decreases by 1 when shinfo gets remapped by the PV drivers. Then
> increases by 1 on every future migration (which is the shinfo Xenheap page
> getting changed into a domheap page), and then decreases by 1 when shinfo
> gets remapped by the PV drivers.
> 
> But even setting things out exactly right as above, the end result is the
> same: I *still* cannot explain Annie's result.

The bug in her driver is that its only remapping shinfo page, and NOT
the 2 shared grant frames. tot_pages hence increases by 2 every
migration. I can see all in kdb.  tot_pages goes up by 3, then down by 1
as shared info frame is remapped, and remains there. Next migration, it
goes up by 3, down by 1 again.  So each migration leaks 2 frames. The initial
difference is 21 frames between tot and max, hence after 10 migrations
it fails. (BTW, no max_mem specified in config file, I'm told it means no POD).

On linux side, driver remaps shinfo page + both grant frames. So, it goes up
by 3 for a moment, then comes remap and down by 3, back to where it was. If
tot_pages == max_pages, then mig will fail. Which brings me to a question,
to test out balloon changes, what would be the best way to get tot_pages
equal to max_pages. xm mem-set doesn't quite get me there. Occassionally
I see the two same after starting guest, but I've not figured out what
causes that to happen.

thakns
Mukesh



>  -- Keir
> 
>> This is why I still cannot understand or explain Annie's experimental
>> result.
> 

Re: Error restoring DomU when using GPLPV

[ANNIE LI]

Hi,
> Actually of course you do the right thing with the shinfo page, so actually
> one page per migration does get switched back to being a Xenheap page (the
> shinfo page) and tot_pages actually increases by 3 on the first migration,
> then decreases by 1 when shinfo gets remapped by the PV drivers. Then
> increases by 1 on every future migration (which is the shinfo Xenheap page
> getting changed into a domheap page), and then decreases by 1 when shinfo
> gets remapped by the PV drivers.
>
> But even setting things out exactly right as above, the end result is the
> same: I *still* cannot explain Annie's result.
The root cause is that winpv driver did not re-map gnttab frames during 
resuming.
Thanks Mukesh very much.

My initial implementation was to map all 32 grant table pages during 
initialization, and then balloon down
those pages during driver first load. However,  i leaked those 32 grant 
pages if i did not re-map those pages
during resuming. This is why Save/restore can work only once.

My second implementation is to map corresponding grant frames device 
needs instead of all 32 grant table.
But it will leak 2 frames every migration because of missing re-mapping 
grant tables.

Then i tried to re-map the grant table during resuming, and balloon down 
shinfo+gntab driver first load.
I did save/restore several times, did not hit any problem. Furthermore, 
i also tried to map 64 grant table pages
during initialization and ballooned down those pages, all work fine.

I will do more test to make sure it and update here.

Thanks
Annie.

Re: Error restoring DomU when using GPLPV

[ANNIE LI]

> I will do more test to make sure it and update here.
I tried to map 256 grant frames during initialization and balloon down 
256+1(shinfo+gnttab) pages driver first
load. Then i did save/restore for 50 times, and live migration for 10 
times. No error occurs.

Thanks
Annie.

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

On 16/09/2009 12:10, "ANNIE LI" <annie.li@oracle.com> wrote:

>> I will do more test to make sure it and update here.
> I tried to map 256 grant frames during initialization and balloon down
> 256+1(shinfo+gnttab) pages driver first
> load. Then i did save/restore for 50 times, and live migration for 10
> times. No error occurs.

Okay, well I still can't explain why that fixes it, but clearly it does. So
that's good. :-)

 -- Keir

RE: Error restoring DomU when using GPLPV

[Dan Magenheimer]

Before we close down this thread, I have a concern:

According to Mukesh, the fix to this bug is dependent
on the pv drivers tracking tot_pages for a domain
and ballooning to ensure tot_pages+3 does not exceed
max_pages for the domain.

Well, tmem can affect tot_pages for a domain inside
the hypervisor without any notification to pv drivers
or the balloon driver.  And I'd imagine that PoD and
future memory optimization mechanisms such as
swapping and page-sharing may do the same.

So this solution seems very fragile.

Dan

> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Sent: Wednesday, September 16, 2009 6:28 AM
> To: Annie Li
> Cc: Joshua West; Dan Magenheimer; xen-devel; Kurt Hackel; 
> James Harper;
> Wayne Gong
> Subject: Re: [Xen-devel] Error restoring DomU when using GPLPV
> 
> 
> On 16/09/2009 12:10, "ANNIE LI" <annie.li@oracle.com> wrote:
> 
> >> I will do more test to make sure it and update here.
> > I tried to map 256 grant frames during initialization and 
> balloon down
> > 256+1(shinfo+gnttab) pages driver first
> > load. Then i did save/restore for 50 times, and live 
> migration for 10
> > times. No error occurs.
> 
> Okay, well I still can't explain why that fixes it, but 
> clearly it does. So
> that's good. :-)
> 
>  -- Keir
> 
> 
>

Re: Error restoring DomU when using GPLPV

[Mukesh Rathor]

just in case someone missed the thread earlier,

3 = 1 shinfo + 2 gnt frames default.

so, tot_pages + shinfo + num gnt frames.


Mukesh



Dan Magenheimer wrote:
> Before we close down this thread, I have a concern:
> 
> According to Mukesh, the fix to this bug is dependent
> on the pv drivers tracking tot_pages for a domain
> and ballooning to ensure tot_pages+3 does not exceed
> max_pages for the domain.
> 
> Well, tmem can affect tot_pages for a domain inside
> the hypervisor without any notification to pv drivers
> or the balloon driver.  And I'd imagine that PoD and
> future memory optimization mechanisms such as
> swapping and page-sharing may do the same.
> 
> So this solution seems very fragile.
> 
> Dan
> 
>> -----Original Message-----
>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
>> Sent: Wednesday, September 16, 2009 6:28 AM
>> To: Annie Li
>> Cc: Joshua West; Dan Magenheimer; xen-devel; Kurt Hackel; 
>> James Harper;
>> Wayne Gong
>> Subject: Re: [Xen-devel] Error restoring DomU when using GPLPV
>>
>>
>> On 16/09/2009 12:10, "ANNIE LI" <annie.li@oracle.com> wrote:
>>
>>>> I will do more test to make sure it and update here.
>>> I tried to map 256 grant frames during initialization and 
>> balloon down
>>> 256+1(shinfo+gnttab) pages driver first
>>> load. Then i did save/restore for 50 times, and live 
>> migration for 10
>>> times. No error occurs.
>> Okay, well I still can't explain why that fixes it, but 
>> clearly it does. So
>> that's good. :-)
>>
>>  -- Keir
>>
>>
>>
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Error restoring DomU when using GPLPV

[Keir Fraser]

Yeah, all the PV drivers are having to do is balloon down one page for every
Xenheap page they map. There's no further complexity than that, so let's not
make a mountain out of a molehill. The approach as discussed and now
implemented should work fine with tmem I think.

 -- Keir

On 16/09/2009 21:50, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:

> just in case someone missed the thread earlier,
> 
> 3 = 1 shinfo + 2 gnt frames default.
> 
> so, tot_pages + shinfo + num gnt frames.
> 
> 
> Mukesh
> 
> 
> 
> Dan Magenheimer wrote:
>> Before we close down this thread, I have a concern:
>> 
>> According to Mukesh, the fix to this bug is dependent
>> on the pv drivers tracking tot_pages for a domain
>> and ballooning to ensure tot_pages+3 does not exceed
>> max_pages for the domain.
>> 
>> Well, tmem can affect tot_pages for a domain inside
>> the hypervisor without any notification to pv drivers
>> or the balloon driver.  And I'd imagine that PoD and
>> future memory optimization mechanisms such as
>> swapping and page-sharing may do the same.
>> 
>> So this solution seems very fragile.
>> 
>> Dan
>> 
>>> -----Original Message-----
>>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
>>> Sent: Wednesday, September 16, 2009 6:28 AM
>>> To: Annie Li
>>> Cc: Joshua West; Dan Magenheimer; xen-devel; Kurt Hackel;
>>> James Harper;
>>> Wayne Gong
>>> Subject: Re: [Xen-devel] Error restoring DomU when using GPLPV
>>> 
>>> 
>>> On 16/09/2009 12:10, "ANNIE LI" <annie.li@oracle.com> wrote:
>>> 
>>>>> I will do more test to make sure it and update here.
>>>> I tried to map 256 grant frames during initialization and
>>> balloon down
>>>> 256+1(shinfo+gnttab) pages driver first
>>>> load. Then i did save/restore for 50 times, and live
>>> migration for 10
>>>> times. No error occurs.
>>> Okay, well I still can't explain why that fixes it, but
>>> clearly it does. So
>>> that's good. :-)
>>> 
>>>  -- Keir
>>> 
>>> 
>>> 
>> 
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel

RE: Error restoring DomU when using GPLPV

[Dan Magenheimer]

The problem is that every page that is ballooned down by
the balloon driver can be slurped up as a private-
persistent ("preswap") page by tmem.  Private-persistent
pages contain indirectly-accessible domain data, are counted
against the domain's tot_pages, and are migrated along with
the domain-directly-accessible pages.

So any temporary mapping of xenheap pages into domheap,
such as occurs during restore/migration, can cause max_pages
to be exceeded.

This isn't a problem today for tmem because tmem only runs
in PV domains today, but I suspect the fragileness of this
approach will come back and bite us.  It reminds me
of the classic "shell game".

Is there a per-domain counter of these special pages
somewhere?  If so, a MEMF flag could subtract this
from max_pages in the limit check in assign_pages(),
e.g.:

max = d->max_pages;
if ( memflags & MEMF_no_special )
    max -= d->special_pages;
<snip>
    if ( unlikely((d->tot_pages + ... > max )
        /* Over-allocation */

(Special_pages counts any xenheap pages
that contain domain-specific data that needs
to be retained across a migration.)

Dan

> -----Original Message-----
> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> Sent: Thursday, September 17, 2009 12:21 AM
> To: Mukesh Rathor; Dan Magenheimer
> Cc: Annie Li; Joshua West; James Harper; xen-devel; Wayne Gong; Kurt
> Hackel
> Subject: Re: [Xen-devel] Error restoring DomU when using GPLPV
> 
> 
> Yeah, all the PV drivers are having to do is balloon down one 
> page for every
> Xenheap page they map. There's no further complexity than 
> that, so let's not
> make a mountain out of a molehill. The approach as discussed and now
> implemented should work fine with tmem I think.
> 
>  -- Keir
> 
> On 16/09/2009 21:50, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:
> 
> > just in case someone missed the thread earlier,
> > 
> > 3 = 1 shinfo + 2 gnt frames default.
> > 
> > so, tot_pages + shinfo + num gnt frames.
> > 
> > 
> > Mukesh
> > 
> > 
> > 
> > Dan Magenheimer wrote:
> >> Before we close down this thread, I have a concern:
> >> 
> >> According to Mukesh, the fix to this bug is dependent
> >> on the pv drivers tracking tot_pages for a domain
> >> and ballooning to ensure tot_pages+3 does not exceed
> >> max_pages for the domain.
> >> 
> >> Well, tmem can affect tot_pages for a domain inside
> >> the hypervisor without any notification to pv drivers
> >> or the balloon driver.  And I'd imagine that PoD and
> >> future memory optimization mechanisms such as
> >> swapping and page-sharing may do the same.
> >> 
> >> So this solution seems very fragile.
> >> 
> >> Dan
> >> 
> >>> -----Original Message-----
> >>> From: Keir Fraser [mailto:keir.fraser@eu.citrix.com]
> >>> Sent: Wednesday, September 16, 2009 6:28 AM
> >>> To: Annie Li
> >>> Cc: Joshua West; Dan Magenheimer; xen-devel; Kurt Hackel;
> >>> James Harper;
> >>> Wayne Gong
> >>> Subject: Re: [Xen-devel] Error restoring DomU when using GPLPV
> >>> 
> >>> 
> >>> On 16/09/2009 12:10, "ANNIE LI" <annie.li@oracle.com> wrote:
> >>> 
> >>>>> I will do more test to make sure it and update here.
> >>>> I tried to map 256 grant frames during initialization and
> >>> balloon down
> >>>> 256+1(shinfo+gnttab) pages driver first
> >>>> load. Then i did save/restore for 50 times, and live
> >>> migration for 10
> >>>> times. No error occurs.
> >>> Okay, well I still can't explain why that fixes it, but
> >>> clearly it does. So
> >>> that's good. :-)
> >>> 
> >>>  -- Keir
> >>> 
> >>> 
> >>> 
> >> 
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xensource.com
> >> http://lists.xensource.com/xen-devel
> 
> 
>

Re: Error restoring DomU when using GPLPV / fix for GPLPV drivers

[Pasi Kärkkäinen]

On Wed, Sep 16, 2009 at 07:10:19PM +0800, ANNIE LI wrote:
> 
> >I will do more test to make sure it and update here.
> I tried to map 256 grant frames during initialization and balloon down 
> 256+1(shinfo+gnttab) pages driver first
> load. Then i did save/restore for 50 times, and live migration for 10 
> times. No error occurs.
> 

James: I guess this same fix should be applied to GPLPV drivers? 

-- Pasi

Re: Error restoring DomU when using GPLPV / fix for GPLPV drivers

[Keith Coleman]

On Thu, Sep 24, 2009 at 4:24 PM, Pasi Kärkkäinen <pasik@iki.fi> wrote:
> On Wed, Sep 16, 2009 at 07:10:19PM +0800, ANNIE LI wrote:
>>
>> >I will do more test to make sure it and update here.
>> I tried to map 256 grant frames during initialization and balloon down
>> 256+1(shinfo+gnttab) pages driver first
>> load. Then i did save/restore for 50 times, and live migration for 10
>> times. No error occurs.
>>
>
> James: I guess this same fix should be applied to GPLPV drivers?
>

The latest GPLPV drivers still can't restore but the discussion of
this issue has gone quiet. Is this a lost cause?

I'm using xen-3.4.1, official 2.6.18-xen kernel,
gplpv_fre_wnet_x86_0.10.0.130.msi

2:~# xm save win2 win2.save
2:~# xm restore win2.save
Error: /usr/lib64/xen/bin/xc_restore 4 49 2 3 1 1 1 failed
Usage: xm restore <CheckpointFile> [-p]

Restore a domain from a saved state.
  -p, --paused                   Do not unpause domain after restoring it


Keith Coleman