treatment grant frames during save/restore

treatment grant frames during save/restore

[Jan Beulich]

>From what I can tell, grant frames get no special treatment by the tools
during save, and hence get treated as normal memory during restore.
If that's correct, it would seem that there's an accounting issue, since
during restore a normal RAM page will be allocated by the tools for each
grant frame, hence a domain that prior to save was below its limit
could now end up crossing that limit, and thus unexpectedly fail to
restore. If that's not a possibility, what am I missing?

Thanks, Jan

Re: treatment grant frames during save/restore

[Jan Beulich]

>>> On 27.05.10 at 17:27, "Jan Beulich" <JBeulich@novell.com> wrote:
> From what I can tell, grant frames get no special treatment by the tools
> during save, and hence get treated as normal memory during restore.
> If that's correct, it would seem that there's an accounting issue, since
> during restore a normal RAM page will be allocated by the tools for each
> grant frame, hence a domain that prior to save was below its limit
> could now end up crossing that limit, and thus unexpectedly fail to
> restore. If that's not a possibility, what am I missing?

Ooops, just noticed that I didn't say explicitly that this is for HVM
guests.

Thanks, Jan

RE: treatment grant frames during save/restore

[Dan Magenheimer]

> -----Original Message-----
> From: Jan Beulich [mailto:JBeulich@novell.com]
> Sent: Thursday, May 27, 2010 9:38 AM
> To: xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] treatment grant frames during save/restore
> 
> >>> On 27.05.10 at 17:27, "Jan Beulich" <JBeulich@novell.com> wrote:
> > From what I can tell, grant frames get no special treatment by the
> tools
> > during save, and hence get treated as normal memory during restore.
> > If that's correct, it would seem that there's an accounting issue,
> since
> > during restore a normal RAM page will be allocated by the tools for
> each
> > grant frame, hence a domain that prior to save was below its limit
> > could now end up crossing that limit, and thus unexpectedly fail to
> > restore. If that's not a possibility, what am I missing?
> 
> Ooops, just noticed that I didn't say explicitly that this is for HVM
> guests.

There was some discussion about this last September.  Here's one link:

http://lists.xensource.com/archives/html/xen-devel/2009-09/msg00578.html 

Re: treatment grant frames during save/restore

[Keir Fraser]

On 27/05/2010 16:27, "Jan Beulich" <JBeulich@novell.com> wrote:

>> From what I can tell, grant frames get no special treatment by the tools
> during save, and hence get treated as normal memory during restore.
> If that's correct, it would seem that there's an accounting issue, since
> during restore a normal RAM page will be allocated by the tools for each
> grant frame, hence a domain that prior to save was below its limit
> could now end up crossing that limit, and thus unexpectedly fail to
> restore. If that's not a possibility, what am I missing?

Yes, it's an issue. Fixing is tricky since in some cases dom0 *wants* to be
able to map domU special Xen-heap pages. So we need to be able to specify
some kind of flag to say 'really map this domain's domain-heap RAM pages
only on this request' and preferably tunnel that flag through existing dom0
kernels so that it makes it unscathed down to the Xen hypercall. That's a
bit tricky I think, unless we do nasty things like steal bits from the
existing domid or pte.val fields to mmu_update(). Else we need dom0 kernel
mods too, which is a pain in the bum, but I suppose we could do that with
fallback to what we do currently.

 -- Keir

> Thanks, Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: treatment grant frames during save/restore

[Jan Beulich]

>>> On 27.05.10 at 18:02, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> Yes, it's an issue. Fixing is tricky since in some cases dom0 *wants* to be
> able to map domU special Xen-heap pages. So we need to be able to specify
> some kind of flag to say 'really map this domain's domain-heap RAM pages
> only on this request' and preferably tunnel that flag through existing dom0
> kernels so that it makes it unscathed down to the Xen hypercall. That's a
> bit tricky I think, unless we do nasty things like steal bits from the
> existing domid or pte.val fields to mmu_update(). Else we need dom0 kernel
> mods too, which is a pain in the bum, but I suppose we could do that with
> fallback to what we do currently.

Wouldn't is suffice to adjust XEN_DOMCTL_getpageframeinfo{2,3} to
report these pages as special (not sure whether a new type would
be needed, or whether simply returning XEN_DOMCTL_PFINFO_XTAB
would be acceptable), thus allowing the tools to skip them as
necessary? This of course implies that the tools would need to issue
the call for HVM guests (currently I think they do so only for PV ones),
that the actual page contents is irrelevant post-restore, and that
there being a hole in the physical address space post-restore isn't
a problem for the pv drivers.

Jan

Re: treatment grant frames during save/restore

[Keir Fraser]

On 28/05/2010 09:01, "Jan Beulich" <JBeulich@novell.com> wrote:

> Wouldn't is suffice to adjust XEN_DOMCTL_getpageframeinfo{2,3} to
> report these pages as special (not sure whether a new type would
> be needed, or whether simply returning XEN_DOMCTL_PFINFO_XTAB
> would be acceptable), thus allowing the tools to skip them as
> necessary? This of course implies that the tools would need to issue
> the call for HVM guests (currently I think they do so only for PV ones),
> that the actual page contents is irrelevant post-restore, and that
> there being a hole in the physical address space post-restore isn't
> a problem for the pv drivers.

Yep that would probably work. And also potentially gets rid of one lot of
"if-hvm-else-pv" branched code in xc_domain_save.c. I'd take a patch to do
that if you want to pick this item up.

 -- Keir

Re: treatment grant frames during save/restore

[Jan Beulich]

>>> On 28.05.10 at 10:22, Keir Fraser <keir.fraser@eu.citrix.com> wrote:
> On 28/05/2010 09:01, "Jan Beulich" <JBeulich@novell.com> wrote:
> 
>> Wouldn't is suffice to adjust XEN_DOMCTL_getpageframeinfo{2,3} to
>> report these pages as special (not sure whether a new type would
>> be needed, or whether simply returning XEN_DOMCTL_PFINFO_XTAB
>> would be acceptable), thus allowing the tools to skip them as
>> necessary? This of course implies that the tools would need to issue
>> the call for HVM guests (currently I think they do so only for PV ones),
>> that the actual page contents is irrelevant post-restore, and that
>> there being a hole in the physical address space post-restore isn't
>> a problem for the pv drivers.
> 
> Yep that would probably work. And also potentially gets rid of one lot of
> "if-hvm-else-pv" branched code in xc_domain_save.c. I'd take a patch to do
> that if you want to pick this item up.

I'm afraid it won't: The live_p2m table gets created for pv guests only,
but is needed as a prerequisite to calling XEN_DOMCTL_getpageframeinfo*
(which wants MFNs as input). Hence while the hypervisor side patch is
trivial, it doesn't get us any closer to a solution to the problem at hand.

Unless we (re-)define the meaning of the input array to this domctl to
specify gmfn-s rather than mfn-s (at least for the hvm case; for
auto-translate pv guests, quite obviously the save code wouldn't
work anyway, but for those passing in gmfn-s would seem the
natural thing here).

Or unless we want to add code to libxc to create a live_p2m for all
guests (which I wouldn't want to take on).

Jan

Re: treatment grant frames during save/restore

[Keir Fraser]

On 31/05/2010 15:58, "Jan Beulich" <JBeulich@novell.com> wrote:

>> Yep that would probably work. And also potentially gets rid of one lot of
>> "if-hvm-else-pv" branched code in xc_domain_save.c. I'd take a patch to do
>> that if you want to pick this item up.
> 
> I'm afraid it won't: The live_p2m table gets created for pv guests only,
> but is needed as a prerequisite to calling XEN_DOMCTL_getpageframeinfo*
> (which wants MFNs as input). Hence while the hypervisor side patch is
> trivial, it doesn't get us any closer to a solution to the problem at hand.
> 
> Unless we (re-)define the meaning of the input array to this domctl to
> specify gmfn-s rather than mfn-s (at least for the hvm case; for
> auto-translate pv guests, quite obviously the save code wouldn't
> work anyway, but for those passing in gmfn-s would seem the
> natural thing here).

This is fine. We can (re-)define the domctl interface as we like, and anyway
I'm pretty sure noone uses that particular domctl on HVM guests (yet).

 -- Keir

> Or unless we want to add code to libxc to create a live_p2m for all
> guests (which I wouldn't want to take on).