several Qs about domain live migration

several Qs about domain live migration

[Tian, Kevin]

Is analysis_phase just for debug purpose, since 'runs' is just zero
when invoked?

Where is the possible place to mark pfn_type to as some special 
type for hvm domain, like invalid pfn?
if ( hvm )
	pfn_type[batch] = n;
else
	pfn_type[batch] = pfn_to_mfn(n);

I didn't find code to check mmio holes. Then how does restore
side differentiate normal ram from those holes, for a hvm with
>4G memory?

Thanks,
Kevin

Re: several Qs about domain live migration

[Keir Fraser]

On 17/11/08 06:50, "Tian, Kevin" <kevin.tian@intel.com> wrote:

> I didn't find code to check mmio holes. Then how does restore
> side differentiate normal ram from those holes, for a hvm with
>> 4G memory?

I don't think the save side transmits entries for non-existent pages.

 -- Keir

RE: several Qs about domain live migration

[Tian, Kevin]

>From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] 
>Sent: Monday, November 17, 2008 3:45 PM
>
>On 17/11/08 06:50, "Tian, Kevin" <kevin.tian@intel.com> wrote:
>
>> I didn't find code to check mmio holes. Then how does restore
>> side differentiate normal ram from those holes, for a hvm with
>>> 4G memory?
>
>I don't think the save side transmits entries for non-existent pages.
>

Yes, that's also my original assumption. But after reading code,
I didn't find lines to seperate those non-existence pages. I saw
one of your change (18533) to skip page patches which contain
no valid pages:
+            if ( hvm )
+            {
+                /* Look for and skip completely empty batches. */
+                for ( j = 0; j < batch; j++ )
+                    if ( (pfn_type[j] & XEN_DOMCTL_PFINFO_LTAB_MASK) !=
+                         XEN_DOMCTL_PFINFO_XTAB )
+                        break;
+                if ( j == batch )
+                {
+                    munmap(region_base, batch*PAGE_SIZE);
+                    continue; /* bail on this batch: no valid pages */
+                }
+            }
+            else

But I don't know where XEN_DOMCTL_PFINFO_XTAB may be
set for hvm domain. Could you help spot it to me?

Thanks,
Kevin

Re: several Qs about domain live migration

[Keir Fraser]

On 17/11/08 07:49, "Tian, Kevin" <kevin.tian@intel.com> wrote:

> But I don't know where XEN_DOMCTL_PFINFO_XTAB may be
> set for hvm domain. Could you help spot it to me?

xc_map_foreign_batch()

 -- Keir

RE: several Qs about domain live migration

[Tian, Kevin]

>From: Keir Fraser [mailto:keir.fraser@eu.citrix.com] 
>Sent: Monday, November 17, 2008 4:40 PM
>On 17/11/08 07:49, "Tian, Kevin" <kevin.tian@intel.com> wrote:
>
>> But I don't know where XEN_DOMCTL_PFINFO_XTAB may be
>> set for hvm domain. Could you help spot it to me?
>
>xc_map_foreign_batch()
>
> -- Keir

Got it. Thank you.
Kevin

Re: several Qs about domain live migration

[Zhai, Edwin]

Keir,

We are trying to fix a HVM live migration bug, and found "Use main memory for 
video memory" in r18383 cause guest hang after restore.

Following changes make all valid pages migrated, including vlapic page 
(0xFEE00), and share page(0xFFFFF), so an extra memory population for these 2 
pages would override previous mapping then cause guest hang (if using vlapic 
acceleration).

What do you think of possible fixing? Skipping these specific pages in 
xc_domain_save except video memory, or change the HVM domain creation.


Thanks,

=====================================================================
diff -r 2397555ebcc2 -r dade7f0bdc8d tools/libxc/xc_domain_save.c
--- a/tools/libxc/xc_domain_save.c      Wed Aug 27 13:31:01 2008 +0100
+++ b/tools/libxc/xc_domain_save.c      Wed Aug 27 14:53:39 2008 +0100
@@ -1109,12 +1109,6 @@ int xc_domain_save(int xc_handle, int io
                 if ( !((test_bit(n, to_send) && !test_bit(n, to_skip)) ||
                        (test_bit(n, to_send) && last_iter) ||
                        (test_bit(n, to_fix)  && last_iter)) )
-                    continue;
-
-                /* Skip PFNs that aren't really there */
-                if ( hvm && ((n >= 0xa0 && n < 0xc0) /* VGA hole */
-                             || (n >= (HVM_BELOW_4G_MMIO_START >> PAGE_SHIFT)
-                                 && n < (1ULL<<32) >> PAGE_SHIFT)) /* MMIO */ )
                     continue;

                 /*

On Mon, Nov 17, 2008 at 04:40:19PM +0800, Keir Fraser wrote:
> 
> 
> 
> On 17/11/08 07:49, "Tian, Kevin" <kevin.tian@intel.com> wrote:
> 
> > But I don't know where XEN_DOMCTL_PFINFO_XTAB may be
> > set for hvm domain. Could you help spot it to me?
> 
> xc_map_foreign_batch()
> 
>  -- Keir
> 
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 

-- 
best rgds,
edwin

Re: several Qs about domain live migration

[Keir Fraser]

On 18/11/08 01:17, "Zhai, Edwin" <edwin.zhai@intel.com> wrote:

> We are trying to fix a HVM live migration bug, and found "Use main memory for
> video memory" in r18383 cause guest hang after restore.
> 
> Following changes make all valid pages migrated, including vlapic page
> (0xFEE00), and share page(0xFFFFF), so an extra memory population for these 2
> pages would override previous mapping then cause guest hang (if using vlapic
> acceleration).
> 
> What do you think of possible fixing? Skipping these specific pages in
> xc_domain_save except video memory, or change the HVM domain creation.

Try changing gmfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e)) in mod_l1_entry()
to:
  mfn = gfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e), &p2mt);
  if ( !p2m_is_ram(p2mt) || (mfn == INVALID_MFN) )
    ....

This will prevent foreign mappings of 'mmio' pages. Pretty sensible and will
fix this problem. 

 -- Keir

Re: several Qs about domain live migration

[Zhai, Edwin]

Keir Fraser wrote:
> On 18/11/08 01:17, "Zhai, Edwin" <edwin.zhai@intel.com> wrote:
>
>   
>> We are trying to fix a HVM live migration bug, and found "Use main memory for
>> video memory" in r18383 cause guest hang after restore.
>>
>> Following changes make all valid pages migrated, including vlapic page
>> (0xFEE00), and share page(0xFFFFF), so an extra memory population for these 2
>> pages would override previous mapping then cause guest hang (if using vlapic
>> acceleration).
>>
>> What do you think of possible fixing? Skipping these specific pages in
>> xc_domain_save except video memory, or change the HVM domain creation.
>>     
>
> Try changing gmfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e)) in mod_l1_entry()
> to:
>   mfn = gfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e), &p2mt);
>   if ( !p2m_is_ram(p2mt) || (mfn == INVALID_MFN) )
>     ....
>
> This will prevent foreign mappings of 'mmio' pages. Pretty sensible and will
> fix this problem. 
>   

It's great!
Does p2m_is_ram return 1 for video memory?
I'll make a small patch and test it.

Thanks,


>  -- Keir
>
>
>   

-- 
best rgds,
edwin

Re: several Qs about domain live migration

[Keir Fraser]

On 18/11/08 08:22, "Zhai, Edwin" <edwin.zhai@intel.com> wrote:

>> Try changing gmfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e)) in mod_l1_entry()
>> to:
>>   mfn = gfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e), &p2mt);
>>   if ( !p2m_is_ram(p2mt) || (mfn == INVALID_MFN) )
>>     ....
>> 
>> This will prevent foreign mappings of 'mmio' pages. Pretty sensible and will
>> fix this problem.
>>   
> 
> It's great!
> Does p2m_is_ram return 1 for video memory?

Yes, since it is just ordinary guest memory. That was the point of Samuel's
patch.

 -- Keir

> I'll make a small patch and test it.

Re: several Qs about domain live migration

[Zhai, Edwin]

Keir,
As my test, video mem and vlapic page are okay, but share page is
still treated as ram.

After going through the source code, I think this should be "no harm".
Although xc_domain_restore populate a new share page first, PV driver
in HVM guest will remap share page when resume, thus destroying
previous mapping and releasing previous page.

So we want to use this simple patch, or mark the share page "not ram"?


Keir Fraser wrote:
>  On 18/11/08 08:22, "Zhai, Edwin" <edwin.zhai@intel.com> wrote:
>
> >> Try changing gmfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e)) in 
mod_l1_entry()
> >> to:
> >>   mfn = gfn_to_mfn(FOREIGNDOM, l1e_get_pfn(nl1e), &p2mt);
> >>   if ( !p2m_is_ram(p2mt) || (mfn == INVALID_MFN) )
> >>     ....
> >>
> >> This will prevent foreign mappings of 'mmio' pages. Pretty sensible 
and will
> >> fix this problem.
> >>  
> > It's great!
> > Does p2m_is_ram return 1 for video memory?
>
>  Yes, since it is just ordinary guest memory. That was the point of 
Samuel's
>  patch.
>
>  -- Keir
>
> > I'll make a small patch and test it.
>
>
>
>  _______________________________________________
>  Xen-devel mailing list
>  Xen-devel@lists.xensource.com
>  http://lists.xensource.com/xen-devel
>

-- 
best rgds,
edwin

Re: several Qs about domain live migration

[Keir Fraser]

On 18/11/08 09:35, "Zhai, Edwin" <edwin.zhai@intel.com> wrote:

> Keir,
> As my test, video mem and vlapic page are okay, but share page is
> still treated as ram.
> 
> After going through the source code, I think this should be "no harm".
> Although xc_domain_restore populate a new share page first, PV driver
> in HVM guest will remap share page when resume, thus destroying
> previous mapping and releasing previous page.
> 
> So we want to use this simple patch, or mark the share page "not ram"?

It's fine. The shared page mapping is not set up until after normal guest
memory is restored. So it's safe.

 -- Keir