super page with live migration

super page with live migration

[Zhai, Edwin]

Keir,

As you know, we try to allocate super pages in xc_hvm_build for better 
performance in EPT case. But the same logic is missing in xc_domain_restore.

When try to add the logic, I found it is blocked by the lazy populate algorithm 
in restore -- only populating the pages received from source side rather than 
doing it at one time.

So the result is the EPT guest has performance drop after live migration:(

Do you have any plan to change the lazy populate algorithm? The purpose of it, I 
believe, is to make restore process work without guest memory layout knowledge.  

But different memory allocation process may cause some issues, such as perf drop 
after live migration.

Thanks,



-- 
best rgds,
edwin

Re: super page with live migration

[Keir Fraser]

On 26/9/08 08:45, "Zhai, Edwin" <edwin.zhai@intel.com> wrote:

> As you know, we try to allocate super pages in xc_hvm_build for better
> performance in EPT case. But the same logic is missing in xc_domain_restore.
> 
> When try to add the logic, I found it is blocked by the lazy populate
> algorithm 
> in restore -- only populating the pages received from source side rather than
> doing it at one time.
> 
> So the result is the EPT guest has performance drop after live migration:(
> 
> Do you have any plan to change the lazy populate algorithm? The purpose of it,
> I 
> believe, is to make restore process work without guest memory layout
> knowledge.  

Yes: If pseudo-phys page is not yet populated in target domain, AND it is
first page of a 2MB extent, AND no other pages in that extent are yet
populated, AND the next pages in the save-image stream populate that extent
in order, THEN allocate a superpage. This is made trickier by the fact that
the next 511 pages (to make the 2MB extent) might be split across a batch
boundary. So we'll have to optimistically allocate a superpage in that case,
and then shatter it if it turns out that the contiguous stream of
pseudo-phys addresses is broken in the next batch.

 -- Keir

Re: Re: super page with live migration

[Tim Deegan]

At 10:03 +0100 on 26 Sep (1222423431), Keir Fraser wrote:
> Yes: If pseudo-phys page is not yet populated in target domain, AND it is
> first page of a 2MB extent, AND no other pages in that extent are yet
> populated, AND the next pages in the save-image stream populate that extent
> in order, THEN allocate a superpage. This is made trickier by the fact that
> the next 511 pages (to make the 2MB extent) might be split across a batch
> boundary. So we'll have to optimistically allocate a superpage in that case,
> and then shatter it if it turns out that the contiguous stream of
> pseudo-phys addresses is broken in the next batch.

Or flag the start-of-a-superpage frames when saving them? 

Tim.

-- 
Tim Deegan <Tim.Deegan@citrix.com>
Principal Software Engineer, Citrix Systems (R&D) Ltd.
[Company #02300071, SL9 0DZ, UK.]

Re: Re: super page with live migration

[Keir Fraser]

On 26/9/08 10:08, "Tim Deegan" <Tim.Deegan@citrix.com> wrote:

> At 10:03 +0100 on 26 Sep (1222423431), Keir Fraser wrote:
>> Yes: If pseudo-phys page is not yet populated in target domain, AND it is
>> first page of a 2MB extent, AND no other pages in that extent are yet
>> populated, AND the next pages in the save-image stream populate that extent
>> in order, THEN allocate a superpage. This is made trickier by the fact that
>> the next 511 pages (to make the 2MB extent) might be split across a batch
>> boundary. So we'll have to optimistically allocate a superpage in that case,
>> and then shatter it if it turns out that the contiguous stream of
>> pseudo-phys addresses is broken in the next batch.
> 
> Or flag the start-of-a-superpage frames when saving them?

It's not clear to me this is significantly easier to hack in on the save
side, and restore side would still need modifying, and it would change the
save format (albeit probably not hard to do it in a backward-compatible
way). I'm not absolutely dead set on my stated approach though.

 -- Keir

RE: Re: super page with live migration

[Ian Pratt]

> > Or flag the start-of-a-superpage frames when saving them?
> 
> It's not clear to me this is significantly easier to hack in on the
save
> side, and restore side would still need modifying, and it would change
> the save format (albeit probably not hard to do it in a
backward-compatible
> way). I'm not absolutely dead set on my stated approach though.

Also, since you typically migrate VMs to less busy machines it would be
good to take the opportunity to use more 2MB frames. 

>> Yes: If pseudo-phys page is not yet populated in target domain, AND 
>> it is first page of a 2MB extent, AND no other pages in that extent 
>> are yet populated, AND the next pages in the save-image stream 
>> populate that extent in order, THEN allocate a superpage. This is 
>> made trickier by the fact that the next 511 pages (to make the 2MB 
>> extent) might be split across a batch boundary. So we'll have to 
>> optimistically allocate a superpage in that case, and then shatter it

>> if it turns out that the contiguous stream of pseudo-phys addresses
is broken in the next batch.

Rather than shattering the 2MB frame, it would be better to copy the
pages out into a heap-allocated buffer, free the 2MB frame, and then
allocate order-0 pages and copy the data in.

This avoids needlessly fragmenting xen's memory.

Thanks,
Ian

Re: super page with live migration

[Zhai, Edwin]

Keir Fraser wrote:
> On 26/9/08 08:45, "Zhai, Edwin" <edwin.zhai@intel.com> wrote:
>
>   
>> As you know, we try to allocate super pages in xc_hvm_build for better
>> performance in EPT case. But the same logic is missing in xc_domain_restore.
>>
>> When try to add the logic, I found it is blocked by the lazy populate
>> algorithm 
>> in restore -- only populating the pages received from source side rather than
>> doing it at one time.
>>
>> So the result is the EPT guest has performance drop after live migration:(
>>
>> Do you have any plan to change the lazy populate algorithm? The purpose of it,
>> I 
>> believe, is to make restore process work without guest memory layout
>> knowledge.  
>>     
>
> Yes: If pseudo-phys page is not yet populated in target domain, AND it is
> first page of a 2MB extent, AND no other pages in that extent are yet
> populated, AND the next pages in the save-image stream populate that extent
> in order, THEN allocate a superpage. This is made trickier by the fact that
> the next 511 pages (to make the 2MB extent) might be split across a batch
> boundary. So we'll have to optimistically allocate a superpage in that case,
> and then shatter it if it turns out that the contiguous stream of
> pseudo-phys addresses is broken in the next batch.
>   

It's really tricky logic, and may make the migration process longer:(

Maybe the flag the start-of-a-superpage as Tim said can make it easier.



Thanks,


>  -- Keir
>
>
>   

RE: Re: super page with live migration

[Tian, Kevin]

>From: Ian Pratt
>Sent: 2008年9月26日 18:18
>
>>> if it turns out that the contiguous stream of pseudo-phys addresses
>is broken in the next batch.
>
>Rather than shattering the 2MB frame, it would be better to copy the
>pages out into a heap-allocated buffer, free the 2MB frame, and then
>allocate order-0 pages and copy the data in.
>
>This avoids needlessly fragmenting xen's memory.
>

I'm not familiar with restore proecss. Just a curious question.
Is it difficult or intrusive if we still keep 2MB frame even when
next 511 pages across batch boundary? Then when pages in
future batches come, we just copy them into previously
allocated 2MB frame covering them. I'm just not sure about
the possibility for an individual 4k pseudo-phys page occuring
in early patch, which then result most following superpages
across batch boundary, and thus few 2MB frame can be
allocated in target side. Maybe this concern is not true due to
batch creation in save process? :-)

Thanks,
Kevin

RE: Re: super page with live migration

[Ian Pratt]

> I'm not familiar with restore proecss. Just a curious question.
> Is it difficult or intrusive if we still keep 2MB frame even when
> next 511 pages across batch boundary? 

We should optimistically allocate a 2MB frame if the rest of the batch
we can see is contiguous.

> Then when pages in
> future batches come, we just copy them into previously
> allocated 2MB frame covering them. 

Yes, they'll already have p2m table entries as a result of the earlier
allocation.

If the frames turn out not to be contiguous, we should allocate a buffer
in the heap, copy the earlier pages into it, free the 2MB frame,
allocate 4KB frames and copy the data in.

> I'm just not sure about
> the possibility for an individual 4k pseudo-phys page occuring
> in early patch, which then result most following superpages
> across batch boundary, and thus few 2MB frame can be
> allocated in target side. Maybe this concern is not true due to
> batch creation in save process? :-)

The first iteration sends frames in-order, so it should be fine.

Ian


> Thanks,
> Kevin

RE: Re: super page with live migration

[Tian, Kevin]

>From: Ian Pratt [mailto:Ian.Pratt@eu.citrix.com]
>Sent: Sunday, September 28, 2008 10:17 PM
>
>> I'm not familiar with restore proecss. Just a curious question.
>> Is it difficult or intrusive if we still keep 2MB frame even when
>> next 511 pages across batch boundary?
>
>We should optimistically allocate a 2MB frame if the rest of the batch
>we can see is contiguous.
>
>> Then when pages in
>> future batches come, we just copy them into previously
>> allocated 2MB frame covering them.
>
>Yes, they'll already have p2m table entries as a result of the earlier
>allocation.
>
>If the frames turn out not to be contiguous, we should
>allocate a buffer
>in the heap, copy the earlier pages into it, free the 2MB frame,
>allocate 4KB frames and copy the data in.
>
>> I'm just not sure about
>> the possibility for an individual 4k pseudo-phys page occuring
>> in early patch, which then result most following superpages
>> across batch boundary, and thus few 2MB frame can be
>> allocated in target side. Maybe this concern is not true due to
>> batch creation in save process? :-)
>
>The first iteration sends frames in-order, so it should be fine.
>

OK, same page then. I misunderstood your comments a bit
before.

Thanks,
Kevin