Re: Stalls on Live Migration of VMs with a lot of memory

[Peter Lieven <pl@dlh.net>]

On 04.01.2012 13:28, Paolo Bonzini wrote:
> On 01/04/2012 12:42 PM, Peter Lieven wrote:
>>>
>> ok, then i misunderstood the ram blocks thing. i thought the guest ram
>> would consist of a collection of ram blocks.
>> then let me describe it differntly. would it make sense to process
>> bigger portions of memory (e.g. 1M) in stage 2 to reduce the number of
>> calls to cpu_physical_memory_reset_dirty and instead run it on bigger
>> portions of memory. we might loose a few dirty pages but they will be
>> tracked in the next iteration in stage 2 or in stage 3 at least. what
>> would be necessary is that nobody marks a page dirty
>> while i copy the dirty information for the portion of memory i want to
>> process.
>
> Dirty memory tracking is done by the hypervisor and must be done at 
> page granularity.
ok, so this is unfortunately no option.

thus my only option at the moment is to limit the runtime of the while 
loop in stage 2 or
are there any post 1.0 patches in git that might already help?

i tried to limit it to migrate_max_downtime() and this at least resolves 
the problem with
the vm stalls. however, migration speed is very limited (approx. 80MB/s 
on a 10G link).
with that.


>
>>>> - in stage 3 the vm is stopped, right? so there can't be any more 
>>>> dirty
>>>> blocks after scanning the whole memory once?
>>>
>>> No, stage 3 is entered when there are very few dirty memory pages
>>> remaining.  This may happen after scanning the whole memory many
>>> times.  It may even never happen if migration does not converge
>>> because of low bandwidth or too strict downtime requirements.
>>>
>> ok, is there a chance that i lose one final page if it is modified just
>> after i walked over it and i found no other page dirty (so bytes_sent 
>> = 0).
>
> No, of course not.  Stage 3 will send all missing pages while the VM 
> is stopped.  There is a chance that the guest will go crazy and start 
> touching lots of pages at exactly the wrong time, and thus the 
> downtime will be longer than expected.  However, that's a necessary 
> evil; if you cannot accept that, post-copy migration would provide a 
> completely different set of tradeoffs.
i don't suffer from long downtimes in stage 3. my issue is in stage 2.
>
> (BTW, bytes_sent = 0 is very rare).
i know, but when the vm is stopped there is no issue. i understood your 
"No, stage 3 is entered ..." wrong ;-)

thanks for your help and explainations.

peter
>
> Paolo