Re: [RFC] postcopy livemigration proposal

[Dor Laor <dlaor@redhat.com>]

On 08/08/2011 01:59 PM, Nadav Har'El wrote:
>>> * What's is postcopy livemigration
>>> It is is yet another live migration mechanism for Qemu/KVM, which
>>> implements the migration technique known as "postcopy" or "lazy"
>>> migration. Just after the "migrate" command is invoked, the execution
>>> host of a VM is instantaneously switched to a destination host.
>
> Sounds like a cool idea.
>
>>> The benefit is, total migration time is shorter because it transfer
>>> a page only once. On the other hand precopy may repeat sending same pages
>>> again and again because they can be dirtied.
>>> The switching time from the source to the destination is several
>>> hunderds mili seconds so that it enables quick load balancing.
>>> For details, please refer to the papers.
>
> While these are the obvious benefits, the possible downside (that, as
> always, depends on the workload) is the amount of time that the guest
> workload runs more slowly than usual, waiting for pages it needs to
> continue. There are a whole spectrum between the guest pausing completely
> (which would solve all the problems of migration, but is often considered
> unacceptible) and running at full-speed. Is it acceptable that the guest
> runs at 90% speed during the migration? 50%? 10%?
> I guess we could have nothing to lose from having both options, and choosing
> the most appropriate technique for each guest!

+1

>
>> That's terrific  (nice video also)!
>> Orit and myself had the exact same idea too (now we can't patent it..).
>
> I think new implementation is not the only reason why you cannot patent
> this idea :-) Demand-paged migration has actually been discussed (and done)
> for nearly a quarter of a century (!) in the area of *process* migration.
>
> The first use I'm aware of was in CMU's Accent 1987 - see [1].
> Another paper, [2], written in 1991, discusses how process migration is done
> in UCB's Sprite operating system, and evaluates the various alternatives
> common at the time (20 years ago), including what it calls "lazy copying"
> is more-or-less the same thing as "post copy". Mosix (a project which, in some
> sense, is still alive to day) also used some sort of cross between pre-copying
> (of dirty pages) and copying on-demand of clean pages (from their backing
> store on the source machine).
>
>
> References
> [1] "Attacking the Process Migration Bottleneck"
>       http://www.nd.edu/~dthain/courses/cse598z/fall2004/papers/accent.pdf

w/o reading the internals, patents enable you to implement an existing 
idea on a new field. Anyway, there won't be no patent in this case. 
Still let's have the kvm innovation merged.

> [2]  "Transparent Process Migration: Design Alternatives and the Sprite
>       Implementation"
>       http://nd.edu/~dthain/courses/cse598z/fall2004/papers/sprite-migration.pdf
>
>> Advantages:
>>          - Virtual machines are using more and more memory resources ,
>>          for a virtual machine with very large working set doing live
>>          migration with reasonable down time is impossible today.
>
> If a guest actually constantly uses (working set) most of its allocated
> memory, it will basically be unable to do any significant amount of work
> on the destination VM until this large working set is transfered to the
> destination. So in this scenario, "post copying" doesn't give any
> significant advantages over plain-old "pause guest and send it to the
> destination". Or am I missing something?

There is one key advantage in this scheme/use case - if you have a guest 
with a very large working set, you'll need a very large downtime in 
order to migrate it with today's algorithm. With post copy (aka 
streaming/demand paging), the guest won't have any downtime but will run 
slower than expected.

There are guests today that is impractical to really live migrate them.

btw: Even today, marking pages RO also carries some performance penalty.

>
>> Disadvantageous:
>>          - During the live migration the guest will run slower than in
>>          today's live migration. We need to remember that even today
>>          guests suffer from performance penalty on the source during the
>>          COW stage (memory copy).
>
> I wonder if something like asynchronous page faults can help somewhat with
> multi-process guest workloads (and modified (PV) guest OS).

They should come in to play for some extent. Note that only newer Linux 
guest will enjoy of them.

>
>>          - Failure of the source or destination or the network will cause
>>          us to lose the running virtual machine. Those failures are very
>>          rare.
>
> How is this different from a VM running on a single machine that fails?
> Just that the small probability of failure (roughly) doubles for the
> relatively-short duration of the transfer?

Exactly my point, this is not a major disadvantage because of this low 
probability.