Re: [Qemu-devel] [PATCH 00/21][RFC] postcopy live migration

[Michael Roth <mdroth@linux.vnet.ibm.com>]

On 01/01/2012 03:52 AM, Dor Laor wrote:
> On 12/30/2011 12:39 AM, Anthony Liguori wrote:
>> On 12/28/2011 07:25 PM, Isaku Yamahata wrote:
>>> Intro
>>> =====
>>> This patch series implements postcopy live migration.[1]
>>> As discussed at KVM forum 2011, dedicated character device is used for
>>> distributed shared memory between migration source and destination.
>>> Now we can discuss/benchmark/compare with precopy. I believe there are
>>> much rooms for improvement.
>>>
>>> [1] http://wiki.qemu.org/Features/PostCopyLiveMigration
>>>
>>>
>>> Usage
>>> =====
>>> You need load umem character device on the host before starting
>>> migration.
>>> Postcopy can be used for tcg and kvm accelarator. The implementation
>>> depend
>>> on only linux umem character device. But the driver dependent code is
>>> split
>>> into a file.
>>> I tested only host page size == guest page size case, but the
>>> implementation
>>> allows host page size != guest page size case.
>>>
>>> The following options are added with this patch series.
>>> - incoming part
>>> command line options
>>> -postcopy [-postcopy-flags<flags>]
>>> where flags is for changing behavior for benchmark/debugging
>>> Currently the following flags are available
>>> 0: default
>>> 1: enable touching page request
>>>
>>> example:
>>> qemu -postcopy -incoming tcp:0:4444 -monitor stdio -machine accel=kvm
>>>
>>> - outging part
>>> options for migrate command
>>> migrate [-p [-n]] URI
>>> -p: indicate postcopy migration
>>> -n: disable background transferring pages: This is for
>>> benchmark/debugging
>>>
>>> example:
>>> migrate -p -n tcp:<dest ip address>:4444
>>>
>>>
>>> TODO
>>> ====
>>> - benchmark/evaluation. Especially how async page fault affects the
>>> result.
>>
>> I'll review this series next week (Mike/Juan, please also review when
>> you can).
>>
>> But we really need to think hard about whether this is the right thing
>> to take into the tree. I worry a lot about the fact that we don't test
>> pre-copy migration nearly enough and adding a second form just
>> introduces more things to test.
>
> It is an issue but it can't be a merge criteria, Isaku is not blame of
> pre copy live migration lack of testing.
>
> I would say that 90% of issues of live migration problems are not
> related to the pre|post stage but more of issues of device model save
> state. So post-copy shouldn't add a significant regression here.
>
> Probably it will be good to ask every migration patch writer to write an
> additional unit test for migration.
>
>> It's also not clear to me why post-copy is better. If you were going to
>> sit down and explain to someone building a management tool when they
>> should use pre-copy and when they should use post-copy, what would you
>> tell them?
>
> Today, we have a default of max-downtime of 100ms.
> If either the guest work set size or the host networking throughput
> can't match the downtime, migration won't end.
> The mgmt user options are:
> - increase the downtime more and more to an actual stop
> - fail migrate
>
> W/ post-copy there is another option.
> Performance measurements will teach us (probably prior to commit) when
> this stage is valuable. Most likely, we better try first with pre-copy
> and if we can't meet the downtime we can optionally use post-copy.

Umesh's paper seems to already have strong indications that at least 1 
iteration of pre-copy is optimal in terms of downtime, so I wonder if 
we're starting off on the wrong track with the all or nothing approach 
taken with this series?

I only have Umesh's paper to go off (which, granted, notes shadow paging 
(which I guess we effectively have with these patches) as a potential 
improvement to the pseudo swap device used there), but otherwise it 
seems like we'd just get more downtime as the target starts choking on 
network-based page faults.

It's probably not too useful to speculate on performance at this point, 
but I think it'll be easier to get the data (and hit closer to the mark 
suggested by the paper) if we started off assuming that pre-copy should 
still be in play. Maybe something like:

migrate -d tcp:host:123 -p[ostcopy] <after x iterations>

x=0 for post-copy only, no -p for pre-copy-only

Also seems a bit cleaner. And if post-copy proves to be optimal we just 
make -p 0 implied... minor details at this point, but the main thing is 
that integrating better with the pre-copy code will make it easier to 
determine what the sweet spot is and how much we stand to gain.

>
> Here's a paper by Umesh (the migration thread writer):
> http://osnet.cs.binghamton.edu/publications/hines09postcopy_osr.pdf
>
> Regards,
> Dor
>
>>
>> Regards,
>>
>> Anthony Liguori
>>
>
>