[Qemu-devel] Re: QEMU interfaces for image streaming and post-copy block migration

[Anthony Liguori <aliguori@linux.vnet.ibm.com>]

On 09/07/2010 10:03 AM, Daniel P. Berrange wrote:
> On Tue, Sep 07, 2010 at 08:41:44AM -0500, Anthony Liguori wrote:
>    
>> Hi,
>>
>> We've got copy-on-read and image streaming working in QED and before
>> going much further, I wanted to bounce some interfaces off of the
>> libvirt folks to make sure our final interface makes sense.
>>
>> Here's the basic idea:
>>      
> [snip]
>
>    
>> A related topic is block migration.  Today we support pre-copy migration
>> which means we transfer the block device and then do a live migration.
>> Another approach is to do a live migration, and on the source, run a
>> block server using image streaming on the destination to move the device.
>>
>> With QED, to implement this one would:
>>
>> 1) launch qemu-nbd on the source while the guest is running
>> 2) create a qed file on the destination with copy-on-read enabled and a
>> backing file using nbd: to point to the source qemu-nbd
>> 3) run qemu -incoming on the destination with the qed file
>> 4) execute the migration
>> 5) when migration completes, begin streaming on the destination to
>> complete the copy
>> 6) when the streaming is complete, shut down the qemu-nbd instance on
>> the source
>>      
> IMHO, adding further network sockets is the one thing we absolutely
> don't want to do to migration. I don't much like the idea of launching
> extra daemons either.
>    

One of the use cases I'm trying to accommodate is migration to free 
resources.  By launching a qemu-nbd daemon, we can kill the source qemu 
process and free up all of the associated memory.

>> This is a bit involved and we could potentially automate some of this in
>> qemu by launching qemu-nbd and providing commands to do some of this.
>> Again though, I think the question is what type of interfaces would
>> libvirt prefer?  Low level interfaces + recipes on how to do high level
>> things or higher level interfaces?
>>      
> I think it should be done entirely within the main QEMU migration
> socket. I know this isn't possible with the current impl, since it
> is unidirectional, preventing the target sending the source requests
> for specific data blocks. If we made migration socket bi-directional
> I think we could do it all within qemu with no external helpers
> or extra sockets
>
>   1. Create empty qed file on the destination with copy on read
>      enable backing file pointing to a special 'migrate:' protocol
>    

Why not just point migration and nbd to a unix domain socket and then 
multiplex the two protocols at a higher level?

>   2. Run qemu -incoming on the destination with with the qed file
>   3. execute the migration
>   4. when migration completes, target QEMU continues streaming blocks
>      from the soruce qemu.
>   5. when streaming is complete, source qemu can shutdown.
>
>
> Both your original proposal and mine here seem to have a pretty
> bad failure scenario though. After the cut-over point where the
> VM cpus start running on the destination QEMU, AFAICT, any failure
> on the source before block streaming complete leaves you dead in
> the water.  The source VM no longer has up2date RAM contents and
> the destination VM does not yet have a complete disk image.
>    

Yes.  It's a trade off.  However, pre-copy doesn't really change your 
likelihood of catastrophic failure because if you were going to fail in 
the source, it was going to happen before you completed the block 
transfer anyway.

The advantage of post-copy is that you immediately free resources on the 
source so as a reaction to pressure from overcommit, it's tremendously 
useful.

I still think pre-copy has it's place though.

Regards,

Anthony Liguori


> Regards,
> Daniel
>