[Qemu-devel] Killing block migration in qemu?

[Qemu-devel] Killing block migration in qemu?

[Paolo Bonzini]

Hi all,

following discussions yesterday with Juan Quintela and Marcelo Tosatti, 
here is my humble proposal: remove block migration from qemu master.  It 
seems to me that keeping block migration is going to slow down further 
improvements on migration.  The main problems are:

1) there are very good reasons to move migration to a separate thread. 
Only a limited amount of extra locking, perhaps none is needed in order 
to do so for RAM and devices.  But the block drivers pretty much need to 
run under the I/O thread lock, and coroutines will not help if the I/O 
thread is taken by another thread.  It's hard/unreliable/pointless to 
ping-pong migration between threads.

2) there already are plans to reimplement block migration... it's called 
streaming :) and not coincidentially it reuses some of the block 
migration code.


Here is how it would go:

1) remotely stream the block devices from machine A to machine B.  Keep 
them in sync via the mirroring block device when the streaming is finished.

2) As soon as they are in sync, start live migration.

3) When the source QEMU exits, the destination machine runs  and 
switches over to its local storage.


Now, how do you do remote streaming?  NBD.  It does anything you need 
except perhaps TRIM---also easily added.

1a) Start qemu-nbd on the remote machine, writing to the destination image.

1b) Start (local) live streaming of the source block device to an NBD 
client that points to the qemu-nbd occurrence.

2a) Start qemu -S -incoming on the remote machine.  Use the same image 
you used in (1a) for the disk.

2b) As soon as streaming finishes, _keep mirroring_ on the source and 
start live migration.

3) Wait until after the NBD server closes the connection with the client 
before starting the destination machine.


Advantages:

1) Streaming and migration cannot be done at the same time, this should 
make the limitation much easier to fix.  I don't know if Marcelo's code 
can stream multiple devices at the time but, even if it can't, taking 
migration out of the picture should make that easier.

2) Easier to make improvements to migration.

3) Perhaps separating block/RAM migration would make convergence of 
pre-copy migration easier?

Disadvantages:

1) It requires one listening port per disk on the destination machine 
but, if desired, libvirt can tunnel the NBD connection just like it 
tunnels the migration data---it can even encrypt it and whatnot.

2) Complicated to do by hand.  If you want to simplify this, qemu-nbd 
can be embedded in qemu itself as -incoming-nbd or as a new monitor 
command, and similar syntactic sugar can be provided on the source.  But 
this requires more changes and the sequence would in practice be managed 
by libvirt, of course.  The manual steps still allow the idea to be 
demonstrated with zero changes to the code besides streaming.

3) ... put yours here.


How does it sound?

Paolo

Re: [Qemu-devel] Killing block migration in qemu?

[Stefan Hajnoczi]

On Wed, Aug 17, 2011 at 5:51 PM, Paolo Bonzini <pbonzini@redhat.com> wrote:
> following discussions yesterday with Juan Quintela and Marcelo Tosatti, here
> is my humble proposal: remove block migration from qemu master.  It seems to
> me that keeping block migration is going to slow down further improvements
> on migration.  The main problems are:
>
> 1) there are very good reasons to move migration to a separate thread. Only
> a limited amount of extra locking, perhaps none is needed in order to do so
> for RAM and devices.  But the block drivers pretty much need to run under
> the I/O thread lock, and coroutines will not help if the I/O thread is taken
> by another thread.  It's hard/unreliable/pointless to ping-pong migration
> between threads.

The image streaming approach will also run in the I/O thread for the
mid-term future.  Is the problem that the block migration code today
is too tied into the actual migration code path and therefore stops
from using it when migration happens in a separate thread?

>
> 2) there already are plans to reimplement block migration... it's called
> streaming :) and not coincidentially it reuses some of the block migration
> code.

What are the concrete issues with the existing block migration code?
I'm not disagreeing that we should move to a streaming approach but I
simply don't know the details of the existing block migration code.

> Here is how it would go:

This sounds reasonable.  In fact we can do both pre-copy and post-copy
block migration using streaming (+mirroring).

Stefan

Re: [Qemu-devel] Killing block migration in qemu?

[Paolo Bonzini]

On 08/17/2011 10:21 AM, Stefan Hajnoczi wrote:
> On Wed, Aug 17, 2011 at 5:51 PM, Paolo Bonzini<pbonzini@redhat.com>  wrote:
>> following discussions yesterday with Juan Quintela and Marcelo Tosatti, here
>> is my humble proposal: remove block migration from qemu master.  It seems to
>> me that keeping block migration is going to slow down further improvements
>> on migration.  The main problems are:
>>
>> 1) there are very good reasons to move migration to a separate thread. Only
>> a limited amount of extra locking, perhaps none is needed in order to do so
>> for RAM and devices.  But the block drivers pretty much need to run under
>> the I/O thread lock, and coroutines will not help if the I/O thread is taken
>> by another thread.  It's hard/unreliable/pointless to ping-pong migration
>> between threads.
>
> The image streaming approach will also run in the I/O thread for the
> mid-term future.  Is the problem that the block migration code today
> is too tied into the actual migration code path and therefore stops
> from using it when migration happens in a separate thread?

Yes, the problem is that it is much harder to move block migration to a 
separate thread.  It will use a separate coroutine loader and busy loop 
on coroutine locks while holding the iothread.  I'm not even sure it 
won't deadlock.

Block streaming is anyway asynchronous, so it does not hurt that it 
stays in the I/O thread.

>> 2) there already are plans to reimplement block migration... it's called
>> streaming :) and not coincidentially it reuses some of the block migration
>> code.
>
> What are the concrete issues with the existing block migration code?

The code is fine, but it conflicts with a bunch of goals we have for RAM 
live migration.

> This sounds reasonable.  In fact we can do both pre-copy and post-copy
> block migration using streaming (+mirroring).

Right, post-copy is just another name of copy-on-read.  Pre-copy RAM 
migration and post-copy block migration might be the best of both worlds 
in many settings, since pre-copy block migration takes ages.

Paolo

Re: [Qemu-devel] Killing block migration in qemu?

[Anthony Liguori]

On 08/17/2011 11:51 AM, Paolo Bonzini wrote:
> Hi all,
>
> following discussions yesterday with Juan Quintela and Marcelo Tosatti,
> here is my humble proposal: remove block migration from qemu master.

+1

I was going to propose the same thing after the conference actually.

Regards,

Anthony Liguori