[Qemu-devel] [PATCH 0/3] dataplane: virtio-blk live migration with x-data-plane=on

[Qemu-devel] [PATCH 0/3] dataplane: virtio-blk live migration with x-data-plane=on

[Stefan Hajnoczi]

These patches add live migration support to -device virtio-blk-pci,x-data-plane=on.

Patch 1 has already been posted and merged into the block tree.  I have
included it for convenience.

Patches 2 & 3 implement a switch from dataplane mode back to regular virtio-blk
mode when migration starts.  This way live migration works.

If migration is cancelled or the guest accesses the virtio-blk device after
completion, dataplane starts again.

Since this approach is so small, it's more palatable for QEMU 1.6 than trying
to make vring.c log dirty memory.  It makes dataplane usable in situations
where live migration is a requirement.

Stefan Hajnoczi (3):
  dataplane: sync virtio.c and vring.c virtqueue state
  migration: notify migration state before starting thread
  dataplane: enable virtio-blk x-data-plane=on live migration

 hw/block/dataplane/virtio-blk.c     | 19 +++++++++----------
 hw/block/virtio-blk.c               | 32 ++++++++++++++++++++++++++++++++
 hw/virtio/dataplane/vring.c         |  8 +++++---
 include/hw/virtio/dataplane/vring.h |  2 +-
 include/hw/virtio/virtio-blk.h      |  1 +
 migration.c                         |  4 +++-
 6 files changed, 51 insertions(+), 15 deletions(-)

-- 
1.8.1.4

Re: [Qemu-devel] [PATCH 0/3] dataplane: virtio-blk live migration with x-data-plane=on

[yinyin]

hi, stefan:
	I use systemtap to test this patch,the migration will success. But I found the dataplane will start again after migration start. the virtio_blk_handle_output will start dataplane.


virtio_blk_data_plane_stop pid:29037 tid:29037
 0x6680fe : virtio_blk_data_plane_stop+0x0/0x232 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x667da2 : virtio_blk_data_plane_destroy+0x33/0x70 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x66a2e4 : virtio_blk_migration_state_changed+0x7c/0x12d [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x7740f4 : notifier_list_notify+0x59/0x79 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x5a5b16 : migrate_fd_connect+0xc6/0x104 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x5a3dba : tcp_wait_for_connect+0x6e/0x84 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x76c755 : wait_for_connect+0x170/0x19a [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x5a2e2c : qemu_iohandler_poll+0xec/0x188 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x5a3834 : main_loop_wait+0x92/0xc9 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x630083 : main_loop+0x5d/0x82 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x636954 : main+0x3666/0x369a [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x37cf01ecdd [/lib64/libc-2.12.so+0x1ecdd/0x393000]
virtio_blk_data_plane_start pid:29037 tid:29037
 0x667ddf : virtio_blk_data_plane_start+0x0/0x31f [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x669802 : virtio_blk_handle_output+0x9c/0x118 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x6ac6ff : virtio_queue_notify_vq+0x92/0xa8 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x6ae128 : virtio_queue_host_notifier_read+0x50/0x66 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x6ae1bb : virtio_queue_set_host_notifier_fd_handler+0x7d/0x93 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x59cdf1 : virtio_pci_set_host_notifier_internal+0x132/0x157 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x59e98c : virtio_pci_set_host_notifier+0x73/0x8e [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x6682a8 : virtio_blk_data_plane_stop+0x1aa/0x232 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x667da2 : virtio_blk_data_plane_destroy+0x33/0x70 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x66a2e4 : virtio_blk_migration_state_changed+0x7c/0x12d [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x7740f4 : notifier_list_notify+0x59/0x79 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x5a5b16 : migrate_fd_connect+0xc6/0x104 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x5a3dba : tcp_wait_for_connect+0x6e/0x84 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x76c755 : wait_for_connect+0x170/0x19a [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x5a2e2c : qemu_iohandler_poll+0xec/0x188 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x5a3834 : main_loop_wait+0x92/0xc9 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x630083 : main_loop+0x5d/0x82 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x636954 : main+0x3666/0x369a [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x37cf01ecdd [/lib64/libc-2.12.so+0x1ecdd/0x393000]


systemtap scripts:
probe process("/root/dataplane/qemu/x86_64-softmmu/qemu-kvm").function("virtio_blk_data_plane_start")
{
  printf("virtio_blk_data_plane_start pid:%d tid:%d\n",pid(),tid())
  print_ubacktrace();
}


probe process("/root/dataplane/qemu/x86_64-softmmu/qemu-kvm").function("virtio_blk_data_plane_stop")
{
  printf("virtio_blk_data_plane_stop pid:%d tid:%d\n",pid(),tid())
  print_ubacktrace();
}


Yin Yin
yin.yin@cs2c.com.cn

Re: [Qemu-devel] [PATCH 0/3] dataplane: virtio-blk live migration with x-data-plane=on

[Stefan Hajnoczi]

On Fri, Jul 19, 2013 at 01:33:12PM +0800, yinyin wrote:
> 	I use systemtap to test this patch,the migration will success. But I found the dataplane will start again after migration start. the virtio_blk_handle_output will start dataplane.

Hi Yin Yin,
Thank you for testing the patch.  It is not clear to me whether you
encountered a problem or not.

It is expected that the destination will start the dataplane thread.
Was there a crash or some reason why you posted these traces?

Stefan

Re: [Qemu-devel] [PATCH 0/3] dataplane: virtio-blk live migration with x-data-plane=on

[yinyin]

Hi, Stefan:
	during the migration, the source, not the destination, will start dataplane again....
	I think the process of migration with dataplane as follows:
1. migration begin to start
2. the migration source stop the dataplane
3. do migration ...
4. migration completed, the destination start the dataplane.

when migration start, the source dataplane should already stopped, and not start again, if there is no cancel or abort.
But the trace show that, in step 3 above, the source dataplane will be start by virtio_blk_handle_output. I'm afraid of some inconsistent will happen there.Is it right?

There is no crash found, I just use this trace to understand the flow of dataplane migration. 

Yin Yin
yin.yin@cs2c.com.cn

 
在 2013-7-23，下午4:51，Stefan Hajnoczi <stefanha@redhat.com> 写道：

> On Fri, Jul 19, 2013 at 01:33:12PM +0800, yinyin wrote:
>> 	I use systemtap to test this patch,the migration will success. But I found the dataplane will start again after migration start. the virtio_blk_handle_output will start dataplane.
> 
> Hi Yin Yin,
> Thank you for testing the patch.  It is not clear to me whether you
> encountered a problem or not.
> 
> It is expected that the destination will start the dataplane thread.
> Was there a crash or some reason why you posted these traces?
> 
> Stefan
> 

Re: [Qemu-devel] [PATCH 0/3] dataplane: virtio-blk live migration with x-data-plane=on

[Andreas Färber]

Hi,

Am 23.07.2013 11:19, schrieb yinyin:
> Hi, Stefan:
> 	during the migration, the source, not the destination, will start dataplane again....
> 	I think the process of migration with dataplane as follows:
> 1. migration begin to start
> 2. the migration source stop the dataplane
> 3. do migration ...
> 4. migration completed, the destination start the dataplane.

I can't speak for the dataplane, but in general the source guest is
expected to continue working during live migration (that's the "live"
part) until it has been fully transferred to the destination.

HTH,
Andreas

> when migration start, the source dataplane should already stopped, and not start again, if there is no cancel or abort.
> But the trace show that, in step 3 above, the source dataplane will be start by virtio_blk_handle_output. I'm afraid of some inconsistent will happen there.Is it right?
> 
> There is no crash found, I just use this trace to understand the flow of dataplane migration. 
> 
> Yin Yin
> yin.yin@cs2c.com.cn
> 
>  
> 在 2013-7-23，下午4:51，Stefan Hajnoczi <stefanha@redhat.com> 写道：
> 
>> On Fri, Jul 19, 2013 at 01:33:12PM +0800, yinyin wrote:
>>> 	I use systemtap to test this patch,the migration will success. But I found the dataplane will start again after migration start. the virtio_blk_handle_output will start dataplane.
>>
>> Hi Yin Yin,
>> Thank you for testing the patch.  It is not clear to me whether you
>> encountered a problem or not.
>>
>> It is expected that the destination will start the dataplane thread.
>> Was there a crash or some reason why you posted these traces?
>>
>> Stefan
>>
> 
> 


-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 0/3] dataplane: virtio-blk live migration with x-data-plane=on

[yinyin]

Hi,
在 2013-7-23，下午5:30，Andreas Färber <afaerber@suse.de> 写道：

> Hi,
> 
> Am 23.07.2013 11:19, schrieb yinyin:
>> Hi, Stefan:
>> 	during the migration, the source, not the destination, will start dataplane again....
>> 	I think the process of migration with dataplane as follows:
>> 1. migration begin to start
>> 2. the migration source stop the dataplane
>> 3. do migration ...
>> 4. migration completed, the destination start the dataplane.
> 
> I can't speak for the dataplane, but in general the source guest is
> expected to continue working during live migration (that's the "live"
> part) until it has been fully transferred to the destination.

when dataplane stop, the source guest can continue work, but not use dataplane thread.

> 
> HTH,
> Andreas
> 
>> when migration start, the source dataplane should already stopped, and not start again, if there is no cancel or abort.
>> But the trace show that, in step 3 above, the source dataplane will be start by virtio_blk_handle_output. I'm afraid of some inconsistent will happen there.Is it right?
>> 
>> There is no crash found, I just use this trace to understand the flow of dataplane migration. 
>> 
>> Yin Yin
>> yin.yin@cs2c.com.cn
>> 
>> 
>> 在 2013-7-23，下午4:51，Stefan Hajnoczi <stefanha@redhat.com> 写道：
>> 
>>> On Fri, Jul 19, 2013 at 01:33:12PM +0800, yinyin wrote:
>>>> 	I use systemtap to test this patch,the migration will success. But I found the dataplane will start again after migration start. the virtio_blk_handle_output will start dataplane.
>>> 
>>> Hi Yin Yin,
>>> Thank you for testing the patch.  It is not clear to me whether you
>>> encountered a problem or not.
>>> 
>>> It is expected that the destination will start the dataplane thread.
>>> Was there a crash or some reason why you posted these traces?
>>> 
>>> Stefan
>>> 
>> 
>> 
> 
> 
> -- 
> SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
> GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg
> 

Re: [Qemu-devel] [PATCH 0/3] dataplane: virtio-blk live migration with x-data-plane=on

[Andreas Färber]

Am 23.07.2013 11:43, schrieb yinyin:
> 在 2013-7-23，下午5:30，Andreas Färber <afaerber@suse.de> 写道：
>> Am 23.07.2013 11:19, schrieb yinyin:
>>> 	during the migration, the source, not the destination, will start dataplane again....
>>> 	I think the process of migration with dataplane as follows:
>>> 1. migration begin to start
>>> 2. the migration source stop the dataplane
>>> 3. do migration ...
>>> 4. migration completed, the destination start the dataplane.
>>
>> I can't speak for the dataplane, but in general the source guest is
>> expected to continue working during live migration (that's the "live"
>> part) until it has been fully transferred to the destination.
> 
> when dataplane stop, the source guest can continue work, but not use dataplane thread.

So how would it do I/O then?

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] [PATCH 0/3] dataplane: virtio-blk live migration with x-data-plane=on

[Stefan Hajnoczi]

On Tue, Jul 23, 2013 at 11:19 AM, yinyin <yin.yin@cs2c.com.cn> wrote:
>         during the migration, the source, not the destination, will start dataplane again....

Thanks for explaining.  The backtrace you posted is harmless.  The
code is written to work like this.

I have annotated it explaining what is going on:

virtio_blk_data_plane_start pid:29037 tid:29037
 0x37cf01ecdd [/lib64/libc-2.12.so+0x1ecdd/0x393000]
 0x636954 : main+0x3666/0x369a [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x630083 : main_loop+0x5d/0x82 [/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x5a3834 : main_loop_wait+0x92/0xc9
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x5a2e2c : qemu_iohandler_poll+0xec/0x188
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x76c755 : wait_for_connect+0x170/0x19a
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x5a3dba : tcp_wait_for_connect+0x6e/0x84
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x5a5b16 : migrate_fd_connect+0xc6/0x104
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]

Live migration is start on the source.

 0x7740f4 : notifier_list_notify+0x59/0x79
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]

Before starting migration we notify listeners that migration is starting.

 0x66a2e4 : virtio_blk_migration_state_changed+0x7c/0x12d
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x667da2 : virtio_blk_data_plane_destroy+0x33/0x70
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]

Dataplane is a listener, it wants to know when migration begins.  It will stop
dataplane and switch to regular virtio-blk operation during live migration
iterations (while RAM is being transferred but the VM is still running on the
source).

 0x6682a8 : virtio_blk_data_plane_stop+0x1aa/0x232
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]

We are stopping dataplane right here.

 0x59e98c : virtio_pci_set_host_notifier+0x73/0x8e
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x59cdf1 : virtio_pci_set_host_notifier_internal+0x132/0x157
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x6ae1bb : virtio_queue_set_host_notifier_fd_handler+0x7d/0x93
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x6ae128 : virtio_queue_host_notifier_read+0x50/0x66
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]
 0x6ac6ff : virtio_queue_notify_vq+0x92/0xa8
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]

As part of stopping dataplane we first flush any pending virtqueue kicks.

 0x669802 : virtio_blk_handle_output+0x9c/0x118
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]

There was a pending virtqueue kick from the guest so we will complete it before
stopping dataplane.

 0x667ddf : virtio_blk_data_plane_start+0x0/0x31f
[/root/dataplane/qemu/x86_64-softmmu/qemu-kvm]

The dataplane thread was not running so it is temporarily started to process
these final requests.  Once they are finished it will stop again.

Stefan