[Qemu-devel] question about live migration with storage

[Qemu-devel] question about live migration with storage

[Zhang Haoyu]

Hi,

I want to live migrate a vm with storage, with regard to the migration of storage,
should I use drive_mirror or traditional mechanism implemented in block-migration.c ?
Any advices?

Thanks,
Zhang Haoyu 

Re: [Qemu-devel] question about live migration with storage

[Zhang Haoyu]

On 2015-01-13 09:49:00, Zhang Haoyu wrote:
>Hi,
>
>I want to live migrate a vm with storage, with regard to the migration of storage,
>should I use drive_mirror or traditional mechanism implemented in block-migration.c ?
Because I don't use libvirtd to manage vm,
if I want to use drive_mirror to perform live migration with storage,
how to organize the flow via script?

>Any advices?
>
>Thanks,
>Zhang Haoyu

Re: [Qemu-devel] question about live migration with storage

[Paolo Bonzini]

On 13/01/2015 03:03, Zhang Haoyu wrote:
> >I want to live migrate a vm with storage, with regard to the migration of storage,
> >should I use drive_mirror or traditional mechanism implemented in block-migration.c ?
> 
> Because I don't use libvirtd to manage vm,
> if I want to use drive_mirror to perform live migration with storage,
> how to organize the flow via script?

The same as libvirt does.

Paolo

Re: [Qemu-devel] question about live migration with storage

[Zhang Haoyu]

On 2015-01-13 17:45:45, Paolo Bonzini wrote:
> 
> On 13/01/2015 03:03, Zhang Haoyu wrote:
> > >I want to live migrate a vm with storage, with regard to the migration of storage,
> > >should I use drive_mirror or traditional mechanism implemented in block-migration.c ?
> > 
> > Because I don't use libvirtd to manage vm,
> > if I want to use drive_mirror to perform live migration with storage,
> > how to organize the flow via script?
> 
> The same as libvirt does.
> 
Hi, Paolo,
what's advantages of drive_mirror over traditional mechanism implemented in block-migration.c ?
Why libvirt use drive_mirror instead of traditional iterative mechanism as the default way
of live migration with non-shared storage?

Thanks,
Zhang Haoyu
> Paolo

Re: [Qemu-devel] question about live migration with storage

[Paolo Bonzini]

On 14/01/2015 03:41, Zhang Haoyu wrote:
> Hi, Paolo,
> what's advantages of drive_mirror over traditional mechanism implemented in block-migration.c ?
> Why libvirt use drive_mirror instead of traditional iterative mechanism as the default way
> of live migration with non-shared storage?

1) Being able to choose which block devices are migrated, and whether
they are migrated incrementally or not.

2) Finer-grain control the parameters of block migration (dirty bitmap
granularity).

3) Block and RAM migration do not share the same socket and thus can
more easily be parallelized.

Note that 1-2 are not yet supported by libvirt as far as I remember.

Paolo

Re: [Qemu-devel] question about live migration with storage

[Zhang Haoyu]

On 2015-01-14 15:42:41, Paolo Bonzini wrote:
> 
> On 14/01/2015 03:41, Zhang Haoyu wrote:
> > Hi, Paolo,
> > what's advantages of drive_mirror over traditional mechanism implemented in block-migration.c ?
> > Why libvirt use drive_mirror instead of traditional iterative mechanism as the default way
> > of live migration with non-shared storage?
> 
> 1) Being able to choose which block devices are migrated, and whether
> they are migrated incrementally or not.
> 
Yes.
> 2) Finer-grain control the parameters of block migration (dirty bitmap
> granularity).
> 
> 3) Block and RAM migration do not share the same socket and thus can
> more easily be parallelized.
> 
But, because of the parallelization, how to calculate the progress?

BTW, the traditional iterative mechanism is buggy-implemented?
e.g., some bugs which are very difficult to fixed, or something?

Thanks,
Zhang Haoyu
> Note that 1-2 are not yet supported by libvirt as far as I remember.
> 
> Paolo

Re: [Qemu-devel] question about live migration with storage

[Paolo Bonzini]

On 14/01/2015 08:58, Zhang Haoyu wrote:
>> 2) Finer-grain control the parameters of block migration (dirty bitmap
>> granularity).
>>
>> 3) Block and RAM migration do not share the same socket and thus can
>> more easily be parallelized.
>>
> But, because of the parallelization, how to calculate the progress?

You can use "query-block-jobs".

> BTW, the traditional iterative mechanism is buggy-implemented?
> e.g., some bugs which are very difficult to fixed, or something?

There are no bugs that corrupt data. It's simply less flexible.

Regarding parallelization, the problems happen especially if you disable
migration bandwidth limits.  Then you'll see large chunks of RAM and
large chunks of block device data being sent.  This is a problem for
convergence in some workloads. Instead, with NBD-based storage migration
everything happens in parallel, and TCP makes sure that bandwidth is
split between the sockets.

If you have a migration bandwidth limit, NBD-based storage migration
will use a separate bandwidth limit for each disk and for RAM. Thus
network usage is less predictable than with block-migration.c.  On the
other hand, storage migration uses a lot of network so it's usually more
likely that you do not have migration bandwidth limits.

Paolo

> Thanks,
> Zhang Haoyu
>> Note that 1-2 are not yet supported by libvirt as far as I remember.
>>
>> Paolo
> 
> 
> 

Re: [Qemu-devel] question about live migration with storage

[Zhang Haoyu]

On 2015-01-14 17:07:08, Paolo Bonzini wrote:
> 
> 
> On 14/01/2015 08:58, Zhang Haoyu wrote:
> >> 2) Finer-grain control the parameters of block migration (dirty bitmap
> >> granularity).
> >>
> >> 3) Block and RAM migration do not share the same socket and thus can
> >> more easily be parallelized.
> >>
drive_mirror job is done in main-thread,
then how to accept the qmp_migrate request while drive_mirror is performing?
If need to wait for the completion of drive_mirror,
how to implement the parallelization between block and ram migration?

Thanks,
Zhang Haoyu
> > But, because of the parallelization, how to calculate the progress?
> 
> You can use "query-block-jobs".
> 
> > BTW, the traditional iterative mechanism is buggy-implemented?
> > e.g., some bugs which are very difficult to fixed, or something?
> 
> There are no bugs that corrupt data. It's simply less flexible.
> 
> Regarding parallelization, the problems happen especially if you disable
> migration bandwidth limits.  Then you'll see large chunks of RAM and
> large chunks of block device data being sent.  This is a problem for
> convergence in some workloads. Instead, with NBD-based storage migration
> everything happens in parallel, and TCP makes sure that bandwidth is
> split between the sockets.
> 
> If you have a migration bandwidth limit, NBD-based storage migration
> will use a separate bandwidth limit for each disk and for RAM. Thus
> network usage is less predictable than with block-migration.c.  On the
> other hand, storage migration uses a lot of network so it's usually more
> likely that you do not have migration bandwidth limits.
> 
> Paolo
> 
> > Thanks,
> > Zhang Haoyu
> >> Note that 1-2 are not yet supported by libvirt as far as I remember.
> >>
> >> Paolo

Re: [Qemu-devel] question about live migration with storage

[Paolo Bonzini]

On 15/01/2015 04:54, Zhang Haoyu wrote:
>>>> > >> 2) Finer-grain control the parameters of block migration (dirty bitmap
>>>> > >> granularity).
>>>> > >>
>>>> > >> 3) Block and RAM migration do not share the same socket and thus can
>>>> > >> more easily be parallelized.
>>>> > >>
> drive_mirror job is done in main-thread,
> then how to accept the qmp_migrate request while drive_mirror is performing?
> If need to wait for the completion of drive_mirror,
> how to implement the parallelization between block and ram migration?

drive_mirror runs in the background, using coroutines.

Paolo

Re: [Qemu-devel] question about live migration with storage

[Zhang Haoyu]

On 2015-01-15 17:11:49, Paolo Bonzini wrote:
> 
> On 15/01/2015 04:54, Zhang Haoyu wrote:
> >>>> > >> 2) Finer-grain control the parameters of block migration (dirty bitmap
> >>>> > >> granularity).
> >>>> > >>
> >>>> > >> 3) Block and RAM migration do not share the same socket and thus can
> >>>> > >> more easily be parallelized.
> >>>> > >>
> > drive_mirror job is done in main-thread,
> > then how to accept the qmp_migrate request while drive_mirror is performing?
> > If need to wait for the completion of drive_mirror,
> > how to implement the parallelization between block and ram migration?
> 
> drive_mirror runs in the background, using coroutines.
> 
I see, when waiting the completion of drive_mirror IO, the coroutine will be
switched back to main-thread to poll and process other events, like qmp request,
then after the IO completed, coroutine will be switched back in main-loop process, right?

Another question:
while starting to migrate storage, will the unused sector be allocated and transferred?
regarding the thin-provisioning qcow2 disk, will extra large IO requests be
performed, and more data be transferred for the unallocated clusters?

Thanks,
Zhang Haoyu
> Paolo

Re: [Qemu-devel] question about live migration with storage

[Paolo Bonzini]

On 15/01/2015 10:56, Zhang Haoyu wrote:
> I see, when waiting the completion of drive_mirror IO, the coroutine will be
> switched back to main-thread to poll and process other events, like qmp request,
> then after the IO completed, coroutine will be switched back in main-loop process, right?
> 
> Another question:
> while starting to migrate storage, will the unused sector be allocated and transferred?
> regarding the thin-provisioning qcow2 disk, will extra large IO requests be
> performed, and more data be transferred for the unallocated clusters?

No.

Paolo

Re: [Qemu-devel] question about live migration with storage

[Zhang Haoyu]

On 2015-01-15 18:08:39, Paolo Bonzini wrote:
> 
> On 15/01/2015 10:56, Zhang Haoyu wrote:
> > I see, when waiting the completion of drive_mirror IO, the coroutine will be
> > switched back to main-thread to poll and process other events, like qmp request,
> > then after the IO completed, coroutine will be switched back in main-loop process, right?
> > 
> > Another question:
> > while starting to migrate storage, will the unused sector be allocated and transferred?
> > regarding the thin-provisioning qcow2 disk, will extra large IO requests be
> > performed, and more data be transferred for the unallocated clusters?
> 
> No.
> 
My test results show that
when using thin-provisioning qcow2 image(created by qemu-img create -f qcow2 preallocation=metadata),
even the unallocated sectors will be transferred to destination, so much data is transferred,
so the qcow2 image in destination is full allocated.

Thanks,
Zhang Haoyu
> Paolo

Re: [Qemu-devel] question about live migration with storage

[Paolo Bonzini]

On 10/03/2015 03:01, Zhang Haoyu wrote:
> My test results show that when using thin-provisioning qcow2
> image(created by qemu-img create -f qcow2 preallocation=metadata), 
> even the unallocated sectors will be transferred to destination, so
> much data is transferred, so the qcow2 image in destination is full
> allocated.

Yes, metadata preallocation actually allocates all sectors.  Only, the
qcow2 image is sparse so the sectors do not consume space on the source.

I think you can avoid the problem by passing file.detect-zeroes=unmap in
the -drive option of the destination QEMU.  Kevin/Peter, can you confirm
this should work?

Paolo