[Qemu-devel] vm live storage migration with snapshots

[Qemu-devel] vm live storage migration with snapshots

[Edward Young]

[-- Attachment #1: Type: text/plain, Size: 716 bytes --]

Hi all,

I'm investigating the ways to improve the live migration performance in
libvirt. I have a question about the vm live storage migration. the
platform is libvirt + qemu + kvm

If we want to migrate a running vm with its virtual disk images to another
node.
we can use 'virsh migrate ....' commands.

What if this vm has a number of disk-only external snapshots? In the
current version, how can live migrate this vm?

Is it possible to blockcommit the snapshots to the base image and later
perform the migration, without shutting down the running vm?

Or is it possible to iteratively transfer all the snapshots to the
destination and later live migrate only the latest new data?

Thanks in advance.

Best,
Ed

[-- Attachment #2: Type: text/html, Size: 940 bytes --]

Re: [Qemu-devel] [libvirt] vm live storage migration with snapshots

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2528 bytes --]

On 02/11/2015 10:08 AM, Edward Young wrote:
> Hi all,

[probably didn't need to cross-post to quite that wide of an audience,
oh well]

> 
> I'm investigating the ways to improve the live migration performance in
> libvirt. I have a question about the vm live storage migration. the
> platform is libvirt + qemu + kvm
> 
> If we want to migrate a running vm with its virtual disk images to another
> node.
> we can use 'virsh migrate ....' commands.
> 
> What if this vm has a number of disk-only external snapshots? In the
> current version, how can live migrate this vm?

Are the snapshots based on shared storage, or local-only storage?

If I understand your question correctly, you are starting with this
situation:

base <- mid <- active

and want to make sure that on the destination, the guest still sees the
same contents as 'active' on the source, whether or not the destination
still sees a backing chain or just one file.

> 
> Is it possible to blockcommit the snapshots to the base image and later
> perform the migration, without shutting down the running vm?

Yes, but that may not be the fastest solution.  That is, you would be
going from:

base <- mid <- active

to

base'

where base' is the active image containing all the committed state, then
migrate that file.  If base is not shared, that means qemu has to
migrate the entire disk state.  And you also no longer have the external
snapshots to revert to on the destination.

> 
> Or is it possible to iteratively transfer all the snapshots to the
> destination and later live migrate only the latest new data?
> 

Yes, that works too, and is probably faster, especially if you have
out-of-band means for sharing read-only state between source and
destination.

In fact, if you can create yet another (temporary) external snapshot,
but this time on shared storage, and have storage backends that let you
instantly remap backing files to be visible into the destination, then
it is as simple as:

create a snapshot with shared storage destination
base <- mid <- active(frozen) <- shared
copy base, mid, and active to the destination
live migrate using shared storage instead of doing storage migration
block-commit shared back into active
base <- mid <- active

where the shared storage only needs to hold as much disk state as
diverges during the time of the live migration.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [libvirt] vm live storage migration with snapshots

[Edward Young]

[-- Attachment #1: Type: text/plain, Size: 3460 bytes --]

Hi Eric,

Thanks for your reply! I have the follow up questions blew.

On Wed, Feb 11, 2015 at 11:52 AM, Eric Blake <eblake@redhat.com> wrote:

> On 02/11/2015 10:08 AM, Edward Young wrote:
> > Hi all,
>
> [probably didn't need to cross-post to quite that wide of an audience,
> oh well]
>
> >
> > I'm investigating the ways to improve the live migration performance in
> > libvirt. I have a question about the vm live storage migration. the
> > platform is libvirt + qemu + kvm
> >
> > If we want to migrate a running vm with its virtual disk images to
> another
> > node.
> > we can use 'virsh migrate ....' commands.
> >
> > What if this vm has a number of disk-only external snapshots? In the
> > current version, how can live migrate this vm?
>
> Are the snapshots based on shared storage, or local-only storage?
>

Yes, I'm talking about the local-only storage.


> If I understand your question correctly, you are starting with this
> situation:
>
> base <- mid <- active
>
> and want to make sure that on the destination, the guest still sees the
> same contents as 'active' on the source, whether or not the destination
> still sees a backing chain or just one file.
>
> >
> > Is it possible to blockcommit the snapshots to the base image and later
> > perform the migration, without shutting down the running vm?
>
> Yes, but that may not be the fastest solution.  That is, you would be
> going from:
>
> base <- mid <- active
>
> to
>
> base'
>
> where base' is the active image containing all the committed state, then
> migrate that file.  If base is not shared, that means qemu has to
> migrate the entire disk state.  And you also no longer have the external
> snapshots to revert to on the destination.
>

Yes, I agree with you. In this case, we need to migrate the entire disk
state. In this case, there is no snapshot involved. we just perform the
regular migration with 'virsh migrate....'. Is this correct?


>
> >
> > Or is it possible to iteratively transfer all the snapshots to the
> > destination and later live migrate only the latest new data?
> >
>
> Yes, that works too, and is probably faster, especially if you have
> out-of-band means for sharing read-only state between source and
> destination.
>

My question is here. If we do not have any shared storage resource between
source and destination (eg. long distance VM migration), how can we migrate
the latest new data to the destination? we can copy the base, mid to
destination manually, then how can we migrate the active snapshot( new data
goes in)? I learned that drive_mirror in qemu is built to finish this, but
do not understand clearly. Could you elaborate for me, or provide an
example?


>
> In fact, if you can create yet another (temporary) external snapshot,
> but this time on shared storage, and have storage backends that let you
> instantly remap backing files to be visible into the destination, then
> it is as simple as:
>
> create a snapshot with shared storage destination
> base <- mid <- active(frozen) <- shared
> copy base, mid, and active to the destination
> live migrate using shared storage instead of doing storage migration
> block-commit shared back into active
> base <- mid <- active
>
> where the shared storage only needs to hold as much disk state as
> diverges during the time of the live migration.
>

Thanks a lot!
Ed

>
> --
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
>
>

[-- Attachment #2: Type: text/html, Size: 4952 bytes --]

Re: [Qemu-devel] [libvirt] vm live storage migration with snapshots

[Eric Blake]

[-- Attachment #1: Type: text/plain, Size: 2080 bytes --]

On 02/11/2015 02:07 PM, Edward Young wrote:
>>> What if this vm has a number of disk-only external snapshots? In the
>>> current version, how can live migrate this vm?
>>
>> Are the snapshots based on shared storage, or local-only storage?
>>
> 
> Yes, I'm talking about the local-only storage.

Okay, glad I guessed as much, then.

> 
> Yes, I agree with you. In this case, we need to migrate the entire disk
> state. In this case, there is no snapshot involved. we just perform the
> regular migration with 'virsh migrate....'. Is this correct?

Using 'virsh migrate --copy-storage-all' would indeed migrate the entire
disk, if you can't supply shared storage.

>>> Or is it possible to iteratively transfer all the snapshots to the
>>> destination and later live migrate only the latest new data?
>>>
>>
>> Yes, that works too, and is probably faster, especially if you have
>> out-of-band means for sharing read-only state between source and
>> destination.
>>
> 
> My question is here. If we do not have any shared storage resource between
> source and destination (eg. long distance VM migration), how can we migrate
> the latest new data to the destination? we can copy the base, mid to
> destination manually, then how can we migrate the active snapshot( new data
> goes in)? I learned that drive_mirror in qemu is built to finish this, but
> do not understand clearly. Could you elaborate for me, or provide an
> example?

Use 'virsh migrate --copy-storage-inc' to migrate only the incremental
changes, which assumes that the destination can already see the same
read-only backing data that the source sees.  In fact, modern
libvirt/qemu does this for you by setting up an NBD server on the
destination, doing a data mirror from the source into the destination
(so that you DO have shared storage, at least for the duration of the
migration), then doing the live migration, then tearing down the NBD
mirroring link.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]

Re: [Qemu-devel] [libvirt] vm live storage migration with snapshots

[Edward Young]

[-- Attachment #1: Type: text/plain, Size: 3116 bytes --]

Hi Eric,

Please see the blew:

On Wed, Feb 11, 2015 at 3:12 PM, Eric Blake <eblake@redhat.com> wrote:

> On 02/11/2015 02:07 PM, Edward Young wrote:
> >>> What if this vm has a number of disk-only external snapshots? In the
> >>> current version, how can live migrate this vm?
> >>
> >> Are the snapshots based on shared storage, or local-only storage?
> >>
> >
> > Yes, I'm talking about the local-only storage.
>
> Okay, glad I guessed as much, then.
>
> >
> > Yes, I agree with you. In this case, we need to migrate the entire disk
> > state. In this case, there is no snapshot involved. we just perform the
> > regular migration with 'virsh migrate....'. Is this correct?
>
> Using 'virsh migrate --copy-storage-all' would indeed migrate the entire
> disk, if you can't supply shared storage.
>
> >>> Or is it possible to iteratively transfer all the snapshots to the
> >>> destination and later live migrate only the latest new data?
> >>>
> >>
> >> Yes, that works too, and is probably faster, especially if you have
> >> out-of-band means for sharing read-only state between source and
> >> destination.
> >>
> >
> > My question is here. If we do not have any shared storage resource
> between
> > source and destination (eg. long distance VM migration), how can we
> migrate
> > the latest new data to the destination? we can copy the base, mid to
> > destination manually, then how can we migrate the active snapshot( new
> data
> > goes in)? I learned that drive_mirror in qemu is built to finish this,
> but
> > do not understand clearly. Could you elaborate for me, or provide an
> > example?
>
> Use 'virsh migrate --copy-storage-inc' to migrate only the incremental
> changes, which assumes that the destination can already see the same
> read-only backing data that the source sees.


I perform a simple test, but failed.

In the source, I create:   base <- mid <- active (2 snapshots, the active
one is the current one)
In order to migrate this vm to the destination, I manually copy both base
and mid to the destination, and put them in the sam e directory
/var/lib/libvirt/images/
Then I call the following commands:
'sudo virsh migrate --live --unsafe --verbose --copy-storage-inc vm1
qemu+ssh://192.168.1.3/system tcp://192.168.1.3'

The error I got is:
error: Requested operation is not valid: cannot migrate domain with 2
snapshots.

I do not know the reason for this.


> In fact, modern
> libvirt/qemu does this for you by setting up an NBD server on the
> destination, doing a data mirror from the source into the destination
> (so that you DO have shared storage, at least for the duration of the
> migration), then doing the live migration, then tearing down the NBD
> mirroring link.
>

Sounds interesting! Can you provide me an example or point me to a link
about how to use this function?

Sorry to take you so much time to answer my questions. I appreciate it
deeply. After I solve this issue, I will write a blog and share these
knowledge with other people.

Best,
Ed


> --
> Eric Blake   eblake redhat com    +1-919-301-3266
> Libvirt virtualization library http://libvirt.org
>
>

[-- Attachment #2: Type: text/html, Size: 4709 bytes --]