Re: snapmirror functionality using qcow2 and rsync?

Re: snapmirror functionality using qcow2 and rsync?

[Fred van Zwieten]

Hello,

The website said user questions are welcome and for one-off questions
I could just mail, so here goes:

Background:
NetApp Snashot functionality gives you application consistent
snapshots of data. Just inform the app a snapshot is about to be made,
depending on the app, it needs to go in to some sort of backup mode,
of just stop and flush outstanding I/O. Next, a snapshot is made and
everything just runs on. Because of the underlying WAFL filesystem
design, the snapshot always points to the blocks at the time of the
creation without needing to do any COW.

Layered on top op this functionality is SnapMirror, where the delta
between this snapshot and a previous snapshot (both being static in
nature), is asynchronously replicated to a second system. There, this
delta is applied to a copy of the disk as a local snapshot.

This setup gives you application consistent data disks on a remote
location as a form of disaster tolerancy. The RPO is the
snapshot/snapmirror frequency.

KVM:
My question is rather simple. Could something like this be implemented
with kvm-img and rsync and/or lvm? I've played with the backing_file
option, but this means I have to shutdown the vm a boot is on the new
file to let this work.

Greetz,

Fred

Re: snapmirror functionality using qcow2 and rsync?

[Stefan Hajnoczi]

On Sun, May 15, 2011 at 1:16 PM, Fred van Zwieten <fvzwieten@gmail.com> wrote:
> Background:
> NetApp Snashot functionality gives you application consistent
> snapshots of data. Just inform the app a snapshot is about to be made,
> depending on the app, it needs to go in to some sort of backup mode,
> of just stop and flush outstanding I/O. Next, a snapshot is made and
> everything just runs on. Because of the underlying WAFL filesystem
> design, the snapshot always points to the blocks at the time of the
> creation without needing to do any COW.
>
> Layered on top op this functionality is SnapMirror, where the delta
> between this snapshot and a previous snapshot (both being static in
> nature), is asynchronously replicated to a second system. There, this
> delta is applied to a copy of the disk as a local snapshot.
>
> This setup gives you application consistent data disks on a remote
> location as a form of disaster tolerancy. The RPO is the
> snapshot/snapmirror frequency.
>
> KVM:
> My question is rather simple. Could something like this be implemented
> with kvm-img and rsync and/or lvm? I've played with the backing_file
> option, but this means I have to shutdown the vm a boot is on the new
> file to let this work.

This recent thread might interest you:
http://lists.gnu.org/archive/html/qemu-devel/2011-05/msg00733.html

Basically you have to cobble it together yourself at the moment but
there is activity around live snapshots and dirty block tracking, so
hopefully they will be available as APIs in the future.

Stefan

Re: snapmirror functionality using qcow2 and rsync?

[Jagane Sundar]

Hello Fred,

As Stefan points out, there is a lot of interest in such functionality.

I am the author of a feature called Livebackup - which is aimed at
solving exactly this problem - backup of live virtual machines. The
additional constraint I place is that backup of the VM should be
enabled regardless of the type of storage in use - file on local disk,
file on NAS, LVM logical volume on local disk, LVM volume on an
iSCSI volume.

More information is at:
http://wiki.qemu.org/Features/Livebackup

I investigated the approach that you outline below - LVM snapshot
followed by transfer of the blocks out of the VM host. The problem
I ran into was that there was no easy way to identify the blocks that
were modified since the last backup was taken. To the best of my
knowledge, rsync uses checksums of entire files to determine whether
a file was modified or not - in our use case the whole virtual disk
is one single file, do rsync will not work well.

One of the 'features' of Livebackup is that I chose to invent my own
rudimentary protocol for transferring the blocks out of the VM
host. iSCSI seems like a standard protocol that would suit the
purpose; however it does not fit well with the architecture of
Livebackup - I would probably need to implement an iSCSI target
function inside Livebackup.

Thanks,
Jagane

On 5/15/2011 7:11 AM, Stefan Hajnoczi wrote:
> On Sun, May 15, 2011 at 1:16 PM, Fred van Zwieten<fvzwieten@gmail.com>  wrote:
>> Background:
>> NetApp Snashot functionality gives you application consistent
>> snapshots of data. Just inform the app a snapshot is about to be made,
>> depending on the app, it needs to go in to some sort of backup mode,
>> of just stop and flush outstanding I/O. Next, a snapshot is made and
>> everything just runs on. Because of the underlying WAFL filesystem
>> design, the snapshot always points to the blocks at the time of the
>> creation without needing to do any COW.
>>
>> Layered on top op this functionality is SnapMirror, where the delta
>> between this snapshot and a previous snapshot (both being static in
>> nature), is asynchronously replicated to a second system. There, this
>> delta is applied to a copy of the disk as a local snapshot.
>>
>> This setup gives you application consistent data disks on a remote
>> location as a form of disaster tolerancy. The RPO is the
>> snapshot/snapmirror frequency.
>>
>> KVM:
>> My question is rather simple. Could something like this be implemented
>> with kvm-img and rsync and/or lvm? I've played with the backing_file
>> option, but this means I have to shutdown the vm a boot is on the new
>> file to let this work.
> This recent thread might interest you:
> http://lists.gnu.org/archive/html/qemu-devel/2011-05/msg00733.html
>
> Basically you have to cobble it together yourself at the moment but
> there is activity around live snapshots and dirty block tracking, so
> hopefully they will be available as APIs in the future.
>
> Stefan

Re: snapmirror functionality using qcow2 and rsync?

[Fred van Zwieten]

Hello Jagane,

My original idea was this: Have a data disk implemented as an LV. Use
drbd or dm-replicator to asynchronously replicate the vg to a remote
location. When I make a snapshot of the LV, the snapshot will be
replicated along with it. Have an application quiesce itself and make
the LV snapshot (doing it from within a vm is also possible, but a bit
more challenging) on the source site. This means there is an
application consistent snapshot available. Problem is the lvm
administration need also to be replicated to the second site. Quite
difficult I think.

Another idea, not using lvm but qcow2, is to create a snapshot in a
separate file from the master file. The snapshot file holds the
changed blocks. Then, when you create a second snapshot, the changes
from that point onwards will be placed in the second snapshot, making
the first one static. The first one can be used for replication. When
that is done, it can be deleted. A new snapshot is created and will
function as the new target for changed blocks and the snapshot before
that one will be replicated and so on. This means 2 snapshot files are
needed to make this work. The snapshots on the other site will be
stored next to a copy of the master file. When needed a vm can be
started on any of these snapshots. Guess what, the NetApp SnapMirror
mechanism also uses two snapshot's to make it work.

I know NetApp and LVM but am rather new to qcow2, so i am not sure if
i'm suggesting anything remotely sane here, so forgive me that's the
case.

Greetz,

Fred



On Sun, May 15, 2011 at 7:16 PM, Jagane Sundar <jagane@sundar.org> wrote:
> Hello Fred,
>
> As Stefan points out, there is a lot of interest in such functionality.
>
> I am the author of a feature called Livebackup - which is aimed at
> solving exactly this problem - backup of live virtual machines. The
> additional constraint I place is that backup of the VM should be
> enabled regardless of the type of storage in use - file on local disk,
> file on NAS, LVM logical volume on local disk, LVM volume on an
> iSCSI volume.
>
> More information is at:
> http://wiki.qemu.org/Features/Livebackup
>
> I investigated the approach that you outline below - LVM snapshot
> followed by transfer of the blocks out of the VM host. The problem
> I ran into was that there was no easy way to identify the blocks that
> were modified since the last backup was taken. To the best of my
> knowledge, rsync uses checksums of entire files to determine whether
> a file was modified or not - in our use case the whole virtual disk
> is one single file, do rsync will not work well.
>
> One of the 'features' of Livebackup is that I chose to invent my own
> rudimentary protocol for transferring the blocks out of the VM
> host. iSCSI seems like a standard protocol that would suit the
> purpose; however it does not fit well with the architecture of
> Livebackup - I would probably need to implement an iSCSI target
> function inside Livebackup.
>
> Thanks,
> Jagane
>
> On 5/15/2011 7:11 AM, Stefan Hajnoczi wrote:
>>
>> On Sun, May 15, 2011 at 1:16 PM, Fred van Zwieten<fvzwieten@gmail.com>
>>  wrote:
>>>
>>> Background:
>>> NetApp Snashot functionality gives you application consistent
>>> snapshots of data. Just inform the app a snapshot is about to be made,
>>> depending on the app, it needs to go in to some sort of backup mode,
>>> of just stop and flush outstanding I/O. Next, a snapshot is made and
>>> everything just runs on. Because of the underlying WAFL filesystem
>>> design, the snapshot always points to the blocks at the time of the
>>> creation without needing to do any COW.
>>>
>>> Layered on top op this functionality is SnapMirror, where the delta
>>> between this snapshot and a previous snapshot (both being static in
>>> nature), is asynchronously replicated to a second system. There, this
>>> delta is applied to a copy of the disk as a local snapshot.
>>>
>>> This setup gives you application consistent data disks on a remote
>>> location as a form of disaster tolerancy. The RPO is the
>>> snapshot/snapmirror frequency.
>>>
>>> KVM:
>>> My question is rather simple. Could something like this be implemented
>>> with kvm-img and rsync and/or lvm? I've played with the backing_file
>>> option, but this means I have to shutdown the vm a boot is on the new
>>> file to let this work.
>>
>> This recent thread might interest you:
>> http://lists.gnu.org/archive/html/qemu-devel/2011-05/msg00733.html
>>
>> Basically you have to cobble it together yourself at the moment but
>> there is activity around live snapshots and dirty block tracking, so
>> hopefully they will be available as APIs in the future.
>>
>> Stefan
>
>

Re: snapmirror functionality using qcow2 and rsync?

[Dor Laor]

On 05/15/2011 10:18 PM, Fred van Zwieten wrote:
> Hello Jagane,
>
> My original idea was this: Have a data disk implemented as an LV. Use
> drbd or dm-replicator to asynchronously replicate the vg to a remote
> location. When I make a snapshot of the LV, the snapshot will be

It can work but it might be way too expensive if you have many unrelated 
LVs on that VG.

> replicated along with it. Have an application quiesce itself and make
> the LV snapshot (doing it from within a vm is also possible, but a bit
> more challenging) on the source site. This means there is an
> application consistent snapshot available. Problem is the lvm
> administration need also to be replicated to the second site. Quite
> difficult I think.
>
> Another idea, not using lvm but qcow2, is to create a snapshot in a
> separate file from the master file. The snapshot file holds the
> changed blocks. Then, when you create a second snapshot, the changes
> from that point onwards will be placed in the second snapshot, making
> the first one static. The first one can be used for replication. When
> that is done, it can be deleted. A new snapshot is created and will
> function as the new target for changed blocks and the snapshot before
> that one will be replicated and so on. This means 2 snapshot files are
> needed to make this work. The snapshots on the other site will be

There is no need to create 2 snapshots. A single snapshot is enough.
In order to have efficient diff sync we'll only need a way to poke into 
the image format file to get the dirty blocks. We can add a verb to 
qemu-img to do that.

> stored next to a copy of the master file. When needed a vm can be
> started on any of these snapshots. Guess what, the NetApp SnapMirror
> mechanism also uses two snapshot's to make it work.

In theory we can even utilize the storage vendor snapshot mechanism, 
NetApp in this case and create snapshots using it and not by 
qcow2/lvm/other.

>
> I know NetApp and LVM but am rather new to qcow2, so i am not sure if
> i'm suggesting anything remotely sane here, so forgive me that's the
> case.
>
> Greetz,
>
> Fred
>
>
>
> On Sun, May 15, 2011 at 7:16 PM, Jagane Sundar<jagane@sundar.org>  wrote:
>> Hello Fred,
>>
>> As Stefan points out, there is a lot of interest in such functionality.
>>
>> I am the author of a feature called Livebackup - which is aimed at
>> solving exactly this problem - backup of live virtual machines. The
>> additional constraint I place is that backup of the VM should be
>> enabled regardless of the type of storage in use - file on local disk,
>> file on NAS, LVM logical volume on local disk, LVM volume on an
>> iSCSI volume.
>>
>> More information is at:
>> http://wiki.qemu.org/Features/Livebackup
>>
>> I investigated the approach that you outline below - LVM snapshot
>> followed by transfer of the blocks out of the VM host. The problem
>> I ran into was that there was no easy way to identify the blocks that
>> were modified since the last backup was taken. To the best of my
>> knowledge, rsync uses checksums of entire files to determine whether
>> a file was modified or not - in our use case the whole virtual disk
>> is one single file, do rsync will not work well.
>>
>> One of the 'features' of Livebackup is that I chose to invent my own
>> rudimentary protocol for transferring the blocks out of the VM
>> host. iSCSI seems like a standard protocol that would suit the
>> purpose; however it does not fit well with the architecture of
>> Livebackup - I would probably need to implement an iSCSI target
>> function inside Livebackup.
>>
>> Thanks,
>> Jagane
>>
>> On 5/15/2011 7:11 AM, Stefan Hajnoczi wrote:
>>>
>>> On Sun, May 15, 2011 at 1:16 PM, Fred van Zwieten<fvzwieten@gmail.com>
>>>   wrote:
>>>>
>>>> Background:
>>>> NetApp Snashot functionality gives you application consistent
>>>> snapshots of data. Just inform the app a snapshot is about to be made,
>>>> depending on the app, it needs to go in to some sort of backup mode,
>>>> of just stop and flush outstanding I/O. Next, a snapshot is made and
>>>> everything just runs on. Because of the underlying WAFL filesystem
>>>> design, the snapshot always points to the blocks at the time of the
>>>> creation without needing to do any COW.
>>>>
>>>> Layered on top op this functionality is SnapMirror, where the delta
>>>> between this snapshot and a previous snapshot (both being static in
>>>> nature), is asynchronously replicated to a second system. There, this
>>>> delta is applied to a copy of the disk as a local snapshot.
>>>>
>>>> This setup gives you application consistent data disks on a remote
>>>> location as a form of disaster tolerancy. The RPO is the
>>>> snapshot/snapmirror frequency.
>>>>
>>>> KVM:
>>>> My question is rather simple. Could something like this be implemented
>>>> with kvm-img and rsync and/or lvm? I've played with the backing_file
>>>> option, but this means I have to shutdown the vm a boot is on the new
>>>> file to let this work.
>>>
>>> This recent thread might interest you:
>>> http://lists.gnu.org/archive/html/qemu-devel/2011-05/msg00733.html
>>>
>>> Basically you have to cobble it together yourself at the moment but
>>> there is activity around live snapshots and dirty block tracking, so
>>> hopefully they will be available as APIs in the future.
>>>
>>> Stefan
>>
>>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html