Livebackup feature for qemu/qemu-kvm

Livebackup feature for qemu/qemu-kvm

[Jagane Sundar]

Hello All,

I have been working on a feature called Livebackup for qemu-kvm 
(actually it is applicable to qemu). This is a feature that enables a 
system administrator or a backup program to take full and incremental 
backups of running VMs. It is explained in more detail here:

http://wiki.qemu.org/Features/Livebackup

I had initiated communication with Stefan a week or so ago regarding 
this feature and we had some discussion comparing it with two other 
proposals - Snapshot and Snapshot2.

I have further updated my documentation, and I have been testing my code 
more intensively over the last week or so. I have two git source trees 
based on qemu and qemu-kvm for this codebase. I will continue to 
maintain both till it is necessary.

At this time I would like to restart the discussion around Livebackup 
and related features.

Having a live backup feature for kvm will make it a better solution for 
IaaS clouds compared to xen. I would like to solicit feedback from all 
of you folks involved in the block subsystem of qemu. Stefan mentioned 
that Jes is the person most intimately involved in the block subsystem, 
so Jes - your feedback is particularly important.

Thanks,
Jagane

Re: Livebackup feature for qemu/qemu-kvm

[Jes Sorensen]

On 05/02/11 00:23, Jagane Sundar wrote:
> Having a live backup feature for kvm will make it a better solution for
> IaaS clouds compared to xen. I would like to solicit feedback from all
> of you folks involved in the block subsystem of qemu. Stefan mentioned
> that Jes is the person most intimately involved in the block subsystem,
> so Jes - your feedback is particularly important.

Jagane,

Reading your page, the first thing I stumble upon under 'Use Cases' is
the reference to EBS storage. What is that?

Under details, I think it is not a good idea to rely on QEMU looking for
any files with specific file name suffixes. It really should be
specified on the command line by the user or admin tool.

Other questions, how do you plan to handle crashes or loss of network
connection in the middle of a livebackup? How about handling corrupted
livebackup files?

In general it looks interesting, you could consider submitting a
presentation about Livebackup to the KVM Forum 2011.

Cheers,
Jes

Re: Livebackup feature for qemu/qemu-kvm

[Jagane Sundar]

On 5/2/2011 7:36 AM, Jes Sorensen wrote:
> On 05/02/11 00:23, Jagane Sundar wrote:
>> Having a live backup feature for kvm will make it a better solution for
>> IaaS clouds compared to xen. I would like to solicit feedback from all
>> of you folks involved in the block subsystem of qemu. Stefan mentioned
>> that Jes is the person most intimately involved in the block subsystem,
>> so Jes - your feedback is particularly important.
> Jagane,
>
> Reading your page, the first thing I stumble upon under 'Use Cases' is
> the reference to EBS storage. What is that?
EBS stands for Elastic Block Storage - Amazon EC2's shared storage 
solution. This is the storage
that comes with guarantees, since it is replicated across machines.
> Under details, I think it is not a good idea to rely on QEMU looking for
> any files with specific file name suffixes. It really should be
> specified on the command line by the user or admin tool.
That's a good idea. Perhaps another attribute in the drive description list,
just like type=virtio, maybe backup=livebackup.
> Other questions, how do you plan to handle crashes or loss of network
> connection in the middle of a livebackup? How about handling corrupted
> livebackup files?
Crashes of various software:
1. qemu crashes during normal operation of the VM:
     When this happens, the livebackup_client is forced to do a full 
backup the next
     time around. Here's how: livebackup writes out the in-memory dirty 
bitmap
     to a dirty bitmap file only at the time of orderly shutdown of 
qemu. Hence,
     the mtime of the virtual disk file is later than the mtime  of the 
livebackup
     dirty bitmap file. This causes livebackup to consider the dirty 
bitmap invalid,
     and forces the livebackup_client to do a full backup next time around.
2. qemu crashes while livebackup is in progress:
     In this case also, the livebackup_client is forced to do a full 
backup the next
     time around. The dirty bitmap file, the COW file used to store 
blocks written
     while a livebackup is in progress, are all deleted, and the 
livebackup client
    is forced to do a full backup next time around.
3. livebackup_client crashes while livebackup is in progress:
     In this case, a new livebackup_client may be started, and it can 
redo the last
     type of backup it was doing - an incremental backup or a full 
backup. Note
     all the blocks of the last backup type need to be transferred over 
again, the
     qemu livebackup code does not keep track of what block the client 
was at.
     It does not need to be a forced full backup.
> In general it looks interesting, you could consider submitting a
> presentation about Livebackup to the KVM Forum 2011.

Glad to know that you think it is interesting. Also, thanks for the 
pointer to
KVM Forum 2011, Jes. It looks like I have a few more weeks to get an 
abstract
in for KVM forum 2011. I will do so.

Jagane

Re: Livebackup feature for qemu/qemu-kvm

[Jes Sorensen]

On 05/03/11 07:58, Jagane Sundar wrote:
> On 5/2/2011 7:36 AM, Jes Sorensen wrote:
>> Reading your page, the first thing I stumble upon under 'Use Cases' is
>> the reference to EBS storage. What is that?
> EBS stands for Elastic Block Storage - Amazon EC2's shared storage 
> solution. This is the storage that comes with guarantees, since it is
> replicated across machines.

I see, it would be good if you made that more explicit in the document
for those who aren't as experienced with EC2, like me.

>> Under details, I think it is not a good idea to rely on QEMU
>> looking for any files with specific file name suffixes. It really
>> should be specified on the command line by the user or admin tool.
> That's a good idea. Perhaps another attribute in the drive description
> list,
> just like type=virtio, maybe backup=livebackup.

I think that is the right way to go.

>> In general it looks interesting, you could consider submitting a
>> presentation about Livebackup to the KVM Forum 2011.
> 
> Glad to know that you think it is interesting. Also, thanks for the 
> pointer to KVM Forum 2011, Jes. It looks like I have a few more weeks
> to get an abstract in for KVM forum 2011. I will do so.

Excellent, please go ahead and submit it!

Cheers,
Jes

Re: Livebackup feature for qemu/qemu-kvm

[Jagane Sundar]

On 5/2/2011 11:59 PM, Jes Sorensen wrote:
> On 05/03/11 07:58, Jagane Sundar wrote:
>> On 5/2/2011 7:36 AM, Jes Sorensen wrote:
>>> Reading your page, the first thing I stumble upon under 'Use Cases' is
>>> the reference to EBS storage. What is that?
>> EBS stands for Elastic Block Storage - Amazon EC2's shared storage
>> solution. This is the storage that comes with guarantees, since it is
>> replicated across machines.
> I see, it would be good if you made that more explicit in the document
> for those who aren't as experienced with EC2, like me.
>
Done. I also added a failure scenarios section. I will ping you once I 
get the synopsis in for kvm forum 2011.

Thanks,
Jagane