[Qemu-devel] [question] incremental backup a running vm

[Qemu-devel] [question] incremental backup a running vm

[Zhang Haoyu]

Hi,

Does drive_mirror support incremental backup a running vm?
Or other mechanism does?

incremental backup a running vm requirements:
First time backup, all of the allocated data will be mirrored to destination,
then a copied bitmap will be saved to a file, then the bitmap file will log dirty for
the changed data.
Next time backup, only the dirty data will be mirrored to destination.
Even the VM shutdown and start after several days,
the bitmap will be loaded while starting vm.
Any ideas?

Thanks,
Zhang Haoyu

Re: [Qemu-devel] [question] incremental backup a running vm

[Paolo Bonzini]

On 21/01/2015 11:32, Zhang Haoyu wrote:
> Hi,
> 
> Does drive_mirror support incremental backup a running vm?
> Or other mechanism does?
> 
> incremental backup a running vm requirements:
> First time backup, all of the allocated data will be mirrored to destination,
> then a copied bitmap will be saved to a file, then the bitmap file will log dirty for
> the changed data.
> Next time backup, only the dirty data will be mirrored to destination.
> Even the VM shutdown and start after several days,
> the bitmap will be loaded while starting vm.
> Any ideas?

Drive-mirror is for storage migration.  For backup there is another job,
drive-backup.  drive-backup copies a point-in-time snapshot of one or
more disks corresponding to when the backup was started.

Incremental backup is being worked on.  You can see patches on the list.

Paolo

Re: [Qemu-devel] [question] incremental backup a running vm

[Kashyap Chamarthy]

On Wed, Jan 21, 2015 at 11:39:44AM +0100, Paolo Bonzini wrote:
> 
> 
> On 21/01/2015 11:32, Zhang Haoyu wrote:
> > Hi,
> > 
> > Does drive_mirror support incremental backup a running vm?
> > Or other mechanism does?
> > 
> > incremental backup a running vm requirements:
> > First time backup, all of the allocated data will be mirrored to destination,
> > then a copied bitmap will be saved to a file, then the bitmap file will log dirty for
> > the changed data.
> > Next time backup, only the dirty data will be mirrored to destination.
> > Even the VM shutdown and start after several days,
> > the bitmap will be loaded while starting vm.
> > Any ideas?
> 
> Drive-mirror is for storage migration.  For backup there is another job,
> drive-backup.  drive-backup copies a point-in-time snapshot of one or
> more disks corresponding to when the backup was started.

Zhang, I've been testing the `drive-backup` command via QMP for a little
while, and it works reasonably well. If you'd like to test it you can
quickly try as below, once you have a QEMU instance runing with QMP
(I invoke my QEMU instances with '-qmp tcp:localhost:4444,server').

The below script invokes the 'drive-backup' QMP command (Ensure you're
using the correct disk, your disk ID might be 'drive-virtio-disk1' :-) )
-----------------
#!/bin/bash
set -x
exec 3<>/dev/tcp/localhost/4444
echo -e "{ 'execute': 'qmp_capabilities' }" >&3
read response <&3
echo $response
echo -e "{ 'execute': 'drive-backup', 'arguments':
      { 'device': 'drive-virtio-disk0', 'sync': 'full', 'target':
      '/export/backup-of-vm1.qcow2', 'mode': 'absolute-paths', 'format': 'qcow2' } }" >&3
read response <&3
echo $response
-----------------

Once the above is invoked succesfully, you can see the success of the
command on the shell where your QMP server is running:
-----------------
[. . .]
char device redirected to /dev/pts/8 (label charserial0)
QEMU waiting for connection on: disconnected:tcp:localhost:4444,server
Formatting '/export/backup-of-vm1.qcow2', fmt=qcow2 size=53687091200 encryption=off cluster_size=65536 lazy_refcounts=off 
-----------------


> Incremental backup is being worked on.  You can see patches on the list.
 

-- 
/kashyap

Re: [Qemu-devel] [question] incremental backup a running vm

[Zhang Haoyu]

On 2015-01-23 07:30:19, Kashyap Chamarthy wrote:
>On Wed, Jan 21, 2015 at 11:39:44AM +0100, Paolo Bonzini wrote:
> > 
> > 
> > On 21/01/2015 11:32, Zhang Haoyu wrote:
> > > Hi,
>> > 
> > > Does drive_mirror support incremental backup a running vm?
> > > Or other mechanism does?
> > > 
>> > incremental backup a running vm requirements:
> > > First time backup, all of the allocated data will be mirrored to destination,
> > > then a copied bitmap will be saved to a file, then the bitmap file will log dirty for
> > > the changed data.
> > > Next time backup, only the dirty data will be mirrored to destination.
>> > Even the VM shutdown and start after several days,
> > > the bitmap will be loaded while starting vm.
> > > Any ideas?
> > 
>> Drive-mirror is for storage migration.  For backup there is another job,
> > drive-backup.  drive-backup copies a point-in-time snapshot of one or
> > more disks corresponding to when the backup was started.
> 
> Zhang, I've been testing the `drive-backup` command via QMP for a little
>while, and it works reasonably well. If you'd like to test it you can
> quickly try as below, once you have a QEMU instance runing with QMP
> (I invoke my QEMU instances with '-qmp tcp:localhost:4444,server').
> 
Hi, Kashyap
I've tried ‘drive_backup’ via QMP,
but the snapshots were missed to backup to destination,
I think the reason is that backup_run() only copy the
guest data regarding qcow2 image.

Thanks,
Zhang Haoyu
>The below script invokes the 'drive-backup' QMP command (Ensure you're
> using the correct disk, your disk ID might be 'drive-virtio-disk1' :-) )
> -----------------
> #!/bin/bash
> set -x
>exec 3<>/dev/tcp/localhost/4444
> echo -e "{ 'execute': 'qmp_capabilities' }" >&3
> read response <&3
> echo $response
>echo -e "{ 'execute': 'drive-backup', 'arguments':
>       { 'device': 'drive-virtio-disk0', 'sync': 'full', 'target':
>       '/export/backup-of-vm1.qcow2', 'mode': 'absolute-paths', 'format': 'qcow2' } }" >&3
> read response <&3
> echo $response
>-----------------
> 
> Once the above is invoked succesfully, you can see the success of the
> command on the shell where your QMP server is running:
>-----------------
> [. . .]
> char device redirected to /dev/pts/8 (label charserial0)
> QEMU waiting for connection on: disconnected:tcp:localhost:4444,server
> Formatting '/export/backup-of-vm1.qcow2', fmt=qcow2 size=53687091200 encryption=off cluster_size=65536 lazy_refcounts=off 
>-----------------
> 
> 
> > Incremental backup is being worked on.  You can see patches on the list.
> 
> 
> -- 
> /kashyap

Re: [Qemu-devel] [question] incremental backup a running vm

[Paolo Bonzini]

On 26/01/2015 02:07, Zhang Haoyu wrote:
> Hi, Kashyap
> I've tried ‘drive_backup’ via QMP,
> but the snapshots were missed to backup to destination,
> I think the reason is that backup_run() only copy the
> guest data regarding qcow2 image.

Yes, that's the case.

QEMU cannot still access internal snapshots while the file is open.
External snapshots are opened read-only, and can be copied with "cp"
while QEMU is running.

Paolo

Re: [Qemu-devel] [question] incremental backup a running vm

[Zhang Haoyu]

On 2015-01-26 17:29:43, Paolo Bonzini wrote:
>
> On 26/01/2015 02:07, Zhang Haoyu wrote:
> > Hi, Kashyap
> > I've tried ‘drive_backup’ via QMP,
>> but the snapshots were missed to backup to destination,
> > I think the reason is that backup_run() only copy the
> > guest data regarding qcow2 image.
> 
>Yes, that's the case.
> 
> QEMU cannot still access internal snapshots while the file is open.
> External snapshots are opened read-only, and can be copied with "cp"
> while QEMU is running.
Thanks, Paolo,
but too many internal snapshots were saved by customers,
switching to external snapshot mechanism has significant impaction
on subsequent upgrade.

Another problem:
drive_backup just implement one time backup,
but I want VMWare's VDP-like backup mechanism.
The initial backup of a virtual machine takes comparatively more time,
because all of the data for that virtual machine is being backed up. 
Subsequent backups of the same virtual machine take less time, because
changed block tracking (log dirty) mechanism is used to only backup the dirty data.
After inittial backup done, even the VM shutdown, but subsequent backup also only 
copy the changed data.

Thanks,
Zhang Haoyu
>
> Paolo

Re: [Qemu-devel] [question] incremental backup a running vm

[Paolo Bonzini]

On 26/01/2015 12:13, Zhang Haoyu wrote:
> Thanks, Paolo,
> but too many internal snapshots were saved by customers,
> switching to external snapshot mechanism has significant impaction
> on subsequent upgrade.

In that case, patches are welcome. :)

> Another problem:
> drive_backup just implement one time backup,
> but I want VMWare's VDP-like backup mechanism.
> The initial backup of a virtual machine takes comparatively more time,
> because all of the data for that virtual machine is being backed up. 
> Subsequent backups of the same virtual machine take less time, because
> changed block tracking (log dirty) mechanism is used to only backup the dirty data.
> After inittial backup done, even the VM shutdown, but subsequent backup also only 
> copy the changed data.

As mentioned before, patches for this are on the list.

Paolo

Re: [Qemu-devel] [question] incremental backup a running vm

[Zhang Haoyu]

On 2015-01-26 19:29:03, Paolo Bonzini wrote:
>
> On 26/01/2015 12:13, Zhang Haoyu wrote:
> > Thanks, Paolo,
> > but too many internal snapshots were saved by customers,
>> switching to external snapshot mechanism has significant impaction
> > on subsequent upgrade.
> 
> In that case, patches are welcome. :)
>
> > Another problem:
> > drive_backup just implement one time backup,
> > but I want VMWare's VDP-like backup mechanism.
> > The initial backup of a virtual machine takes comparatively more time,
>> because all of the data for that virtual machine is being backed up. 
> > Subsequent backups of the same virtual machine take less time, because
> > changed block tracking (log dirty) mechanism is used to only backup the dirty data.
> > After inittial backup done, even the VM shutdown, but subsequent backup also only 
>> copy the changed data.
> 
> As mentioned before, patches for this are on the list.
> 
I see, thanks, Paolo.
> Paolo

Re: [Qemu-devel] [question] incremental backup a running vm

[Kashyap Chamarthy]

On Wed, Jan 21, 2015 at 11:39:44AM +0100, Paolo Bonzini wrote:
> 
> 
> On 21/01/2015 11:32, Zhang Haoyu wrote:
> > Hi,
> > 
> > Does drive_mirror support incremental backup a running vm?
> > Or other mechanism does?
> > 
> > incremental backup a running vm requirements:
> > First time backup, all of the allocated data will be mirrored to destination,
> > then a copied bitmap will be saved to a file, then the bitmap file will log dirty for
> > the changed data.
> > Next time backup, only the dirty data will be mirrored to destination.
> > Even the VM shutdown and start after several days,
> > the bitmap will be loaded while starting vm.
> > Any ideas?
> 
> Drive-mirror is for storage migration.  For backup there is another job,
> drive-backup.  drive-backup copies a point-in-time snapshot of one or
> more disks corresponding to when the backup was started.

Zhang, I've been testing the `drive-backup` command via QMP for a little
while, and it works reasonably well. If you'd like to test it you can
quickly try as below, once you have a QEMU instance runing with QMP
(I invoke my QEMU instances with '-qmp tcp:localhost:4444,server').

The below script invokes the 'drive-backup' QMP command (Ensure you're
using the correct disk, your disk ID might be 'drive-virtio-disk1' :-) )
-----------------
#!/bin/bash                                                                                                                                           
set -x
exec 3<>/dev/tcp/localhost/4444
echo -e "{ 'execute': 'qmp_capabilities' }" >&3
read response <&3
echo $response
echo -e "{ 'execute': 'drive-backup', 'arguments':
      { 'device': 'drive-virtio-disk0', 'sync': 'full', 'target':
      '/export/backup-of-vm1.qcow2', 'mode': 'absolute-paths', 'format': 'qcow2' } }" >&3
read response <&3
echo $response
-----------------

Once the above is invoked succesfully, you can see the success of the
command on the shell where your QMP server is running:
-----------------
[. . .]
char device redirected to /dev/pts/8 (label charserial0)
QEMU waiting for connection on: disconnected:tcp:localhost:4444,server
Formatting '/export/backup-of-vm1.qcow2', fmt=qcow2 size=53687091200 encryption=off cluster_size=65536 lazy_refcounts=off
-----------------

> 
> Incremental backup is being worked on.  You can see patches on the list.
> 
> Paolo
> 

-- 
/kashyap