[Qemu-devel] Dirty block tracking : External file

[Qemu-devel] Dirty block tracking : External file

[Jd]

Hi
     We are looking for dirty block tracking feature in qemu/kvm for 
implementing more efficient incremental backup. I saw some patches 
around it, but from the conversation could not figure out if it is 
complete or not. If it is complete, can some one point to the branch / 
label from which I can get it.

     Could of questions around the feature.

      * Does this work with qcow format only ? Or for any format ? Can I 
use it with ceph, gluster, etc,,etc.

      * How does dirty block tracking to file get affected by snapshots 
? One would assume that it would mark all the blocks changed since 
tracking begun, irrespective of snapshots pull / rebase....etc. This way 
one might copy more blocks than necessary, but is accurate. And this 
allows the user to do different snapshot operations irrespective of 
backups which is a big thing from users perspective.

Thanks
/Jd

Re: [Qemu-devel] Dirty block tracking : External file

[Fam Zheng]

On Tue, 10/28 09:52, Jd wrote:
> Hi
>     We are looking for dirty block tracking feature in qemu/kvm for
> implementing more efficient incremental backup. I saw some patches around
> it, but from the conversation could not figure out if it is complete or not.
> If it is complete, can some one point to the branch / label from which I can
> get it.

It is still unfinished work. No code is written yet for persistent dirty block
tracking, although there are some out-of-date patches for live incremental
backup.

> 
>     Could of questions around the feature.
> 
>      * Does this work with qcow format only ? Or for any format ? Can I use
> it with ceph, gluster, etc,,etc.

It should work with any format, and also network based protocols.

> 
>      * How does dirty block tracking to file get affected by snapshots ? One
> would assume that it would mark all the blocks changed since tracking begun,
> irrespective of snapshots pull / rebase....etc. This way one might copy more
> blocks than necessary, but is accurate. And this allows the user to do
> different snapshot operations irrespective of backups which is a big thing
> from users perspective.

I'm not sure I understand your question but I think block jobs are orthogonal
to incremental backup so both of them run independently of each other. Anyway,
creating a snapshot doesn't change the image's user data, why do you assume it
would mark all the blocks dirty?

Fam

Re: [Qemu-devel] Dirty block tracking : External file

[Stefan Hajnoczi]

On Tue, Oct 28, 2014 at 4:52 PM, Jd <jd_jedi@convirture.com> wrote:
> If it is complete, can some one point to the branch / label from which I can
> get it.

Fam's patches are available in the mailing list archive:
https://lists.gnu.org/archive/html/qemu-devel/2014-03/msg02309.html

A lot of the infrastructure has already been merged.

John Snow and I talked about incremental backup last week to get an
overview of what work remaings.

The dirty bitmap needs to be persistent - that was never implemented
although Fam sent patches for in-memory dirty bitmaps.

The best way to kick the tires is to look at Fam's patch series and
especially the final patch.  The final patch is a test case that shows
the workflow that Fam had in mind.

Stefan

Re: [Qemu-devel] Dirty block tracking : External file

[Stefan Hajnoczi]

On Tue, Oct 28, 2014 at 4:52 PM, Jd <jd_jedi@convirture.com> wrote:
> If it is complete, can some one point to the branch / label from which I can
> get it.

Fam's patches are available in the mailing list archive:
https://lists.gnu.org/archive/html/qemu-devel/2014-03/msg02309.html

A lot of the infrastructure has already been merged.

John Snow and I talked about incremental backup last week to get an
overview of what work remaings.

The dirty bitmap needs to be persistent - that was never implemented
although Fam sent patches for in-memory dirty bitmaps.

The best way to kick the tires is to look at Fam's patch series and
especially the final patch.  The final patch is a test case that shows
the workflow that Fam had in mind.

Stefan

Re: [Qemu-devel] Dirty block tracking : External file

[Stefan Hajnoczi]

On Tue, Oct 28, 2014 at 4:52 PM, Jd <jd_jedi@convirture.com> wrote:
> If it is complete, can some one point to the branch / label from which I can
> get it.

Fam's patches are available in the mailing list archive:
https://lists.gnu.org/archive/html/qemu-devel/2014-03/msg02309.html

A lot of the infrastructure has already been merged.

John Snow and I talked about incremental backup last week to get an
overview of what work remaings.

The dirty bitmap needs to be persistent - that was never implemented
although Fam sent patches for in-memory dirty bitmaps.

The best way to kick the tires is to look at Fam's patch series and
especially the final patch.  The final patch is a test case that shows
the workflow that Fam had in mind.

Stefan