[Qemu-devel] 2018-07-02 Incremental Backup Status Report

[Qemu-devel] 2018-07-02 Incremental Backup Status Report

[John Snow]

I'm tracking the following series for Incremental Backup right now; if
it's not on this list, I don't know about it:

[Qemu-devel] [PATCH v4 0/2] block: formalize and test fleecing​
[Qemu-devel] [PATCH v2 0/3] image fleecing
[Qemu-devel] [PATCH] migration: Appease coverity, skip empty block trees
[Qemu-devel] [PATCH v2 0/2] test NBD bitmap export
[Qemu-devel] [PATCH 0/6] fix persistent bitmaps migration logic​
[Qemu-devel] [PATCH 0/2] dirty bitmap fixes​
[Qemu-devel] [PATCH v2] qcow2: add overlap check for bitmap directory
[Qemu-devel] [PATCH 0/1] blockdev: implement n-ary bitmap merge
[Qemu-devel] [PATCH v2 00/10] qemu-img: add bitmap info output​


Now, in a little more detail below;


# Image fleecing:

In order to support pull-model semantics for incremental backup, we need
working point in time NBD export of live disk images. This was thought
supported/finished for quite some time, but we've had no iotests.

There are two series in flight here:

- [Qemu-devel] [PATCH v4 0/2] block: formalize and test fleecing​
- [Qemu-devel] [PATCH v2 0/3] image fleecing

The first listed there simply loosens restrictions on blockdev-backup
and adds an iotest that this approach works in the simple case.

The second series adds a fleecing filter to combat deficiencies in the
IO serialization that may allow the fleecing node to sometimes see new,
live data instead of the PIT data. See
https://lists.gnu.org/archive/html/qemu-devel/2018-07/msg00150.html for
details.

Our current plan is to merge the first series and to continue looking
into the fleecing filter during freeze as a potential bugfix.


# NBD:

- [Qemu-devel] [PATCH v2 0/2] test NBD bitmap export

Eric's series that adds a quick hack to our NBD client and an iotest 223
that demos that the bitmap support in our NBD server is working
correctly. Notably, we don't have an iotest that combines this with the
image fleecing workflow, so there's discussion there that I've started
for how to combine the two, comments appreciated:
https://lists.gnu.org/archive/html/qemu-devel/2018-07/msg00476.html


# Migration/Persistence:

- [Qemu-devel] [PATCH 0/6] fix persistent bitmaps migration logic​

Needs review, but can go in after freeze as a fix.


# Bitmap manipulation:

- [Qemu-devel] [PATCH 0/1] blockdev: implement n-ary bitmap merge​

Possibly important to have if we wish to add merge into transactions, as
this will greatly simplify the commands needed to be issued to QEMU. Not
yet reviewed, but I might submit a PR for this tomorrow in anticipation
of its usefulness over the 1-bitmap version.

- [Qemu-devel] [PATCH v2 00/10] qemu-img: add bitmap info output​

On hold until persistence is settled, due to interactions with
attempting to cache the bitmap metadata. "offline" bitmap manipulation
will need to occur via QMP on QEMU with no running guest in the
meantime. We will still need a patch for `qemu-img check -r` that allows
us to delete corrupted bitmaps, however. That can be a bugfix for 3.0.

There is a follow-up series that's also on hold because it's blocked on
this series, that adds the actual add/delete/clear offline functionality.


# Miscellaneous:

- [Qemu-devel] [PATCH] migration: Appease coverity, skip empty block trees

Needs a respin based on Stefan's feedback. Only important so that it
doesn't keep annoying Peter Maydell, does not actually fix a bug.

- [Qemu-devel] [PATCH 0/2] dirty bitmap fixes​

Staged: https://github.com/jnsnow/qemu/commits/bitmaps

- [Qemu-devel] [PATCH v2] qcow2: add overlap check for bitmap directory

Needs a V3 after review from Max. Not crucial for release, but nice to have.




So in order to put together a demo from QEMU's point-of-view that both
push and pull model are working, I think we still need:

- iotest 222 and 223 merged, with their respective enablement patches.
- A new iotest (224?) that combines functionality of both; which makes
use of perhaps some new features that allow us to tie bitmap
manipulation to the start of the node fleecing.
- All of the various bugfixes above reviewed and merged.



Please let me know if this roadmap is missing anything.

Thanks,
--js

[Qemu-devel] [Qemu-block] 2018-07-09 Incremental Backup Status Report

[John Snow]

On 07/02/2018 07:47 PM, John Snow wrote:
> I'm tracking the following series for Incremental Backup right now; if
> it's not on this list, I don't know about it:
> 
> [Qemu-devel] [PATCH v4 0/2] block: formalize and test fleecing​

Merged.

> [Qemu-devel] [PATCH v2 0/3] image fleecing

Dropped in favor of the above, but we've then added:

[Qemu-devel] [PATCH v5 0/4] fix image fleecing

Which should have the same effect as preventing race conditions between
serializations that exist across two nodes.

> [Qemu-devel] [PATCH] migration: Appease coverity, skip empty block trees

I need to spin a v2.

> [Qemu-devel] [PATCH v2 0/2] test NBD bitmap export

Merged.

> [Qemu-devel] [PATCH 0/6] fix persistent bitmaps migration logic​

Needs review.

> [Qemu-devel] [PATCH 0/2] dirty bitmap fixes​

Merged.

> [Qemu-devel] [PATCH v2] qcow2: add overlap check for bitmap directory

Merged.

> [Qemu-devel] [PATCH 0/1] blockdev: implement n-ary bitmap merge

Needs review; it's not essential but I suspect it's likely nice to have
for libvirt.

> [Qemu-devel] [PATCH v2 00/10] qemu-img: add bitmap info output​
> 

On hold pending "fix persistent bitmaps migration logic."


New series:
- [PATCH v2 0/4] transaction support for bitmap merge



New status list looks like this, then:

[Qemu-devel] [PATCH] migration: Appease coverity, skip empty block trees
[Qemu-devel] [PATCH v5 0/4] fix image fleecing
[Qemu-devel] [PATCH 0/6] fix persistent bitmaps migration logic​
[Qemu-devel] [PATCH 0/1] blockdev: implement n-ary bitmap merge
[Qemu-devel] [PATCH v2 00/10] qemu-img: add bitmap info output​
[Qemu-devel] [PATCH v2 0/4] transaction support for bitmap merge

I'll be looking at #3, #6 and #1 tomorrow.

Thanks,
--js


> 
> Now, in a little more detail below;
> 
> 
> # Image fleecing:
> 
> In order to support pull-model semantics for incremental backup, we need
> working point in time NBD export of live disk images. This was thought
> supported/finished for quite some time, but we've had no iotests.
> 
> There are two series in flight here:
> 
> - [Qemu-devel] [PATCH v4 0/2] block: formalize and test fleecing​
> - [Qemu-devel] [PATCH v2 0/3] image fleecing
> 
> The first listed there simply loosens restrictions on blockdev-backup
> and adds an iotest that this approach works in the simple case.
> 
> The second series adds a fleecing filter to combat deficiencies in the
> IO serialization that may allow the fleecing node to sometimes see new,
> live data instead of the PIT data. See
> https://lists.gnu.org/archive/html/qemu-devel/2018-07/msg00150.html for
> details.
> 
> Our current plan is to merge the first series and to continue looking
> into the fleecing filter during freeze as a potential bugfix.
> 
> 
> # NBD:
> 
> - [Qemu-devel] [PATCH v2 0/2] test NBD bitmap export
> 
> Eric's series that adds a quick hack to our NBD client and an iotest 223
> that demos that the bitmap support in our NBD server is working
> correctly. Notably, we don't have an iotest that combines this with the
> image fleecing workflow, so there's discussion there that I've started
> for how to combine the two, comments appreciated:
> https://lists.gnu.org/archive/html/qemu-devel/2018-07/msg00476.html
> 
> 
> # Migration/Persistence:
> 
> - [Qemu-devel] [PATCH 0/6] fix persistent bitmaps migration logic​
> 
> Needs review, but can go in after freeze as a fix.
> 
> 
> # Bitmap manipulation:
> 
> - [Qemu-devel] [PATCH 0/1] blockdev: implement n-ary bitmap merge​
> 
> Possibly important to have if we wish to add merge into transactions, as
> this will greatly simplify the commands needed to be issued to QEMU. Not
> yet reviewed, but I might submit a PR for this tomorrow in anticipation
> of its usefulness over the 1-bitmap version.
> 
> - [Qemu-devel] [PATCH v2 00/10] qemu-img: add bitmap info output​
> 
> On hold until persistence is settled, due to interactions with
> attempting to cache the bitmap metadata. "offline" bitmap manipulation
> will need to occur via QMP on QEMU with no running guest in the
> meantime. We will still need a patch for `qemu-img check -r` that allows
> us to delete corrupted bitmaps, however. That can be a bugfix for 3.0.
> 
> There is a follow-up series that's also on hold because it's blocked on
> this series, that adds the actual add/delete/clear offline functionality.
> 
> 
> # Miscellaneous:
> 
> - [Qemu-devel] [PATCH] migration: Appease coverity, skip empty block trees
> 
> Needs a respin based on Stefan's feedback. Only important so that it
> doesn't keep annoying Peter Maydell, does not actually fix a bug.
> 
> - [Qemu-devel] [PATCH 0/2] dirty bitmap fixes​
> 
> Staged: https://github.com/jnsnow/qemu/commits/bitmaps
> 
> - [Qemu-devel] [PATCH v2] qcow2: add overlap check for bitmap directory
> 
> Needs a V3 after review from Max. Not crucial for release, but nice to have.
> 
> 
> 
> 
> So in order to put together a demo from QEMU's point-of-view that both
> push and pull model are working, I think we still need:
> 
> - iotest 222 and 223 merged, with their respective enablement patches.
> - A new iotest (224?) that combines functionality of both; which makes
> use of perhaps some new features that allow us to tie bitmap
> manipulation to the start of the node fleecing.
> - All of the various bugfixes above reviewed and merged.
> 
> 
> 
> Please let me know if this roadmap is missing anything.
> 
> Thanks,
> --js
>