Re: [Qemu-devel] [PATCH v4 RFC] spec: add qcow2-dirty-bitmaps specification

[Max Reitz <mreitz@redhat.com>]

On 2015-12-14 at 23:26, John Snow wrote:
> On 12/14/2015 04:44 PM, Max Reitz wrote:
>> On 14.12.2015 21:45, John Snow wrote:
>>> On 12/14/2015 03:05 PM, Max Reitz wrote:
>>>> On 14.12.2015 18:43, Vladimir Sementsov-Ogievskiy wrote:
>>>>> The new feature for qcow2: storing dirty bitmaps.
>>>>>
>>>>> Only dirty bitmaps relative to this qcow2 image should be stored in it.
>>>>>
>>>>> Strings started from +# are RFC-strings, not to be commited of course.
>>>>>
>>>>>
>>>>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>>>> ---
>>>>>
>>>>>   docs/specs/qcow2.txt | 151 ++++++++++++++++++++++++++++++++++++++++++++++++++-
>>>>>   1 file changed, 150 insertions(+), 1 deletion(-)

[...]

> If the auto bit is on, it's a must.
>
> Let's try again:
>
> "1: auto
>      This bitmap is of the Dirty Tracking Type and must accurately
>      reflect all writes to the virtual disk image by any application
>      that would write to it. This bitmap should not lose the auto bit
>      except by user intervention."
>
> Or something like that? The idea is that it will indeed be "autoloaded"
> and made active, but some bitmaps might be stored in various
> frozen/unactive states where this is not desirable.

Sounds good to me.

[...]

>> (If you've already decided to put the backup progress into the target
>> image, you may find that offtopic block not very helpful; however, it
>> may help you get an idea of when I consider some information to be
>> beneficial to a qcow2 file.)
>>
>
> <Offtopic>
> There's no code written specifically for this yet, so we're still OK. It
> could reasonably go into either the source or destination:
>
> Destination: "Here are the sectors I still need" is reasonable, as then
> you can easily go and fetch the missing bits you need.
>
> However, if the source was modified after the bitmap was written to the
> destination, we have no way of knowing if anything has been updated
> since we started the backup.
>
> If we store it in the source: "Here's what's changed [and what we never
> copied] since we started that backup," we can even make offline writes
> to the source after we stop the backup, then resume it later and still
> have it work out.
>
> In the destination model, we point to the source for where to get the
> rest of the data. In the source model, we point to the destination of
> our unfinished backup.
>
> One definitely feels more independently useful: "Here's how to get the
> rest of my data" vs "I was being used for an operation you may or may
> not care about.", though the less meaningful one might actually be
> easier and more flexible to maintain from QEMU's standpoint.

...OK, let's worry about that later. :-)

> Offtopic indeed.
>
> </Offtopic>
>
> The only real point is that there *might* be different semantics in the
> future, but we may also be able to get away with overloading the
> existing type entirely.
>
> The reserved type enum will work out OK for this.

Yep.

[...]

>>> I'm not opposed to listing some "nice" information when available.
>>>
>>> last_backup /path/to/incremental.5.qcow2
>>> base_backup /path/to/reference.qcow2
>>>
>>
>> I don't think we need the base_backup since you can get that by walking
>> through the backing chain of the reference point (the backup target),
>> but it probably won't hurt if you can make a good general semantic
>> connection to the dirty bitmap.
>>
>> Max
>>
>
> Yes, probably not. How would you suggest this be implemented, also? Does
> the bitmap object need to begin tracking a last-known backup
> filename/identifier? I remembered you didn't seem too happy about
> keeping filenames in memory for QEMU, but we currently don't track this
> information at all. Answer has so far been "It's up to management!"

Maybe we can keep it up to management. I think you suggested that we may 
give the management layer a way of writing the reference point 
information into the dirty bitmap information. That would work, then.

Other than that... Well, whenever we have a transaction which clears a 
dirty bitmap, we'd have to analyse the other operations within that 
transaction, and if there's some blockdev-backup or drive-backup job 
there, we'd have to take the target image's filename and keep it in the 
bitmap structure, probably, yes. Ugly indeed.

The question is whether management tools are actually keen on 
implementing this feature for us... We'll see.

Max