Re: [Qemu-devel] [PATCH v4 RFC] spec: add qcow2-dirty-bitmaps specification

[Max Reitz <mreitz@redhat.com>]

On 2015-12-15 at 10:58, Kevin Wolf wrote:
> Am 14.12.2015 um 21:05 hat Max Reitz geschrieben:
>> On 14.12.2015 18:43, Vladimir Sementsov-Ogievskiy wrote:
>>> The new feature for qcow2: storing dirty bitmaps.
>>>
>>> Only dirty bitmaps relative to this qcow2 image should be stored in it.
>>>
>>> Strings started from +# are RFC-strings, not to be commited of course.
>>>
>>>
>>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>
> First of all, I think we may need some improvemnts on details and wording
> here and there, but the format in general looks quite reasonable to me
> (at the first sight at least).
>

[...]

>> Not so good: While now any program knows how to read the bitmap and that
>> it does refer to this qcow2 file, it's interpretation is not so easy

*its

>> still. Generally, a dirty bitmap has some reference point, that is the
>> state of the disk when the bitmap was cleared or created. For instance,
>> for incremental backups, whenever you create a backup based on a dirty
>> bitmap, the dirty bitmap is cleared and the backup target is then said
>> reference point.
>> I think it would be nice to put that reference point (i.e. the name of
>> an image file that contains the clean image) into the dirty bitmap
>> header, if possible.
>
> I don't think it's a valid assumption that the reference point has a
> file name.

Right, but you can't really do anything useful with the dirty bitmap if 
you don't know that reference point.

I do realize that it may not always possible and often very cumbersome 
to come up with some filename to put here, but I still consider it very 
valuable information if available.

John suggested there to be a way for the management layer to put in this 
information if available. That may save qemu from the trouble of having 
to figure it out.

Unless you/we can think of a better way of describing the reference 
point (note: we don't necessarily have an existing reference point at 
all, you can just clear the dirty bitmap at your whim), I'd be 
completely fine with just adding a reference point filename field that 
may be empty if that information is unknown, and in practice just always 
leave it empty for now.

>            Which makes me wonder... How do dirty bitmaps and internal
> snapshots play together?
>
> What I guess could be done is storing a creation date if you think this
> would be useful.

Only in that you could look through the local filesystem if perchance 
you find a disk image with the same size as the qcow2 image in question 
and roughly the right creation date...

[...]

>>> +             19:    granularity_bits
>>> +                    Granularity bits. Valid values are: 0 - 31.
>>> +# Now, qemu allows creating bitmaps with granularity as a 32bit value. And
>>> +# there are no reasons of increasing it.
>>
>> Good (implicit) question. I can't imagine any reason for wanting to have
>> a coarser granularity than 2 GB, but I do think there may be a need in
>> the future for some people.
>>
>> Once again, I think we should discriminate between what is generally a
>> useful limitation and what is simply due to qemu not supporting anything
>> else right now.
>>
>> Thus, I think it would be better to increase the range to 0 - 63 and
>> make a note that qemu only supports values up to 31 right now.
>
> Why 63 and not 255 (the maximum for this one-byte field)? As soon as we
> pick an arbitrary value, we can just take whatever qemu actually
> supports. We can always increase the limit later.

Because I'm willing to take a bet that we won't need granularities 
greater than 2^63 even in twenty years of now.

Also, qcow2 simply doesn't support images with more than ~2^64 clusters.

[...]

>>> +
>>> +        variable:   The name of the bitmap (not null terminated). Should be
>>> +                    unique among all dirty bitmap names within the Dirty
>>> +                    bitmaps extension.
>>> +
>>> +        variable:   Padding to round up the Dirty Bitmap Directory Entry size
>>> +                    to the next multiple of 8.
>>
>> What I'd like here is variable additional information based on the
>> bitmap type. For some types, this may be absolutely necessary; for dirty
>> tracking bitmaps it depends on what we do about the reference point thing.
>
> This could turn out useful, yes.
>
> I also think some extensibility mechanism would be good, like extra_data
> in internal snapshots.
>
>> The reference point thing is the following: As mentioned at the top, I'd
>> like there to be some kind of description of what the clean state was.
>> As far as I know, this is generally a backup in the form of a file. In
>> that case, we could put that filename here.
>
> Who says that it's a local file? Or that it hasn't been moved since we
> created the bitmap?

The former: Nobody, but if it is, it's valuable information. (I won't 
suggest json: filenames, because that's not very useful information to 
any program outside of qemu.)

The latter: Good point. But the same applies to backing filenames (and 
for the case of incremental backups, which is probably mainly what we 
want to use this feature for, the reference point filename just is the 
backing filename of the backup to be created).

Max