Re: [Qemu-devel] [PATCH 1/8] spec: add qcow2-dirty-bitmaps specification

[Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>]

On 10.06.2015 18:34, Kevin Wolf wrote:
> Am 08.06.2015 um 17:21 hat Vladimir Sementsov-Ogievskiy geschrieben:
>> From: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
>>
>> Persistent dirty bitmaps will be saved into qcow2 files. It may be used
>> as 'internal' bitmaps (for qcow2 drives) or as 'external' bitmaps for
>> other drives (there may be qcow2 file with zero disk size but with
>> several dirty bitmaps for other drives).
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>>   docs/specs/qcow2.txt | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 66 insertions(+)
>>
>> diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
>> index 121dfc8..0fffba2 100644
>> --- a/docs/specs/qcow2.txt
>> +++ b/docs/specs/qcow2.txt
>> @@ -123,6 +123,7 @@ be stored. Each extension has a structure like the following:
>>                           0x00000000 - End of the header extension area
>>                           0xE2792ACA - Backing file format name
>>                           0x6803f857 - Feature name table
>> +                        0x23852875 - Dirty bitmaps
>>                           other      - Unknown header extension, can be safely
>>                                        ignored
>>   
>> @@ -166,6 +167,19 @@ the header extension data. Each entry look like this:
>>                       terminated if it has full length)
>>   
>>   
>> +== Dirty bitmaps ==
>> +
>> +Dirty bitmaps is an optional header extension. It provides a possibility of
>> +storing dirty bitmaps in qcow2 image. The fields are:
>> +
>> +          0 -  3:  nb_dirty_bitmaps
>> +                   Number of dirty bitmaps contained in the image
>> +
>> +          4 - 11:  dirty_bitmaps_offset
>> +                   Offset into the image file at which the dirty bitmaps table
>> +                   starts. Must be aligned to a cluster boundary.
>> +
>> +
>>   == Host cluster management ==
> You need to use a compatibility flag because for old qemu versions, the
> dirty bitmaps (and associated metadata) are leaked clusters and qemu-img
> check would "repair" them by resetting the refcount to 0.
>
> At second sight, I see that your patches add an autoclear flag.
> Presumably the contents of the dirty bitmaps is outdated when you
> accessed the image with an older version, so this seems right. We just
> need to document it.
>
>>   qcow2 manages the allocation of host clusters by maintaining a reference count
>> @@ -360,3 +374,55 @@ Snapshot table entry:
>>   
>>           variable:   Padding to round up the snapshot table entry size to the
>>                       next multiple of 8.
>> +
>> +
>> +== Dirty bitmaps ==
>> +
>> +The feature supports storing several dirty bitmaps in the qcow2 file.
>> +
>> +=== Cluster mapping ===
>> +
>> +Dirty bitmaps are stored using a ONE-level structure for the mapping of
>> +bitmaps to host clusters. There is only an L1 table.
>> +
>> +The L1 table has a variable size (stored in the Bitmap table entry) and may
>> +use multiple clusters, however it must be contiguous in the image file.
>> +
>> +Given an offset into the bitmap, the offset into the image file can be
>> +obtained as follows:
>> +
>> +    offset = l1_table[offset / cluster_size] + (offset % cluster_size)
>> +
>> +L1 table entry:
>> +
>> +    Bit  0 -  61:   Standard cluster descriptor
>> +
>> +        62 -  63:   Reserved
> Stefan already mentioned that we don't have a "L1" when there is only
> one level, and that you shouldn't reuse the cluster descriptors from L2
> tables.
>
>> +=== Bitmap table ===
>> +
>> +A directory of all bitmaps is stored in the bitmap table, a contiguous area in
>> +the image file, whose starting offset and length are given by the header fields
>> +dirty_bitmaps_offset and nb_dirty_bitmaps. The entries of the bitmap table have
>> +variable length, depending on the length of name and extra data.
>> +
>> +Bitmap table entry:
>> +
>> +    Byte 0 -  7:    Offset into the image file at which the L1 table for the
>> +                    bitmap starts. Must be aligned to a cluster boundary.
>> +
>> +         8 - 11:    Number of entries in the L1 table of the bitmap
> Worth using 64 bits here? This can only cover 4 * 512 GB = 2 TB for the
> smallest possible cluster size. Though it's 65536 * 512 = 32 PB for the
> default, which might be enough for a while.

We store the bitmap in RAM.. I think 2 TB bitmap should not appear, 
larger granularity should be used for big disks.

>
>> +        12 - 15:    Bitmap granularity in bytes
>> +
>> +        16 - 23:    Bitmap size in sectors
> Please don't use sectors, that's a meaningless unit. Bytes is better.
>
>> +        24 - 25:    Size of the bitmap name
> We should use a smaller limit than the possible 64k to avoid too large
> memory allocations. Nobody needs really long bitmap names.
>
>> +
>> +        variable:   The name of the bitmap (not null terminated)
>> +
>> +        variable:   Padding to round up the bitmap table entry size to the
>> +                    next multiple of 8.
> Kevin


-- 
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.