Re: [Qemu-devel] [PATCH 03/17] spec: add qcow2-dirty-bitmaps specification

[Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>]

On 05.09.2015 19:43, Vladimir Sementsov-Ogievskiy wrote:
> Persistent dirty bitmaps will be saved into qcow2 files. It may be used
> as 'internal' bitmaps (for qcow2 drives) or as 'external' bitmaps for
> other drives (there may be qcow2 file with zero disk size but with
> several dirty bitmaps for other drives).
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
>   docs/specs/qcow2.txt | 127 ++++++++++++++++++++++++++++++++++++++++++++++++++-
>   1 file changed, 126 insertions(+), 1 deletion(-)
>
> diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt
> index 121dfc8..5fc0365 100644
> --- a/docs/specs/qcow2.txt
> +++ b/docs/specs/qcow2.txt
> @@ -103,7 +103,13 @@ in the description of a field.
>                       write to an image with unknown auto-clear features if it
>                       clears the respective bits from this field first.
>   
> -                    Bits 0-63:  Reserved (set to 0)
> +                    Bit 0:      Dirty bitmaps bit. If this bit is set then
> +                                there is a _consistent_ Dirty bitmaps extension
> +                                in the image. If it is not set, but there is a
> +                                Dirty bitmaps extension, its data should be
> +                                considered as inconsistent.
> +
> +                    Bits 1-63:  Reserved (set to 0)
>   
>            96 -  99:  refcount_order
>                       Describes the width of a reference count block entry (width
> @@ -123,6 +129,7 @@ be stored. Each extension has a structure like the following:
>                           0x00000000 - End of the header extension area
>                           0xE2792ACA - Backing file format name
>                           0x6803f857 - Feature name table
> +                        0x23852875 - Dirty bitmaps
>                           other      - Unknown header extension, can be safely
>                                        ignored
>   
> @@ -166,6 +173,24 @@ the header extension data. Each entry look like this:
>                       terminated if it has full length)
>   
>   
> +== Dirty bitmaps ==
> +
> +Dirty bitmaps is an optional header extension. It provides an ability to store
> +dirty bitmaps in a qcow2 image. The fields are:
> +
> +          0 -  3:  nb_dirty_bitmaps
> +                   The number of dirty bitmaps contained in the image. Valid
> +                   values: 0 - 65535.
> +
> +          4 -  7:  dirty_bitmap_directory_size
> +                   Size of the Dirty Bitmap Directory in bytes. Valid values:
> +                   0 - 67108864 (= 1024 * nb_dirty_bitmaps).
> +
> +          8 - 15:  dirty_bitmap_directory_offset
> +                   Offset into the image file at which the Dirty Bitmap
> +                   Directory starts. Must be aligned to a cluster boundary.
> +
> +
>   == Host cluster management ==
>   
>   qcow2 manages the allocation of host clusters by maintaining a reference count
> @@ -360,3 +385,103 @@ Snapshot table entry:
>   
>           variable:   Padding to round up the snapshot table entry size to the
>                       next multiple of 8.
> +
> +
> +== Dirty bitmaps ==
> +
> +The feature supports storing dirty bitmaps in a qcow2 image.
> +
> +=== Cluster mapping ===
> +
> +Dirty bitmaps are stored using a ONE-level structure for the mapping of
> +bitmaps to host clusters. It is called Dirty Bitmap Table.
> +
> +The Dirty Bitmap Table has a variable size (stored in the Dirty Bitmap
> +Directory Entry) and may use multiple clusters, however it must be contiguous
> +in the image file.
> +
> +Given an offset (in bytes) into the bitmap, the offset into the image file can
> +be obtained as follows:
> +
> +    byte_offset =
> +        dirty_bitmap_table[offset / cluster_size] + (offset % cluster_size)
> +
> +Taking into accout the granularity of the bitmap, an offset in bits into the
> +image file can be obtained like this:
> +
> +    bit_offset =
> +        byte_offset(bit_nr / granularity / 8) * 8 + (bit_nr / granularity) % 8
> +
> +Here bit_nr is a number of "virtual" bit of the bitmap, which is covered by
> +"physical" bit with number (bit_nr / granularity).
> +
> +Dirty Bitmap Table entry:
> +
> +    Bit  0 -  8:    Reserved
> +
> +         9 - 55:    Bits 9-55 of host cluster offset. Must be aligned to a
> +                    cluster boundary. If the offset is 0, the cluster is
> +                    unallocated, and should be read as all zeros.
> +
> +        56 - 63:    Reserved
> +
> +=== Dirty Bitmap Directory ===
> +
> +Each dirty bitmap, saved in the image is described in the Dirty Bitmap
> +Directory entry. Dirty Bitmap Directory is a contiguous area in the image file,
> +whose starting offset and length are given by the header extension fields
> +dirty_bitmap_directory_offset and dirty_bitmap_directory_size. The entries of
> +the bitmap directory have variable length, depending on the length of the
> +bitmap name.
> +
> +Dirty Bitmap Directory Entry:
> +
> +    Byte 0 -  7:    dirty_bitmap_table_offset
> +                    Offset into the image file at which the Dirty Bitmap Table
> +                    for the bitmap starts. Must be aligned to a cluster
> +                    boundary.
> +
> +         8 - 15:    nb_virtual_bits
> +                    Number of "virtual" bits in the bitmap. Number of
> +                    "physical" bits would be:
> +                    (nb_virtual_bits + granularity - 1) / granularity
> +
> +        16 - 19:    dirty_bitmap_table_size
> +                    Number of entries in the Dirty Bitmap Table of the bitmap.
> +                    Valid values: 0 - 0x8000000.
> +                    Also, (dirty_bitmap_table_size * cluster_size) should not
> +                    be greater than 0x20000000 (512 MB)
> +
> +        20 - 23:    granularity_bits
> +                    Granularity bits. Valid values are: 0 - 63.
> +
> +                    Granularity is calculated as
> +                        granularity = 1 << granularity_bits
> +
> +                    Granularity of the bitmap is how many "virtual" bits
> +                    accounts for one "physical" bit.
> +
> +        24 - 27:    flags
> +                    Bit
> +                      0: in_use
> +                         The bitmap is in use and may be inconsistent.
> +
> +                      1: self
> +                         The bitmap is a dirty bitmap for the containing image.
> +
> +                      2: auto
> +                         The bitmap should be autoloaded as block dirty bitmap.
> +                         Only available if bit 1 (self) is set.
> +
> +                      3: read_only
> +                         The bitmap should not be rewritten.
> +
> +                    Bits 4 - 31 are reserved.

Is this appropriate as field, reserved for future extensiion? Or we need 
an additional one? Do we need scheme like with snapshots? (somthing like 
field 'additional_area_size', and additional offset of this size after 
the name)

> +
> +        28 - 29:    name_size
> +                    Size of the bitmap name. Valid values: 0 - 1023.
> +
> +        variable:   The name of the bitmap (not null terminated).
> +
> +        variable:   Padding to round up the Dirty Bitmap Directory Entry size to
> +                    the next multiple of 8.


-- 
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.