From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:56282) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZTsMs-0000tO-Dj for qemu-devel@nongnu.org; Mon, 24 Aug 2015 10:04:59 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1ZTsMl-0007Ev-UK for qemu-devel@nongnu.org; Mon, 24 Aug 2015 10:04:58 -0400 Received: from relay.parallels.com ([195.214.232.42]:46348) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1ZTsMl-0007DH-IG for qemu-devel@nongnu.org; Mon, 24 Aug 2015 10:04:51 -0400 Message-ID: <55DB2475.2010402@virtuozzo.com> Date: Mon, 24 Aug 2015 17:04:37 +0300 From: Vladimir Sementsov-Ogievskiy MIME-Version: 1.0 References: <1433776886-27239-1-git-send-email-vsementsov@virtuozzo.com> <1433776886-27239-2-git-send-email-vsementsov@virtuozzo.com> In-Reply-To: <1433776886-27239-2-git-send-email-vsementsov@virtuozzo.com> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH 1/8] spec: add qcow2-dirty-bitmaps specification List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel@nongnu.org Cc: kwolf@redhat.com, Vladimir Sementsov-Ogievskiy , stefanha@redhat.com, pbonzini@redhat.com, den@openvz.org, jsnow@redhat.com About structs and constraints: == Optional Header == 64bit nb_dirty_bitmaps valid: 1 - 65536. I think here should not be 0, in this case dirty-bitmap-optional-header should not exist at all. Should it instead be 0 - 65536 64bit dirty_bitmaps_offset valid: any, but dirty_bitmaps_offset % cluster_size = 0 == Dirty BItmap Directory Enrty ( = bitmap header) == 64bit dirty_bitmap_table_offset valid: any, but dirty_bitmap_table_offset % cluster_size = 0 64bit nb_virtual_bits (before it was called bitmap_size) valid: no direct constraints (as for disk size), but it should be <= dirty_bitmap_table_size * cluster_size * 8 * bitmap_granularity 32bit dirty_bitmap_table_size ? The bitmap will take ~ dirty_bitmap_table_size * cluster_size bytes in RAM. What the limit should be for it? for bdrv dirty bitmaps max disk size = QCOW_MAX_L1_SIZE / 8 * (cluster_size / 8) * cluster_size = 524288 * cluster_size ^ 2, dirty bitmap covers the following size: dirty_bitmap_table_size * cluster_size * 8 * byte_granularity therefore, to cover the whole disk, dirty_bitmap_table_size * cluster_size * 8 * byte_granularity >= 524288 * cluster_size ^ 2 i.e. dirty_bitmap_table_size >= 65536 * cluster_size / byte_granularity max cluster size is 0x200000 ( = 1 << 21 ), min granularity = sector_size = 512, so dirty_bitmap_table_size >= 0x10000000, and this limit should be enough for any current disk configuration, but, with cluster of size 1 << 21 and such dirty_bitmap_table_size we will have tooo huge bitmap in RAM and it is unreal.. 32bit bitmap_granularity_bits ( before it was bitmap_granularity) valid; 0 - 63 (as for HBitmap) (1 << bitmap_granularity_bits) is number of virtual bits in one physical bit. Not related to sectors/bytes, etc. Let this format be closer to HBitmap than to BdrvDirtyBitmap 16bit name_size valid: 1 - 1023. // should it be 0 - 1023 ? /* name follows */ /* offset to 8 bytes boundary follows */ ==About granularity== To make things more general, granularity is not related to disk/bytes/sectors. Granularity is just number of virtual bits in one physical bit. Then, for bdrv dirty bitmaps (let's assume, that it may be not only one type of store dirty bitmaps), let establish, that one virtual bit is corresponding to one sector of the disk, i.e., byte granularity would be: byte_granularity = (1 << bitmap_granularity_bits) * 512 for ex: 65536 = (1 << 7) * 512 On 08.06.2015 18:21, Vladimir Sementsov-Ogievskiy wrote: > From: Vladimir Sementsov-Ogievskiy > > Persistent dirty bitmaps will be saved into qcow2 files. It may be used > as 'internal' bitmaps (for qcow2 drives) or as 'external' bitmaps for > other drives (there may be qcow2 file with zero disk size but with > several dirty bitmaps for other drives). > > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > docs/specs/qcow2.txt | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 66 insertions(+) > > diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt > index 121dfc8..0fffba2 100644 > --- a/docs/specs/qcow2.txt > +++ b/docs/specs/qcow2.txt > @@ -123,6 +123,7 @@ be stored. Each extension has a structure like the following: > 0x00000000 - End of the header extension area > 0xE2792ACA - Backing file format name > 0x6803f857 - Feature name table > + 0x23852875 - Dirty bitmaps > other - Unknown header extension, can be safely > ignored > > @@ -166,6 +167,19 @@ the header extension data. Each entry look like this: > terminated if it has full length) > > > +== Dirty bitmaps == > + > +Dirty bitmaps is an optional header extension. It provides a possibility of > +storing dirty bitmaps in qcow2 image. The fields are: > + > + 0 - 3: nb_dirty_bitmaps > + Number of dirty bitmaps contained in the image > + > + 4 - 11: dirty_bitmaps_offset > + Offset into the image file at which the dirty bitmaps table > + starts. Must be aligned to a cluster boundary. > + > + > == Host cluster management == > > qcow2 manages the allocation of host clusters by maintaining a reference count > @@ -360,3 +374,55 @@ Snapshot table entry: > > variable: Padding to round up the snapshot table entry size to the > next multiple of 8. > + > + > +== Dirty bitmaps == > + > +The feature supports storing several dirty bitmaps in the qcow2 file. > + > +=== Cluster mapping === > + > +Dirty bitmaps are stored using a ONE-level structure for the mapping of > +bitmaps to host clusters. There is only an L1 table. > + > +The L1 table has a variable size (stored in the Bitmap table entry) and may > +use multiple clusters, however it must be contiguous in the image file. > + > +Given an offset into the bitmap, the offset into the image file can be > +obtained as follows: > + > + offset = l1_table[offset / cluster_size] + (offset % cluster_size) > + > +L1 table entry: > + > + Bit 0 - 61: Standard cluster descriptor > + > + 62 - 63: Reserved > + > +=== Bitmap table === > + > +A directory of all bitmaps is stored in the bitmap table, a contiguous area in > +the image file, whose starting offset and length are given by the header fields > +dirty_bitmaps_offset and nb_dirty_bitmaps. The entries of the bitmap table have > +variable length, depending on the length of name and extra data. > + > +Bitmap table entry: > + > + Byte 0 - 7: Offset into the image file at which the L1 table for the > + bitmap starts. Must be aligned to a cluster boundary. > + > + 8 - 11: Number of entries in the L1 table of the bitmap > + > + 12 - 15: Bitmap granularity in bytes > + > + 16 - 23: Bitmap size in sectors > + > + 24 - 25: Size of the bitmap name > + > + variable: The name of the bitmap (not null terminated) > + > + variable: Padding to round up the bitmap table entry size to the > + next multiple of 8. > + > +The fields "size", "granularity" and "name" are corresponding with the fields > +in struct BdrvDirtyBitmap. -- Best regards, Vladimir * now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.