From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:33689) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zbt1n-0005f6-GT for qemu-devel@nongnu.org; Tue, 15 Sep 2015 12:24:21 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Zbt1i-0004eW-7b for qemu-devel@nongnu.org; Tue, 15 Sep 2015 12:24:19 -0400 Received: from mx1.redhat.com ([209.132.183.28]:46627) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Zbt1h-0004eI-Ts for qemu-devel@nongnu.org; Tue, 15 Sep 2015 12:24:14 -0400 References: <1441471439-6157-1-git-send-email-vsementsov@virtuozzo.com> <1441471439-6157-4-git-send-email-vsementsov@virtuozzo.com> From: Eric Blake Message-ID: <55F84627.5040105@redhat.com> Date: Tue, 15 Sep 2015 10:24:07 -0600 MIME-Version: 1.0 In-Reply-To: <1441471439-6157-4-git-send-email-vsementsov@virtuozzo.com> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="bnr9naXpd8PDSDS0D8wWhxuk5rRnKEj0P" Subject: Re: [Qemu-devel] [PATCH 03/17] spec: add qcow2-dirty-bitmaps specification List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Vladimir Sementsov-Ogievskiy , qemu-devel@nongnu.org Cc: kwolf@redhat.com, den@openvz.org, jsnow@redhat.com, stefanha@redhat.com, pbonzini@redhat.com This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --bnr9naXpd8PDSDS0D8wWhxuk5rRnKEj0P Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 09/05/2015 10:43 AM, Vladimir Sementsov-Ogievskiy wrote: > Persistent dirty bitmaps will be saved into qcow2 files. It may be used= > as 'internal' bitmaps (for qcow2 drives) or as 'external' bitmaps for > other drives (there may be qcow2 file with zero disk size but with > several dirty bitmaps for other drives). >=20 > Signed-off-by: Vladimir Sementsov-Ogievskiy > --- > docs/specs/qcow2.txt | 127 +++++++++++++++++++++++++++++++++++++++++++= +++++++- > 1 file changed, 126 insertions(+), 1 deletion(-) >=20 > diff --git a/docs/specs/qcow2.txt b/docs/specs/qcow2.txt > index 121dfc8..5fc0365 100644 > --- a/docs/specs/qcow2.txt > +++ b/docs/specs/qcow2.txt > @@ -103,7 +103,13 @@ in the description of a field. > write to an image with unknown auto-clear features= if it > clears the respective bits from this field first. > =20 > - Bits 0-63: Reserved (set to 0) > + Bit 0: Dirty bitmaps bit. If this bit is set = then > + there is a _consistent_ Dirty bitmaps = extension > + in the image. If it is not set, but th= ere is a > + Dirty bitmaps extension, its data shou= ld be > + considered as inconsistent. Thanks for documenting this. I don't know that we use underscore for _emphasis_ anywhere else in the file, but I don't have any better suggestions. Should you also require that it is an error if this bit is set but no Dirty bitmap extension header is present? > + > + Bits 1-63: Reserved (set to 0) > =20 > 96 - 99: refcount_order > Describes the width of a reference count block ent= ry (width > @@ -123,6 +129,7 @@ be stored. Each extension has a structure like the = following: > 0x00000000 - End of the header extension area > 0xE2792ACA - Backing file format name > 0x6803f857 - Feature name table > + 0x23852875 - Dirty bitmaps > other - Unknown header extension, can be = safely > ignored > =20 > @@ -166,6 +173,24 @@ the header extension data. Each entry look like th= is: > terminated if it has full length) > =20 > =20 > +=3D=3D Dirty bitmaps =3D=3D > + > +Dirty bitmaps is an optional header extension. It provides an ability = to store > +dirty bitmaps in a qcow2 image. The fields are: Might not hurt to remind the reader about the auto-clear feature bit mentioned earlier controlling whether this extension can be trusted as consistent. > + > + 0 - 3: nb_dirty_bitmaps > + The number of dirty bitmaps contained in the image.= Valid > + values: 0 - 65535. > + > + 4 - 7: dirty_bitmap_directory_size > + Size of the Dirty Bitmap Directory in bytes. Valid = values: > + 0 - 67108864 (=3D 1024 * nb_dirty_bitmaps). Is it always going to be 1024 * nb_dirty_bitmaps? If so, why do we need a redundant field? If not, then this wording needs help; from the rest of this text, it looks like you want "at most 1024 * nb_dirty_bitmaps". Also, while Dirty Bitmap Directory entries are variable length (and thus a variable maximum), they do have a minimum size (so the minimum value for dirty_bitmap_directory_size must be larger than 0 unless nb_dirty_bitmaps is 0, in which case why would we have this header extension) > + > + 8 - 15: dirty_bitmap_directory_offset > + Offset into the image file at which the Dirty Bitma= p > + Directory starts. Must be aligned to a cluster boun= dary. > + > + > =3D=3D Host cluster management =3D=3D > =20 > qcow2 manages the allocation of host clusters by maintaining a referen= ce count > @@ -360,3 +385,103 @@ Snapshot table entry: > =20 > variable: Padding to round up the snapshot table entry size = to the > next multiple of 8. > + > + > +=3D=3D Dirty bitmaps =3D=3D > + > +The feature supports storing dirty bitmaps in a qcow2 image. > + > +=3D=3D=3D Cluster mapping =3D=3D=3D > + > +Dirty bitmaps are stored using a ONE-level structure for the mapping o= f > +bitmaps to host clusters. It is called Dirty Bitmap Table. s/ONE/one/ (I didn't see the reason for the emphasis) > + > +The Dirty Bitmap Table has a variable size (stored in the Dirty Bitmap= s/The/Each/ > +Directory Entry) and may use multiple clusters, however it must be con= tiguous > +in the image file. > + > +Given an offset (in bytes) into the bitmap, the offset into the image = file can > +be obtained as follows: > + > + byte_offset =3D > + dirty_bitmap_table[offset / cluster_size] + (offset % cluster_= size) > + > +Taking into accout the granularity of the bitmap, an offset in bits in= to the s/accout/account/ > +image file can be obtained like this: > + > + bit_offset =3D > + byte_offset(bit_nr / granularity / 8) * 8 + (bit_nr / granular= ity) % 8 > + > +Here bit_nr is a number of "virtual" bit of the bitmap, which is cover= ed by > +"physical" bit with number (bit_nr / granularity). I got a bit lost on this sentence. Maybe an example would help? Is the idea that every image has a certain number of clusters, one "virtual" bit per cluster, and then the bitmap compresses multiple clusters into one "physical" bit according to a compression ratio determined by the bitmap granularity? That is, if I have an image with 64k clusters but 128k bitmap granularity, then each physical bit of the bitmap covers 2 clusters as being dirty? > + > +Dirty Bitmap Table entry: > + > + Bit 0 - 8: Reserved s/Reserved/Reserved, must be 0/ > + > + 9 - 55: Bits 9-55 of host cluster offset. Must be aligned = to a > + cluster boundary. If the offset is 0, the cluster = is > + unallocated, and should be read as all zeros. > + > + 56 - 63: Reserved and again (specifying the user must write 0 for now leaves the door open for extension) > + > +=3D=3D=3D Dirty Bitmap Directory =3D=3D=3D > + > +Each dirty bitmap, saved in the image is described in the Dirty Bitmap= s/bitmap,/bitmap/ s/in the Dirty/in a Dirty/ > +Directory entry. Dirty Bitmap Directory is a contiguous area in the im= age file, > +whose starting offset and length are given by the header extension fie= lds > +dirty_bitmap_directory_offset and dirty_bitmap_directory_size. The ent= ries of > +the bitmap directory have variable length, depending on the length of = the > +bitmap name. > + > +Dirty Bitmap Directory Entry: > + > + Byte 0 - 7: dirty_bitmap_table_offset > + Offset into the image file at which the Dirty Bitm= ap Table > + for the bitmap starts. Must be aligned to a cluste= r > + boundary. > + > + 8 - 15: nb_virtual_bits > + Number of "virtual" bits in the bitmap. Number of > + "physical" bits would be: > + (nb_virtual_bits + granularity - 1) / granularity > + > + 16 - 19: dirty_bitmap_table_size > + Number of entries in the Dirty Bitmap Table of the= bitmap. > + Valid values: 0 - 0x8000000. > + Also, (dirty_bitmap_table_size * cluster_size) sho= uld not > + be greater than 0x20000000 (512 MB) > + > + 20 - 23: granularity_bits > + Granularity bits. Valid values are: 0 - 63. > + > + Granularity is calculated as > + granularity =3D 1 << granularity_bits 63 seems like a rather high limit. Even 32 (1 bit covering 4 billion clusters) is huge. > + > + Granularity of the bitmap is how many "virtual" bi= ts > + accounts for one "physical" bit. > + > + 24 - 27: flags > + Bit > + 0: in_use > + The bitmap is in use and may be inconsistent.= > + > + 1: self > + The bitmap is a dirty bitmap for the containi= ng image. > + > + 2: auto > + The bitmap should be autoloaded as block dirt= y bitmap. > + Only available if bit 1 (self) is set. > + > + 3: read_only > + The bitmap should not be rewritten. > + > + Bits 4 - 31 are reserved. > + > + 28 - 29: name_size > + Size of the bitmap name. Valid values: 0 - 1023. This limit is inconsistent with the limit above that a directory entry will be at most 1024 bytes; since you have already burned 30 bytes on essential information. Is 0 allowed (a bitmap with "" as its name), or must the name_size be at least 1? Should you document constraints that each bitmap name should be unique within the file (that is, no two bitmaps in the directory have the same name)? > + > + variable: The name of the bitmap (not null terminated). > + > + variable: Padding to round up the Dirty Bitmap Directory Ent= ry size to > + the next multiple of 8. >=20 --=20 Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --bnr9naXpd8PDSDS0D8wWhxuk5rRnKEj0P Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCAAGBQJV+EYoAAoJEKeha0olJ0NqpukH/A04GWlijr1wY2lmvd/ZcvS4 QGaHIk/IDtUGkffeeBpJA0pdILgNAQq24vXCZGyBBub/4+TLX1TuhtumAHO+peLw byeqRNNUt62+8y3nLj5drFFN/Nke72evleKvMRXGlfSjWML8qakvMCu+iNdK33LE engbFx9qYEOFM+mmLP0X8ro4ZzpjbJk2v04tol6GCsYi2INyqMG6ohipPgdfM0G0 m5eRIBCa0c7JuukVaETltpR03TCVuZUXNUJNHvaU40geE6w3HqddM6ruyMvdox9R 31ZDTJNsdXPjuxTsapzep6KQQ12lBoNIlaw2FkhgICTlafo0300whWI5wKYCZy8= =7pkd -----END PGP SIGNATURE----- --bnr9naXpd8PDSDS0D8wWhxuk5rRnKEj0P--