From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:46425) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aaPIb-0001hJ-7k for qemu-devel@nongnu.org; Mon, 29 Feb 2016 09:59:50 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1aaPIY-0002tP-0l for qemu-devel@nongnu.org; Mon, 29 Feb 2016 09:59:49 -0500 Received: from mx1.redhat.com ([209.132.183.28]:45802) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1aaPIX-0002sy-Ov for qemu-devel@nongnu.org; Mon, 29 Feb 2016 09:59:45 -0500 References: <20160227050038.X5XK3.255642.root@dnvrco-web06> From: Eric Blake Message-ID: <56D45CDD.40400@redhat.com> Date: Mon, 29 Feb 2016 07:59:41 -0700 MIME-Version: 1.0 In-Reply-To: <20160227050038.X5XK3.255642.root@dnvrco-web06> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="ifFoPA2GvebXMK5V6oj8tNGLLowhOrG2x" Subject: Re: [Qemu-devel] QCow2 compression List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: mgreger@cinci.rr.com, qemu-devel@nongnu.org This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --ifFoPA2GvebXMK5V6oj8tNGLLowhOrG2x Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 02/26/2016 10:00 PM, mgreger@cinci.rr.com wrote: > Hello, I am hoping someone here can help me. I am implementing QCow2 su= pport for a PC emulator project and have a couple questions regarding com= pression I haven't been able to figure out on my own. [Can you convince your mailer to wrap long lines? It makes replying to your mail easier] >=20 > First some background: > I am using the information I found at https://people.gnome.org/~markmc/= qcow-image-format.html That page is outdated. Better is the current spec, maintained directly in qemu.git: http://git.qemu.org/?p=3Dqemu.git;a=3Dblob;f=3Ddocs/specs/qcow2.txt > and I have implemented working support for QCow2 images as described th= ere except for snapshots, encryption, and compression. Of these, only com= pression is of immediate use to me. Do NOT implement encryption, at least not in the form documented in anything you've read so far. We have a pending patch to implement LUKS encryption, to replace the insecure existing encryption, although Dan's latest email says it might not land until qemu 2.7: https://lists.gnu.org/archive/html/qemu-devel/2016-02/msg06552.html >=20 > I have some QCow2 images all using 16-bit clusters created using qemu-i= mg 2.1.2 You may want to use current qemu.git, which will soon be 2.5, although compression hasn't really changed since then. > According to the documentation I linked, Better, according to the current documentation: > Compressed Clusters Descriptor (x =3D 62 - (cluster_bits - 8)): > > Bit 0 - x: Host cluster offset. This is usually _not_ aligned to a > cluster boundary! > > x+1 - 61: Compressed size of the images in sectors of 512 bytes > an L2 entry value of 4A C0 00 00 00 3D 97 50. So with default 64k clusters, x =3D 62 - (16 - 8) =3D 54. Bits 0-54 are = the host cluster offset, or 0x003d9750, but that is in terms of host sectors. The comment in block/qcow2.c is telling, and perhaps we should improve the qcow2 spec to make it obvious: - Size of compressed clusters is stored in sectors to reduce bit usage in the cluster offsets. Thus, in your image, the guest compressed data starts at sector 0x003d9750, or host file offset 0x7b2ea000. This value is NOT aligned to a cluster, but IS aligned to a sector (since a sector is the smallest unit we write to), and makes more sense than something ending in 0x50 (which is not sector aligned). > This would lead me to believe the cluster starts at offset 0x3D9750 and has a length of 0x2B 512-byte sectors (or 0x2B times 0x200 =3D 0x5600= ). You are correct about the 64k of guest data being compressed into 0x5600 bytes in the host file, but incorrect at where to read those bytes. >=20 > A final question: I noticed that compressed clusters typically have a r= eference count higher than one, yet there are no snapshots present in the= image. I suspect the count is incremented for each compressed cluster th= at exists even partially within a normal sized cluster region of the file= , but I can find no documentation to this effect and am merely speculatin= g. Am I correct? Yes, you are correct, and yes, the spec I pointed to documents that. Since the L2 entry in your example occupies only 43/128 sectors, any other adjacent clusters from the guest point of view can be compressed into the remaining sectors of the host cluster, and the refcount must be equal to the number of all compressed guest clusters that occupy at least one sector of the host cluster. >=20 > If it is the wrong place to ask these questions, I would appreciate it = if anyone could direct me to a more appropriate venue. Thank you for taki= ng the time to read this and tanks in advance for any assistance. This is the right list. --=20 Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org --ifFoPA2GvebXMK5V6oj8tNGLLowhOrG2x Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Public key at http://people.redhat.com/eblake/eblake.gpg Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCAAGBQJW1FzdAAoJEKeha0olJ0NqTfAH/ipiZAf/FCPlUYAyTT9KzvZT 3oQ3JHKgP1ENs3oMMSkXVZFVhVBy9BqzsLm4xqQxh/hpqLAZ+uhyFRMexLkjIeZG ZMN4xATF7zMbK1U72Qc3/pQoCEA6lg4VwNsTkMLyKlbZB7wjUXphV+/EGl4SL08R Qnylf3Sju3eroGzsvWIIkgOQHquF+kUR6U6KHY8QXB5wgYJnIGPoyq6iD8XYKyGQ SAqNQyMWdFOIXCCIz4zwPiOdIvzpAbPdYxl6ebJ2pw0n1h1ij9C9w5Ol2VDBbXl6 pLIoC5eufqySd186NE8r13dIEWTUKr3552CuuG8KAvCqIt3I/DEVHpNxu6W16D8= =j4wN -----END PGP SIGNATURE----- --ifFoPA2GvebXMK5V6oj8tNGLLowhOrG2x--